Spectral conjugate gradient method improves svm training speed and accuracy

A family of spectral conjugate gradient algorithms derived by least-squares approximations based on a modified quasi--Newton update with application to a revised robust binary classification model

Machine Learning

Summary

The paper addresses a way to improve how computers solve certain math problems called optimization tasks, which are important in machine learning and other areas. The authors develop a new version of an existing optimization method that keeps it stable while making it more efficient and ensuring it finds better solutions. They tested their approach against similar methods and found it performs well on standard benchmark problems. Lastly, they apply their method to improve the training of a type of machine learning model used for classifying data into two groups, showing benefits in speed and accuracy.

What this means in practice

  • For machine learning engineers: Reduce training time and improve accuracy of binary classifiers by applying the new spectral conjugate gradient algorithm to robust SVM models.
  • For optimization software developers: Incorporate the novel spectral parameter selection method to enhance conjugate gradient optimizers for faster convergence on diverse unconstrained problems.

Authors

Saman Babaie-Kafaki, Maryam Khoshsimaye-Bargard, Ahmad Mousavi

Abstract

We develop a spectral three-term modification of the classic Hestenes--Stiefel conjugate gradient algorithm, preserving its anti-jamming characteristic and, simultaneously, taking care of the sufficient descent property. We discuss how a modified secant equation can be extracted from our modification scheme, yielding a memoryless BFGS updating formula. Then, the spectral parameter of our method is obtained by steering its direction toward the given BFGS direction within a least-squares context. Using our technical improvements, we outline the general framework of our algorithm and discuss its theoretical features, including the descent and convergence properties, without the convexity assumption. We put our algorithm to the test in comparison with the three other conjugate gradient algorithms on a set of CUTEr unconstrained optimization test models, comparing the outputs using the Dolan--More measure. Next, we provide a concise evaluation of the results, highlighting the practical advantages of our algorithm. As a real-world case study, we introduce a reduced quadratic surface SVM with the rescaled loss for robust binary classification and apply the proposed algorithm to assess its accuracy and training time against several other SVMs.
View PDFOpen arXiv