Support vector machines struggle with small samples in high dimensions
Asymptotic Properties of Support Vector Machines in High-Dimension, Low-Sample-Size Settings under a Spiked Model
Machine Learning
Summary
Support vector machines (SVMs) are tools that help classify data, but they don't always work well when there are many features but very few samples, a situation common in some fields. The authors show that existing explanations for SVM behavior don't apply when data have a few dominating patterns, called spikes. They find that in these cases, the SVM often misclassifies data and doesn't improve with more data unless a new approach is used. The authors propose a modified SVM method that accounts for these spikes, improving classification when more samples are available.
What this means in practice
- •For machine learning engineers: Improve classification accuracy on datasets with many features but limited samples by using spike-corrected SVM instead of standard SVM approaches.
- •For biomedical data analysts: Better analyze genetic or imaging data where dominant patterns skew classification, leveraging spike-corrected SVM to reduce errors in diagnostic models.
Authors
Yugo Nakayama
Abstract
In this paper, we consider asymptotic properties of the support vector machine (SVM) in high-dimension, low-sample-size (HDLSS) settings under a spiked model. The existing theory of the SVM in the HDLSS context relies on the geometric representation of HDLSS data, which requires that the eigenvalues of the covariance matrices are not dominant. We first show that the geometric representation does not hold under the spiked model. We show that the Gram matrix of HDLSS data converges in distribution to a random matrix, namely, the HDLSS data converge to a random configuration in a finite-dimensional space whose dimension is given by the number of the spikes. We show that the misclassification rates of the SVM do not tend to zero, that is, the SVM does not hold the consistency property. We also show that the bias-corrected SVM (BC-SVM) does not give preferable performance in this setting because the bias term itself should be modified. In order to overcome such difficulties, we propose a spike-corrected SVM (SC-SVM). We show that the SC-SVM holds the consistency property when the sample size goes to infinity, and that the growth of the sample size is essential in the sense that any projection-based procedure fails when the sample size is fixed. Finally, we check the performance of the classifiers by numerical simulations.