Conic classification heads may not keep features within bounded regions

Asymptotic Margins and Feature Confinement in Conic Classification Heads under Softmax Cross-Entropy

Machine Learning

Summary

This paper studies a type of classifier that uses curved boundary regions to separate different classes of data. The authors investigate whether training such classifiers with a common loss function called softmax cross-entropy keeps the features within those bounded regions. They find that even when class regions are bounded, the features themselves can still grow without bound during training if certain mathematical conditions are met. Their theory is confirmed by experiments on image datasets, showing the classifier features are not always confined as the bounded regions might suggest.

What this means in practice

  • For machine learning engineers: Design classification heads knowing that bounded class regions do not guarantee bounded features under softmax training.
  • For computer vision developers: Improve models for recognizing known and rejecting unknown image classes by understanding feature behavior in conic classification heads.

Tested on one dataset.

Authors

Omer Nezih Gerek

Abstract

Classification heads built on conic functions give each class a bounded region of the feature space, and such regions are used to reject samples of unseen classes. We ask whether these regions also keep the features bounded when the head is trained with softmax cross-entropy. For a head whose class scores combine a linear term with a Euclidean norm, we first show that the region of each class is bounded by a quadric of revolution, whose eccentricity is the reciprocal of one learned ratio. We then attach to each class an asymptotic margin, which compares the growth of its score with the growth of all the other scores and can be computed by a second-order cone program. The loss of a class has infimum zero exactly when its margin is positive, and it then approaches zero only along diverging features. When the margin is negative, the loss is coercive and its minimizers lie in an explicit ball. Since softmax depends only on the differences between scores, a class with a bounded region can still have a positive margin, so bounded regions do not confine the features. The result extends to other norms, including the polyhedral conic functions. On CIFAR-10 and CIFAR-100, all 1210 cells of 67 trained conic heads have positive margins, whereas a prototype head keeps nine of its ten cells bounded, as the theory predicts.