Encoding limits restrict what machine learning models can predict
Representable but Unlearned: Encoding Rank and the Interaction-Prediction Floor
Machine LearningArtificial Intelligence
Summary
Sometimes, the way data is encoded limits what a machine learning model can learn, even if it looks like the model should be able to. The authors show a way to calculate the best possible predictions any model can make given the encoding, without any training needed. They tested this on genetic data and found that some encodings merge many data states together, limiting how well interactions can be predicted. However, changing the encoding can restore full predictability, separating the limits of encoding from the limits of model training.
What this means in practice
- •For machine learning engineers: Check whether input encodings limit achievable prediction accuracy before model training to avoid wasted effort on unlearnable data features.
- •For bioinformatics developers: Improve biological interaction prediction by identifying when encoding merges data states that reduce model performance and restoring missing data features.
Authors
Zahra Khodagholi, Niloofar Yousefi
Abstract
Input encodings can restrict which measured contrasts a predictor can jointly reproduce, even when no single contrast is forced to vanish. We compute the attainable contrast space from an encoder's equivalence classes and a fixed contrast design, without labels, loss, or a fitted model; projecting the recorded contrasts onto that space gives an empirical error floor for any unrestricted decoder on those classes. On a 140-rectangle siRNA interaction panel, a graph neural network's training-only feature mask merges 165 endpoint states into 90 classes and cuts the rank of the 140 interaction contrasts to 72. The resulting floor is 0.009980, which is 14.6% of the fitted model's interaction squared error; the fitted model reaches 0.068335, slightly worse than a control predicting no interaction at all. A minimum of three restored chemistry columns recovers full rank. Refitting without the mask removes the floor entirely, yet interaction MSE improves by only 0.000017 under the reported protocol, and the restored columns remain absent from every training input. On a released RNA-splicing predictor, whose encoding is injective on the measured states, the same computation returns the full design rank of 1,986 and a floor of exactly zero. These results separate what an encoding permits from what a fitted model achieves; they do not identify what limits the remaining error. The rank check needs no fits and bounds what any amount of training under a fixed encoding can recover. The project repository is available at https://github.com/shadi97kh/REPRESENTABLE-BUT-UNLEARNED.