Representations exact at inference improve out of distribution generalization
Exactness at Inference: A Representational Criterion for Out-of-Distribution Generalization
Machine LearningArtificial IntelligenceLogic in Computer Science
Summary
Machine learning models often fail when faced with new kinds of data outside what they were trained on. This paper finds that true generalization requires models to compute representations that perfectly match the data’s underlying logic, not just close approximations. The authors show this exactness governs how models extrapolate, and reveal some existing methods fall short because they approximate rather than represent exactly. The work suggests focusing on exact logical representations at inference to better handle new situations.
What this means in practice
- •For machine learning engineers: Identify when models will fail to generalize beyond training data by checking the exactness of their representations at inference.
- •For ai system designers: Design hybrid architectures that propagate strict bounds on inference outputs to improve reliability on new data distributions.
A theory result. No direct application yet.
Authors
Filipe Marinho Rocha, Inês Dutra, Vítor Santos Costa, Luís Paulo Reis
Abstract
A model generalizes outside its training distribution only when it computes a representation structurally equivalent to the generating mechanism, not an approximation fitted to it. Such equivalence is necessary for exactness in and out of distribution, and extrapolation is governed by this exactness at inference, whatever its realization. Tensor Logic shows this: a zero-temperature contraction is equivalent to discrete logic, deducing in place with no artefact extracted, its tensors Boolean, its embeddings orthonormal, only its arithmetic continuous. Lacking infinite recursion it reaches Datalog, not Prolog, and though exact over closed domains it needs external memory to bind a novel entity. The criterion needs neither a discrete representation nor an extracted expression, and constrains inference, not training: an exact marginal in $[0,1]$ passes, a Neural Network thresholded to a hard label does not. Logic Tensor Networks fail it, while differentiable ILP and Tensor Logic at $T=0$ pass. Piecewise-affine extrapolation divergence and an inability to bind novel entities are two faces of a shortfall in exact representability. For hybrid architectures, a propagation rule follows: the output inherits the bounds of every fitted estimator on its path, explaining which axes fail in equivariant models and the ARC-AGI induction/transduction split. Only an exact hypothesis class certifies what the training data leave underdetermined: on a law-derived partition it finds the $56.3\%$ of distant queries that are answerable, which ensembles meet with false confidence and distance metrics rank backwards. Common inductive biases, from symmetries to memory, reach exactness only because humans inject them, an argument for inducing exact representations rather than fitting surrogates whose residuals, even at the arithmetic floor in training, diverge outside the data and compound under composition.