Text representations can look uniform but hide sequence details
Isotropic Yet Undecodable: The Sequential Content-Sufficiency Gap in Latent-Predictive Text Representations
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
This paper investigates if certain text representations keep the order of words intact. The authors show that some representations can appear perfectly regular but actually lose the sequential information needed to recover the original text. They develop a new method called CANOPE, which improves how well ordered information is preserved and accessed. Experiments with corrupted text confirm that just having regular geometric properties does not guarantee good recoverability of word order.
What this means in practice
- •For natural language engineers: Improve text encoding models to retain word order reliably even under corruption for better downstream tasks.
- •For speech recognition developers: Develop more robust text-to-speech systems that handle corrupted input better by using latent representations that preserve token order.
Authors
K. P. Santoso, N. Z. Fadil, F. P. Harsanti, R. V. H. Ginardi, G. N. Iyer
Abstract
We study sequential content sufficiency by investigating whether a representation retains the ordered target information available in its input. An information-theoretic decomposition separates input ambiguity, representation loss, and readout mismatch. We construct recoverable views where perfect agreement and joint isotropic Gaussianity coexist with zero target information, and establish limits imposed by deterministic canonical anchors. Token log-loss provides a one-sided information-loss bound; a fixed-penalty ridge analysis shows why rank alone cannot determine prediction risk. These results motivate CANOPE, a nonautoregressive framework with ordered latent canvases, canonical-token supervision, and geometric regularization. On 40,000 validation sequences, latent-agreement (PL0) and token-grounded (PL2) have nearly identical pooled ranks but reach 13.5% and 98.8% positional Recall@1, respectively, under strong natural corruption when the correct target length is provided. On 3,930 LJSpeech validation utterances, frozen PL2 with a trained MatchaTTS readout yields 21.54% word error rate (WER) on corrupted text, versus 99.22% for frozen PL0, while end-to-end MatchaTTS reaches 10.93%. These results show that geometric regularity alone does not guarantee recoverable sequential content or effective downstream access in the text settings studied here.