Orthrus lossless decoding depends on numerical precision used
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Computation and LanguageArtificial Intelligence
Summary
Orthrus is a method to speed up language model text generation by guessing multiple words at once without changing the original model. The authors tested if Orthrus always produces exactly the same text as the original model when using different numerical precisions. They found that with lower precision (BF16), it only exactly matches about half the time, but with higher precision (FP32), it perfectly matches every time on their tests. This means how "lossless" Orthrus is depends on the computer’s math precision, and researchers should check exact text matches separately from how well the generated text performs on standard tests.
What this means in practice
- •For language model engineers: Choose numerical precision settings that guarantee exact token generation matching when using Orthrus for faster model inference.
- •For machine learning ops teams: Monitor and evaluate the impact of precision choices in production language models that use speculative decoding methods similar to Orthrus.
Authors
Ilya Koziev, Leonid Sinev, Ivan Oseledets
Abstract
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.