Language models learn exact probabilities from rounded and limited numbers
Training Autoregressive Language Models on Rounded, Bounded, and Censored Numerical Labels via Exact Likelihood
Machine Learning
Summary
Some numbers we get from measurements or observations aren't exact because they are rounded, given as ranges, or cut off when a test stops early. The authors came up with a new way to teach language models to understand these kinds of numbers exactly by focusing on their digits and boundaries. This helps the model give better guesses about numbers even when the information is incomplete or uncertain. Their method works well even with very large sets of possible numbers.
What this means in practice
- •For medical data teams: Improve models estimating patient lab results reported as ranges or upper limits by training on exact digit boundaries.
- •For machine learning engineers: Train language models to predict response times or latencies from partially observed or cutoff timing data with exact probabilities.
Authors
Jingquan Yan, Yuwei Miao, Junzhou Huang
Abstract
Numerical labels often specify a set of possible values: a measurement is rounded, a laboratory test reports only a bound, or a program is stopped at a timeout. We introduce boundary marginalization, the first method that trains an autoregressive language model on the exact likelihood of such observations through its existing digit tokens. A fixed-width, order-preserving numeral representation groups consecutive grid values under shared digit prefixes. An interval is a union of such groups, so its exact probability and gradient follow from at most two boundary paths, the successive digit prefixes of its endpoints, each of length logarithmic in the number of grid values. The resulting objective unifies precise, rounded, bounded, and censored supervision and trains digit decisions at the resolution supplied by each observation. The same computation supports exact distributional queries and unbiased gradients for the continuous ranked probability score on exact labels. Experiments show recovery of a conditional distribution from rounded and censored reports alone, improved likelihood on naturally censored molecular measurements, and more accurate code latency distributions on two backbones. Exact interval updates remain tractable as the numerical support grows to one million values.