Compression reveals language model ability to predict new text accurately
Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Computation and LanguageArtificial Intelligence
Summary
It can be hard to tell if large language models really understand new tasks because they might have seen the test questions before. The authors created Uncheatable Eval, a way to test models using freshly published text that the models likely haven't seen. They use how well a model can compress text as a stand-in for how well it predicts that text. Their tests show that bigger models compress better, different model types compress differently as context grows, and better compression links to higher test accuracy. This method helps evaluate base models more reliably without cheating.
What this means in practice
- •For machine learning engineers: Assess base language model quality on fresh text without test data leakage using compression-based evaluation.
- •For ai safety teams: Monitor language model training to detect and prevent contamination of evaluation benchmarks with training data.
Authors
Kaifeng Tan, Yudong Li, Linlin Shen
Abstract
Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is particularly challenging for base models, whose limited instruction-following ability complicates task-based assessment. We introduce Uncheatable Eval, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination. Drawing on the relationship between a model's predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text. We evaluate 80 models across 14 text categories, study how compression changes with context length, and examine the correlation between compression rate and zero-shot MMLU accuracy. Our results yield three main findings: (1) compression performance follows a consistent scaling trend with model size; (2) attention-based, hybrid, and recurrent models differ in how their compression performance changes as more context becomes available; and (3) lower compression rates are strongly associated with higher zero-shot MMLU accuracy. Code is available at https://github.com/Jellyfish042/uncheatable_eval.