Efficient method improves multi-token prediction in language models

ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

Computation and Language

Summary

Predicting multiple future words in a sentence usually requires extra parts in a language model, making it slower and more complex. The authors propose a new way where the model predicts multiple future tokens using its own last layer, avoiding extra components. They also introduce a way to focus training more on easier predictions to keep costs low. This helps the model learn better for tasks like reasoning and code generation with less training effort, and improves how it generates longer texts when used. Their method works well even after more training or fine-tuning.

What this means in practice

  • For natural language processing engineers: Train language models that predict multiple upcoming tokens efficiently without adding model complexity, improving performance on reasoning and coding tasks.
  • For software developers in ai writing tools: Build text generation systems that maintain coherent multi-token drafts for longer output with better alignment, enhancing user experience.$Commercial implications: This paper enables advanced AI writing tools that produce more coherent multi-token drafts, improving product quality and user satisfaction.

Authors

Yuxuan Zhou, Junyuan Shang, Tianyu Wang, Chunrui Zheng, Zichang Liu, Wenbin Chang, Shuohuan Wang, Dianhai Yu

Abstract

Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.