Papers for

software developers in ai writing tools

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Efficient method improves multi-token prediction in language models

ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

Abstract: Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.

Sat 17 OctComputation and Language
The gist
Predicting multiple future words in a sentence usually requires extra parts in a language model, making it slower and more complex. The authors propose a new way where the model predicts multiple future tokens using its own last layer, avoiding extra components. They also introduce a way to focus training more on easier predictions to keep costs low. This helps the model learn better for tasks like reasoning and code generation with less training effort, and improves how it generates longer texts when used. Their method works well even after more training or fine-tuning.
Open → 2610.24766v1