Credit alignment method improves cross-tokenizer model learning scores

CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

Machine LearningComputation and Language

Summary

When computers learn from one model to improve another using different ways of breaking up sentences into tokens, it’s hard to tell exactly how well the learning is going token by token. The authors propose a new method called CAST that carefully shares credit for model improvements across token groups and keeps parts of the student model stable where the teacher gives no guidance. This helps the student model learn more accurately from the teacher, even when they use different tokenizers. Their tests show that CAST improves performance across various math, science, and coding tasks.

What this means in practice

  • For machine learning engineers: Improve model compression and transfer learning between models using different tokenization methods in production AI systems.
  • For natural language processing teams: Enhance training procedures when fine-tuning language models that use different tokenizers to better match teacher feedback.
  • For financial technology developers: Improve accuracy of AI models that translate between tokenized financial texts produced by different vendor systems.$Commercial implications: Enables creation of better AI-powered translation tools for complex financial documents across tokenization schemes, which financial institutions could purchase.

Authors

Jingru Yang, Jinhang Wu, Minjing Shi, Shengfeng He, Xuyan Li, Man Tan, Biye Li

Abstract

Cross-tokenizer on-policy distillation (OPD) requires aligning teacher and student sequences before their scores can be compared. Decoded-text alignment provides such correspondence at the chunk level, but leaves a distinct problem unresolved: credit alignment. A chunk-level signal specifies how much a span should change, but not how that signal should be distributed across student tokens, while tokens without teacher correspondence remain unconstrained. We propose CAST (Chunk-aware credit Assignment with Selective Token Anchoring), a shift-or-stay framework that addresses both issues. For aligned chunks, CAST redistributes each chunk's advantage across student tokens using predictive uncertainty and local teacher-signal smoothness, emphasizing learning opportunities supported by locally consistent teacher feedback while preserving the chunk's total advantage. The resulting weights exactly solve a KL-regularized credit-allocation problem. For unaligned positions, CAST selectively anchors the student to its initial policy through a KL penalty, limiting drift where teacher supervision is unavailable. Because credit assignment is separated from chunk construction, CAST also applies to punctuation-defined or fixed-length chunks under shared tokenization. Across six mathematics, science, and code benchmarks, CAST improves the average over the strongest evaluated baseline by $2.45$ and $1.32$ points for cross-tokenizer transfer from Qwen3-8B to Mistral-7B and Llama-3.1-8B, respectively, and by $2.10$ points for shared-tokenizer transfer to Qwen3-1.7B.