Papers for

automated legal technology teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Bridged self-distillation improves learning in smaller ai models

Bridged Self-Distillation for On-Policy Learning

Abstract: On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student. Existing methods handle these settings through different forms of supervision. To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces. The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion. The student generates rollouts without this context, while a copy of the student conditioned on it, serving as the self-teacher, scores the same rollouts in the student's own token space. BSD uses the resulting token-level guidance to adjust updates on unsuccessful rollouts, with a verifier providing the outcome signal. As the external teacher is queried only to produce text, the same training procedure applies to all teacher scenarios: shared-tokenizer, cross-tokenizer, and closed API teachers. Across these scenarios, BSD improves over the base model and outperforms baselines tailored to each scenario, offering a novel and unified pathway for distillation. Code is available: https://github.com/TUMLegalTech/Bridged-Self-Distillation

Mon 19 OctComputation and Language
The gist
Smaller AI models often struggle to learn from larger, more capable models because they can’t copy complex behavior or probabilities easily. The authors introduce Bridged Self-Distillation (BSD), a new way for smaller models to learn from larger teachers by using intermediate reasoning steps as guidance. The smaller model creates its own attempts, and a version of itself that has access to the teacher’s reasoning scores these attempts, helping it learn better. This approach works even when the teacher and student use different systems or when some probabilities aren’t available, making distillation more flexible and effective.
Open → 2610.26397v1