New method tracks original model in ai fine tuning steps
Retrospective Distillation Attribution via Normalized Response Similarity
Artificial IntelligenceComputation and LanguageCryptography and SecurityMachine Learning
Summary
It can be hard to tell where a new AI model originally learned from, especially after it has been updated multiple times. The authors propose a way to check an AI’s past by looking only at its output text patterns, without needing access to the model’s inner details or earlier versions. Their method finds unique ‘signatures’ left by the original teaching model, even after many updates. This helps people audit AI models and understand their origins better.
What this means in practice
- •For ai compliance teams: Identify the original source model behind fine-tuned AI systems using only their text outputs for audit and regulatory purposes.
- •For machine learning engineers: Trace and verify how much influence initial teacher models retain through subsequent fine-tuning and optimization steps.
- •For language technology providers: Develop tools to certify and trace AI model lineage in commercial language products from output text alone.$Commercial implications: Enables commercial certification services for AI model provenance, improving trust and transparency in AI-powered applications.
Authors
Minwoo Jang, Jaechang Kim, Minhyeon Oh, Jeongyeon Hwang, Jungseul Ok
Abstract
Model distillation transfers capabilities through supervised fine-tuning (SFT) on teacher responses, often collected from commercial APIs, raising questions of model provenance. Existing distillation attribution methods have been largely evaluated on students immediately after the SFT step. However, a distilled model may undergo further SFT, preference optimization, or reinforcement learning before release, while an auditor may lack access to the pre-distillation checkpoint required by reference-based attribution. To close this gap, we propose SCOUT, an output-only method that aggregates recurring *syntactic patterns* into candidate profiles, filters low-contrast patterns, and calibrates student--candidate distances against inter-candidate distances. SCOUT supports attribution and abstention using only current texts, without model weights, token likelihoods, or historical checkpoints. Auditing publicly released descendants of distilled models spanning diverse post-training objectives, SCOUT consistently identifies the distillation source. Furthermore, tracing teacher-associated *syntactic signatures* along training trajectories reveals that they emerge during distillation and persist through subsequent preference optimization and reinforcement learning.