LLM checkpoint updates reveal harmful fine-tuning traces reliably

Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates

Machine LearningArtificial IntelligenceCryptography and Security

Summary

Fine-tuning large language models to produce harmful outputs leaves behind clear clues in the model's saved states, even without running the model. The authors found a way to measure these clues by comparing the model updates to reference examples of harmful and safe fine-tuning. This lets auditors detect harmful behavior without testing the model’s outputs directly, which is useful when evaluations are incomplete or unavailable. Their method, called TRACE, provides a continuous harmfulness score from only the model’s weights.

What this means in practice

  • For model safety auditors: Detect harmful fine-tuning in language models by analyzing checkpoint updates without needing to query the model or see the fine-tuning data.
  • For llm developers: Assess the impact of safety or utility fine-tuning on model parameters to better understand training effects on different model scales and configurations.

Authors

Ziqun Bao, Xinyu Zhang, Yuchen Shao, Chengcheng Wan

Abstract

Safety auditing of post-trained large language models typically relies on model behavior, requiring model execution and depending on the coverage of available evaluations. This work asks a different question: Do the target behaviors optimized during supervised fine-tuning (SFT) leave readable evidence directly in checkpoint updates? We find that harmful-compliance SFT induces a continuous, objective-dependent ordering in checkpoint-update space. Using a reference geometry defined by pure harmful-compliance, safety-targeted, and benign-utility SFT, we find that a checkpoint-level coordinate s_H tracks controlled harmful-objective composition with Spearman correlations of 0.986-0.992 across four 7-8B backbones, with the same ordering persisting at larger model scales. Matched compliance-versus-refusal controls show that this checkpoint trace reflects the SFT objective rather than harmful-input exposure, while additional controls rule out simple explanations based on harmful-example count or generic training intensity. Building on this structure, we introduce TRACE, a weights-only auditing method that localizes an unknown checkpoint update relative to frozen harmful and non-harmful reference prototypes and converts this geometry into a continuous harmful-objective score. TRACE requires neither model queries nor access to the unknown SFT data, and can be evaluated directly from checkpoint updates. Across distribution shifts, unseen data, different SFT configurations, partial checkpoint access, and LoRA/full-parameter fine-tuning, the trace remains stable and is positively associated with independently measured attack success rates. TRACE remains informative even at low harmful-objective proportions, providing a complementary auditing signal when behavioral evaluation is unavailable or incomplete. Code is available at https://anonymous.4open.science/r/Code4TRACE-54D3.