Removing harmful information does not guarantee model tamper resistance
Removing Information Content Does Not Certify Tamper Resistance in Open-Weight Models
Machine LearningCryptography and Security
Summary
People want to make machine learning models safer by removing sensitive or harmful information from them before sharing. This paper shows that just taking out such information from a model’s weights doesn’t always stop an attacker from quickly changing the model through fine-tuning. The authors explain that how a model’s parameters are structured and the order of training steps can let attackers bypass these protections, even when the apparent sensitive information is zero. This means new ways are needed to really certify that models resist tampering beyond just looking at the information inside them.
What this means in practice
- •For machine learning engineers: Evaluate model vulnerability to fine-tuning attacks beyond information content to improve deployment security.
- •For ai security teams: Design tamper resistance strategies that address attack dynamics as well as data leakage in shared models.
Authors
Domenic Rosati, Alessa Carbo, Ali Dadsetan, Hong Huang, Matthew Young, Subhabrata Majumdar, Frank Rudzicz, Hassan Sajjad
Abstract
Does removing harmful information make open-weight models resistant to fine-tuning attacks? We show that mutual information at release alone cannot universally certify slow recovery. Function-preserving reparameterizations leave information unchanged while altering gradient-descent geometry, so an invariant certificate is bounded by the fastest reachable parameterization. We apply this principle to weight--data mutual information under training-data filtering and label--representation mutual information under capability removal. Training order can change recovery time at fixed weight--data information, while exact representation-level independence can preserve the entire parameter Jacobian. An explicit construction has both information quantities equal to zero and recovers in one gradient step. Controlled experiments illustrate order-dependent recovery and parameterization-dependent attack speed. These results identify the missing requirement for certification: constraints on attack dynamics beyond mutual information at release.