Task format influences fine-tuning success in prompt-safety models
Task Format as a Factor in Adaptation: A Controlled Comparison of LoRA and Full Fine-Tuning for Prompt-Safety Classification
Computation and Language
Summary
Fine-tuning language models to classify whether prompts are safe depends on various choices, including how the task is formatted. The authors compared two fine-tuning methods, LoRA and full fine-tuning, on the same model and found that full fine-tuning usually performs a bit better, but this can change with different training setups. Having the model explain its decisions improved LoRA's performance. They also showed that good accuracy on familiar data doesn’t guarantee strong performance on harder, out-of-domain examples or adversarial cases.
What this means in practice
- •For machine learning engineers: Choose task format and training setup to optimize prompt-safety classifiers for deployed models balancing accuracy and robustness.
- •For security teams in ai companies: Develop safer AI systems by understanding trade-offs in tuning models for detecting unsafe prompts, especially against adversarial inputs.
Authors
John Alderete, Srirakshith Betageri, Jennifer Flynn, Alex Greenhalgh, Josh Rubin, Yoshita Varma Annam
Abstract
Fine-tuning a language model for classification involves many design decisions. Comparisons of low-rank adaptation (LoRA) with full fine-tuning are common but often confounded, and task format is rarely studied. We compare both regimes on prompt-safety classification with a matched 400M encoder-decoder pair (Ettin) across three task formats, with up to 15 seeds per cell and pre-specified analyses. At the pre-specified learning rates, full fine-tuning leads LoRA by 2-3 pp in every architecture and format trained on the label alone. On the encoder, the gap closes once both rates are chosen by the same search. Training LoRA to write a reason after its verdict adds 4.0 pp and closes the gap, but a content-free sentence of equal length does not. Trained models score above a prompted 8B in domain but fall behind it on two of three out-of-distribution corpora, trailing by 8-23 pp on adversarial jailbreaks and flagging 27-66% of benign but harmful-looking prompts as unsafe, against the 8B's 2-3%. Together, the results suggest that the adaptation regime matters less than how each model is trained, its learning rate, and training target, and that in-domain accuracy alone is an incomplete guide to robustness.