Language models identify code style not author identity in zero-shot tests
Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
Artificial IntelligenceComputation and LanguageSoftware Engineering
Summary
Sometimes computer programs try to guess if a piece of code was written by themselves or another program. This study found that large language models don’t really recognize their own code but instead rely on surface details like length and style. When these obvious clues are removed, the models perform no better than guessing. The authors suggest using balanced accuracy and simple baseline tests to report these results more clearly.
What this means in practice
- •For software quality teams: Assess the reliability of AI code review tools by understanding when models rely on superficial code features rather than true authorship recognition.
- •For software security teams: Evaluate risks of collusion between AI models in code generation by testing if models can detect their own code beyond surface cues.
Authors
Ehsan Barkhordar, Surendrabikram Thapa
Abstract
If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act as evaluators in four tasks: picking their own solution from a pair, judging whether a single solution is their own, identifying which of two solutions a named model wrote, and judging quality blind. In the single-solution task, balanced accuracy is 49-58% for all 15 model-benchmark combinations, while raw accuracy (38-67%) mostly reflects how readily a model claims authorship. In the pairwise task, accuracy across 14 evaluator-opponent combinations correlates at r=0.93 with how often the evaluator's solution is longer. Attribution to a named model succeeds on some pairs and is consistently inverted on others. A rule-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance; the other two follow a length difference it leaves, although a trained classifier still separates most normalized pairs. Claude Haiku's self-preference also disappears. We recommend reporting balanced accuracy, heuristic baselines, and label consistency.