Round-Trip Mutation Testing: Translating Code to Natural Language Intent and back
2026-07-03 • Software Engineering
Software Engineering
AI summaryⓘ
The authors introduce Round-Trip Mutation Testing (RTM), a new method that creates slight program changes (mutants) by translating code to natural language and back using large language models (LLMs). These mutants simulate common coding mistakes caused by misunderstandings of the program's intended behavior. They tested RTM on real buggy code and found it creates more varied and meaningful mutants than traditional methods, helping design smaller but stronger test sets that catch more bugs. This suggests RTM could improve how software testing finds faults related to specification errors.
Mutation TestingLarge Language Models (LLMs)Program IntentSoftware TestingMutantsSemantic DifferencesBug DetectionSpecification MisunderstandingsTest SuitesCode Translation
Authors
Asma Hamidi, Cedric Richter, Ahmed Khanfir, Mike Papadakis
Abstract
This paper presents Round-Trip Mutation Testing (RTM), a novel approach that generates mutants from LLM mistranslations between a program code and its intent. Leveraging the generative capability of LLMs from and to programming and natural language, and given an input program, our approach predicts its intent, that is used to generate programs, which when different from the original one, constitute the output mutants. The approach produces additionally mutants, stemming from artificially provoked mistranslations, by mutating the intent prior to the final programs (mutants) generation. Originating from the propagation of small changes in the intent to the code, our intuition is that these programs would present subtle semantic differences from the original one, simulating likely-to-occur faults that could result from specification misunderstandings, and enabling mutation testing. To evaluate RTM, we run it on 40 real buggy methods and evaluate its effectiveness and cost-efficiency in guiding testing towards detecting the bugs. Our results demonstrate the potential of round-trip mutation testing to produce syntactically more diverse mutants, potentially exposing faults that traditional mutation operators fail to reveal. More interestingly, RTM outperforms traditional pattern-based mutation in producing smaller and stronger test-suites, detecting on average over 4 and 1.7 times more faults when selecting only 4 and 30 tests respectively.