Modernizing legacy PHP code with AI often breaks working programs
When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP
Software Engineering
Summary
The authors studied whether using large language models (LLMs) to update old PHP code works better than leaving the code unchanged. They found that these AI models often broke code that was working fine before, while fixing very few problems. A rule-based tool called Rector caused fewer errors and fixed a few more issues than the AI models. The researchers suggest that evaluations should always compare against not changing the code, and separately measure updates and safety.
What this means in practice
- •For php developers: Decide when not to rely on large language models for updating legacy PHP files to avoid introducing new bugs.
- •For software maintenance teams: Evaluate legacy modernization tools by comparing outputs to unchanged code and testing behavior preservation separately.
Authors
Afridi Mahmud, Abid Hasan Khondaker, Sadif Ahmed, Md Nafiu Rahman, Rifat Shahriyar
Abstract
Pasting a legacy file into an LLM and asking it to "modernize this" is an easy shortcut. But is the result better than leaving the code alone? In this paper, we study this single-file setting with five LLMs, each asked to modernize 100 WordPress 4.0 PHP files to PHP 8.3 while preserving behavior, with no project context or execution feedback. We evaluate every output with 630 hidden behavioral checks and 511 Rector-detected modernization findings. We compare against two baselines: the unchanged originals and the rule-based tool Rector. Often, the answer is no. Untouched, 83 files already pass every check; the best LLM leaves only 59. Each model breaks 32-51 working files while repairing at most 10, whereas Rector breaks 7 and repairs 6. Most of this damage is unnecessary. Only 37 findings (7 percent) require a change under PHP 8.3, yet in compatible files with no required change, the LLMs break 36-60 percent (Rector: 8 percent). The model with the highest modernization score is no more compatible than its closest competitors. Most regressed checks come from files that no longer load at all, often because of type declarations PHP itself rejects. These are errors that need no project context to avoid. Evaluations of LLM-based modernization should therefore measure against a no-change baseline and report modernization and behavioral preservation separately.