Orion tool detects software regressions in pull requests for Python projects
Behavioral Regression Detection for Pull Requests via Structured Oracles
Software Engineering
Summary
Software changes sometimes cause accidental errors called regressions that are hard to spot before merging code updates. The authors designed Orion, a tool that carefully checks code changes in pull requests by creating detailed tests that look for behavior changes. Orion uses smart analysis to understand which parts of the code might be affected and generates tests following rules derived from the change's intent. They tested Orion on a set of real Python code updates and found it better at finding regressions than other tools, showing it could help developers catch errors early.
What this means in practice
- •For python software developers: Screen pull requests for unintended behavior changes by generating targeted tests with strong behavioral oracles.
- •For continuous integration teams: Integrate Orion into CI/CD pipelines to detect potential regressions before code merges in Python projects.
Authors
Yiwen Fu, Binhang Qi, Xiao Li, Runlin Liu, Hailong Sun, Xiang Gao
Abstract
Software regressions are unintended behavioral changes introduced by code modifications and are hard to detect at pull-request (PR) level. Existing methods rely on incomplete tests or directly analyze PRs via LLMs, lacking structured behavioral reasoning and explicit oracles. We present ORION, a staged framework for PR-level regression detection. ORION extracts PR intent and identifies impacted functions via AST-based impact analysis, then synthesizes function-level structured behavioral oracles encoding preserved/changed behaviors, setup constraints and control-flow hints from PR intent, static risk signals and code changes. Guided by these oracles, ORION generates executable tests, conducts oracle-coverage-guided augmentation and static repair, and differentially runs tests on pre- and post-change code. It classifies PRs with deterministic rules and constrained LLM reasoning. We build RegBench-Py, the first Python PR regression benchmark with 154 verified real-world PRs. ORION achieves F1-scores of 0.6125 (DeepSeek-Chat-V3.2) and 0.5578 (Claude-4.5-haiku), outperforming baselines including Testora, Codex and Claude Code. Compared with PatchGuru, ORION supports broader multi-function PRs while remaining competitive on single-function cases. On 1,459 merged PRs, ORION finds 9 reproducible regression candidates, 8 confirmed by maintainers. It takes ~15 minutes per PR, showing practicality for CI/CD screening.