Papers for

automated code reviewers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

ProofPatch keeps Lean proof checks valid as statements evolve

ProofPatch: Reviewing Lean Statements and Keeping the Reviews Valid as Formalizations Evolve

Abstract: The kernel of the Lean proof assistant checks that a proof establishes its statement. Whether the statement says what was intended is left to human reviewers. This is the old problem of validating a specification, and it grows more pressing as language models write formal statements in bulk and agents edit formal developments continuously. We present ProofPatch, a tool that helps reviewers judge Lean statements and keeps their judgments valid while the development changes. A reviewer can ask the kernel targeted questions about a statement, such as whether a hypothesis is needed or whether a counterexample exists. Each decision is stored with fingerprints of the statement as Lean elaborated it and of the definitions it depends on, and any declaration can be pinned so that a patch changing it is blocked, whether a person or an agent built on a large language model (LLM) wrote the patch. Working on elaborated terms rather than source text is the central design choice: notation and formatting changes leave a decision intact, and a changed definition reaches every theorem that uses it. On 229 pairs of consecutive revisions of Mathlib, Lean's mathematics library, ProofPatch sends 118 of 12,898 synthetic decisions back for review. Every text-based alternative misses at least 38 of these or sends back at least 13 times as many. On 1,527 labeled statements, kernel evidence flags unfaithful ones with precision 0.88 to 0.96, and in 344 runs of an LLM agent no change to a pinned statement got through.

Sat 31 OctLogic in Computer ScienceArtificial IntelligenceSoftware Engineering
The gist
Checking that a proof matches its statement is done automatically by Lean, but deciding if the statement says what was really meant is a job for humans. This becomes harder when lots of statements are written or edited automatically by AI. The authors present ProofPatch, a tool that helps reviewers ask detailed questions about Lean statements and stores their answers in a way that stays relevant as the statements or definitions change. ProofPatch tracks exactly what changed and blocks edits that would invalidate reviewers' decisions, making reviews more reliable with evolving formal math libraries.
Open → 2611.01357v1

PyINE framework helps check AI reasoning using verifiable code execution

PyINE: A Framework for Scalable Elicitation and Oversight via Code Execution

Abstract: Reasoning models can remain capable of solving a task while still defaulting to cheaper but misleading shortcuts. This creates a central oversight problem: when a model gives an answer with plausible but incomplete reasoning, can an overseer determine whether that output should be trusted? To study this problem, we introduce PyINE, a framework for scalable elicitation and oversight using instrumented Python programs as a verifiable execution substrate. In PyINE, programs define task environments, execution traces provide authoritative labels for outcomes and intermediate facts, and task variants can be generated mechanically rather than through static human annotation. We instantiate the framework in PyINE-v1, a first release built from nearly one million deterministic execution traces and over 500,000 matched LLM-generated code variants used for counterfactual evaluation. Using standard RL with verifiable rewards on cue-varied tasks, we train a shortcut-following model that improves substantially at predicting execution outcomes while still making systematic errors when misleading human-facing cues conflict with the program's realized behavior. We then evaluate activation probes, trained text classifiers, prompted judges, and a lightweight debate protocol as overseers of this model. We find that performance pooled at the dataset level can hide weak coverage of the failures that matter most: cheap learned overseers often miss rare shortcut-driven errors, while stronger model-based checks are more balanced but substantially costlier and harder to turn into reliable thresholded decisions. PyINE-v1 turns this failure-mode coverage problem into a reusable experimental setting for developing oversight methods that are verifiable, failure-mode-aware, and cost-sensitive.

Sat 3 OctArtificial IntelligenceMachine Learning
The gist
Sometimes AI models give answers that seem correct but rely on misleading shortcuts. The authors created PyINE, a system that uses Python programs to run tasks and record exact step-by-step outcomes. This lets overseers verify if AI reasoning matches the true process, not just the final result. They tested different ways to spot when AI takes shortcuts, finding that simpler checks often miss important errors while stronger checks are costly. PyINE provides a setup to develop and test better methods for watching over AI reasoning.
Open → 2610.04737v1

Adaptive search branches improve agent code optimization by up to 35 percent

Mixture of Self-Improving Branches For Agent Harness Optimization

Abstract: Harness optimization provides a practical setting for recursive self-improvement (RSI), where agent-generated modifications inform subsequent changes through execution feedback. Recent work such as Meta-Harness implements this process through iterative code generation and evaluation, but retains a fixed development set and proposal policy. These constraints channel evolution along a single search trajectory, increasing the risk of converging to a local optimum. We make the improvement process itself adaptive by organizing search into branches with evolving development subsets and proposal policies. Each branch retains development cases solved by more of its leading harnesses than by those of other branches, drops cases solved by every leading harness across all branches, and revises its proposal policy using its own search history. To deploy the resulting complementary harnesses, we propose a router to select one development-selected branch head for each new input before execution. Across mathematical reasoning and agentic coding benchmarks, our system achieves relative improvements over Meta-Harness of 34.8% on Olympiad-level mathematical reasoning, 11.6% on Terminal-Bench 2.0, and 3.8% on SWE-bench Lite, with harness selection and router configuration based solely on development data. These results show that evolving branch objectives and proposal policies can yield complementary harnesses whose strengths a router combines without access to test outcomes.

Tue 29 SeptArtificial Intelligence
The gist
When computer agents try to improve themselves by changing their own code, they can get stuck choosing only one way to improve. The authors found that by letting these agents split into different groups that each try different ideas using feedback from their own successes, they can explore better alternatives. A system decides which group's method to use for new problems, combining their strengths without needing to know the final test results. This approach improved agent performance on math and coding tasks compared to previous methods.
Open → 2609.37834v1

Reasoning models improve answer confidence with smooth calibration

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

Abstract: Reinforcement learning with verifiable rewards (RLVR) trains reasoning models to produce correct answers, but does not ensure that their stated confidence is calibrated. The resulting models are systematically overconfident. Recent methods train calibration inside the RLVR loop by having the model state a numerical confidence alongside its answer, but they all obtain the confidence by sampling it as text. This choice imposes two costs: a sampled confidence introduces variance and in practice collapses to a handful of distinct values, and sampling makes the confidence non-differentiable, forcing the calibration loss through a scalar reward. We propose CREDO (Confidence REaDOut) to replace sampling with a deterministic readout. While RLVR optimizes correctness, CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression. CREDO further turns the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together. Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, and the gains extend to abstention and selective prediction.

Mon 28 SeptMachine LearningArtificial IntelligenceComputation and Language
The gist
Many AI systems that solve problems say how sure they are about their answers, but they often claim to be more confident than they should be. The authors show that current methods which guess confidence as text make the system's certainty jump around and hard to train properly. They created a new method called CREDO that picks confidence values directly from the AI's internal signals in a smooth way, leading to better and more reliable confidence scores. This helps the system know when it might be wrong and when to skip answering, improving overall performance.
Open → 2609.34857v1