ProofPatch keeps Lean proof checks valid as statements evolve

ProofPatch: Reviewing Lean Statements and Keeping the Reviews Valid as Formalizations Evolve

Logic in Computer ScienceArtificial IntelligenceSoftware Engineering

Summary

Checking that a proof matches its statement is done automatically by Lean, but deciding if the statement says what was really meant is a job for humans. This becomes harder when lots of statements are written or edited automatically by AI. The authors present ProofPatch, a tool that helps reviewers ask detailed questions about Lean statements and stores their answers in a way that stays relevant as the statements or definitions change. ProofPatch tracks exactly what changed and blocks edits that would invalidate reviewers' decisions, making reviews more reliable with evolving formal math libraries.

What this means in practice

  • For formal verification teams: Maintain validity of proof statement reviews across evolving formal developments using ProofPatch’s targeted question and patch blocking features.
  • For automated code reviewers: Use ProofPatch’s approach to keep automated review feedback consistent despite changes to code or specification definitions.

Authors

Xujun Che, Depeng Xu, Jian Xiang, Taufiqua Khan

Abstract

The kernel of the Lean proof assistant checks that a proof establishes its statement. Whether the statement says what was intended is left to human reviewers. This is the old problem of validating a specification, and it grows more pressing as language models write formal statements in bulk and agents edit formal developments continuously. We present ProofPatch, a tool that helps reviewers judge Lean statements and keeps their judgments valid while the development changes. A reviewer can ask the kernel targeted questions about a statement, such as whether a hypothesis is needed or whether a counterexample exists. Each decision is stored with fingerprints of the statement as Lean elaborated it and of the definitions it depends on, and any declaration can be pinned so that a patch changing it is blocked, whether a person or an agent built on a large language model (LLM) wrote the patch. Working on elaborated terms rather than source text is the central design choice: notation and formatting changes leave a decision intact, and a changed definition reaches every theorem that uses it. On 229 pairs of consecutive revisions of Mathlib, Lean's mathematics library, ProofPatch sends 118 of 12,898 synthetic decisions back for review. Every text-based alternative misses at least 38 of these or sends back at least 13 times as many. On 1,527 labeled statements, kernel evidence flags unfaithful ones with precision 0.88 to 0.96, and in 344 runs of an LLM agent no change to a pinned statement got through.