Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards
2026-07-11 • Machine Learning
Machine LearningArtificial IntelligenceComputational Engineering, Finance, and Science
AI summaryⓘ
The authors address the challenge of automatically generating code that solves complex math problems called partial differential equations (PDEs), which are important in science and engineering. They introduce a new learning method, RLVP, that improves code generation by using rewards not just for correct program execution but also for how accurate the solutions are compared to the true physics. This method trains a single model on many different types of PDEs and helps it create better solvers even for new, unseen problems. Their approach outperforms traditional methods and shows the model can mix and match learned techniques to solve new equations.
Partial Differential EquationsNumerical SolversReinforcement LearningLarge Language ModelsProgram VerificationPhysics-based RewardsHyperbolic PDEsParabolic PDEsElliptic PDEsZero-shot Learning
Authors
Pengfei Cai, Utkarsh Utkarsh, Alan Edelman, Christopher Vincent Rackauckas, Rafael Gomez-Bombarelli
Abstract
Partial differential equations (PDEs) are foundational to modeling in science and engineering, but constructing reliable numerical solvers remains labor-intensive, demanding expert knowledge of discretization schemes, stability conditions, and boundary treatments. Recent work has begun to frame PDE solving as a code-generation task for large language models (LLMs), yet existing approaches operate primarily at inference time: relying on prompting, debugging, self-refinement, and test-time scaling rather than adapting the model itself. In parallel, reinforcement learning with verifiable rewards has emerged as a post-training paradigm for code and math reasoning, but its verifiers are typically binary: a compiler runs, or a test passes. Such signals discard the graded structure of scientific correctness, where two solvers may both execute and yet differ in solution accuracy by orders of magnitude. In this work, we introduce RLVP: Reinforcement Learning with Verifiable Physics, an RL post-training framework for multi-PDE solver code generation. RLVP addresses this verifiability gap with a hybrid verifier: hard program-validity checks ensure executability, while continuous physics rewards score function-space accuracy and PDE-residual consistency. A single policy is post-trained across diverse PDE families spanning hyperbolic, parabolic, elliptic, and incompressible-flow systems. RLVP improves over both pre-trained and supervised-only baselines on PDE benchmarks, and shows zero-shot improvement transfer to held-out PDEs. We show that a smaller LLM post-trained with RLVP can outperform prompting a frontier model on in-distribution PDE solver generation. The trained policy shows evidence of compositionality in numerical motifs: it recombines stencils, time-stepping schemes, and boundary-handling primitives learned from the PDEs used in training into generated solvers for unseen PDE problems.