Inference-time scaling improves generation despite imperfect verifiers
Imperfect Verifiers, Exact Targets: Inference-Time Scaling for Continuous Generation
Machine Learning
Summary
Generating good outputs in AI models can be helped by checking intermediate steps, but these checks are often imperfect and can mislead the process. The authors propose a method called Continuous-State Verifier-Guided Backtracking (CS-VGB) that repeatedly refines generated results by going back and forth during generation. This approach guarantees that, despite imperfect checking, the final outputs match desired qualities better and converge efficiently. They tested this on image generation and robot planning tasks, showing consistent improvements as more computation is used.
What this means in practice
- •For image generation teams: Improve image generation quality by guiding sampling toward better-rated outputs despite imperfect intermediate scoring.
- •For robotics planners: Enhance multi-robot plan success by refining sampled plans with repeated backtracking and verification.
Authors
Kijung Jeon, Molei Tao
Abstract
Verifier-guided inference-time scaling uses additional computation to score intermediate states and steer generation toward better outputs. However, imperfect intermediate verifiers can distort sampling, and repeated refinement alone does not ensure recovery of the desired reward-tilted distribution. We introduce Continuous-State Verifier-Guided Backtracking (CS-VGB), an inference-time scaling method for sampling from reward-tilted distributions with continuous generative models through repeated denoising and renoising. With correctly paired denoising and renoising transitions, the stationary distribution of CS-VGB at the clean checkpoint equals the desired reward-tilted target, even when intermediate verifiers are inaccurate. Under stated regularity conditions, CS-VGB converges exponentially to the target despite verifier error, and the expected number of updates needed to reach a clean sample grows only linearly with the number of checkpoints. In analytically tractable settings, we empirically demonstrate that CS-VGB recovers the desired reward-tilted distribution despite verifier error, whereas the compared verifier-based sampling methods fail to recover the target under the same verifier errors. We also instantiate the framework with diffusion models and one-step stochastic posterior maps. Across prompt-guided ImageNet generation, facial-attribute generation on CelebA, and multi-robot planning, recurrent CS-VGB search improves reward and task success as inference budgets grow, with image-generation rewards continuing beyond the observed plateaus of denoising-only baselines.