Visual reinforcement learning gains may not improve with more test-time sampling
Do Visual RL Gains Scale at Test Time? A Quantile View of Post-Training
Machine LearningComputer Vision and Pattern Recognition
Summary
Sometimes, teaching a computer to create images using trial and error (reinforcement learning) makes single pictures look better. But this paper finds that when you let the computer pick the best picture out of many tries (test-time scaling), the improvements don’t always hold up and can even get worse. The authors explain this by looking at how rare, very good images change after training. They propose a new method to keep the best images good while still improving typical ones, making the overall results more reliable for harder tasks.
What this means in practice
- •For visual ai developers: Improve image generation quality over multiple inference samplings by optimizing models to maintain high-quality rare outputs during test-time scaling.
- •For content creation teams: Ensure generative tools create consistently high-quality images across diverse and difficult prompts by applying quantile-based post-training optimization.
Authors
Austin Wang, Lexing Ying
Abstract
Reinforcement learning (RL) post-training improves the single-sample quality of visual generative models, but deployment often provides additional inference compute through Best-of-N sampling. Whether RL gains transfer under such test-time scaling, particularly on difficult held-out tasks, remains poorly understood in the visual domain. Our central contribution is to systematically identify and characterize a failure mode of scaling transfer in visual RL: across multiple RL methods, model backbones, and evaluations spanning compositional generation, text rendering, and preference alignment, higher single-sample reward does not necessarily imply a better quality-compute frontier. RL gains may attenuate, saturate, or even reverse at large sampling budgets, with the base model eventually outperforming its post-trained counterpart. We explain these regimes through a quantile-transfer framework, showing that Best-of-N places increasing weight on the upper tail of the reward distribution as N grows. Consequently, RL can improve typical samples while compressing or degrading the rare high-quality outcomes that test-time scaling exploits. As a simple intervention motivated by this analysis, we introduce Quantile Utility Policy Optimization (QUPO), which optimizes a controllable mixture of quantile utilities corresponding to different inference budgets. Experiments on SD3.5-M and BAGEL show that QUPO better preserves upper-tail quality and substantially improves scaling on challenging held-out prompts relative to standard RL post-training. Our results suggest that visual generative models should be evaluated and optimized not only for single-sample quality, but for how post-training reshapes the full reward distribution that inference-time scaling exploits.