Follow the Mean: Reference-Guided Flow Matching
2026-05-11 • Machine Learning
Machine Learning
AI summaryⓘ
The authors propose a new way to control image generation models by changing the examples the model uses, instead of adjusting model parameters or extra networks. They show that by shifting the average reference points (called the conditional endpoint mean), they can steer the generated outputs without retraining the model. They introduce two methods: one that works without training by adjusting a frozen model using a reference set, and another that learns a small correction to better match quality while still allowing example swapping. This suggests generative models can adapt flexibly using data examples rather than updating their internal parameters.
controllable generationflow matchingconditional endpoint meanreference setfrozen modelzero-shot adaptationdata-driven controlsemi-parametric guidancegenerative modelsimage synthesis
Authors
Pedro M. P. Curvo, Maksim Zhdanov, Floor Eijkelboom, Jan-Willem van de Meent
Abstract
Existing approaches to controllable generation typically rely on fine-tuning, auxiliary networks, or test-time search. We show that flow matching admits a different control interface: adaptation through examples. For deterministic interpolants, the velocity field is solely governed by a conditional endpoint mean; shifting this mean shifts the flow itself. This yields a simple principle for controllable generation: steer a pretrained model by changing the reference set it follows. We instantiate this idea in two forms. Reference-Mean Guidance is training-free: it computes a closed-form endpoint-mean correction from a reference bank and applies it to a frozen FLUX.2-klein (4B) model, enabling control of color, identity, style, and structure while keeping the prompt, seed, and weights fixed. Semi-Parametric Guidance amortizes the same idea through an explicit mean anchor and learned residual refiner, matching unconditional DiT-B/4 quality on AFHQv2 while allowing the reference set to be swapped at inference time. These results point to a broader direction: generative models that adapt through data, not parameter updates.