Fenchel tilting enables efficient finetuning of generative models
Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Models
Machine LearningArtificial Intelligence
Summary
Fine-tuning pretrained generative models for specific goals can be challenging and often involves slow or limited methods. The authors propose Fenchel Tilt Flow Control (FTFC), a new technique that separates the process of optimizing preferences from adjusting the model itself, making it more efficient. FTFC works by assigning weights to generated samples based on how well they match desired outcomes, then uses these weights to update the model in one step without complex calculations. This approach allows adapting models to a wide range of goals faster and more accurately than previous ways.
What this means in practice
- •For ai model engineers: Update pretrained generative models efficiently for varied design goals without extensive retraining.
- •For drug discovery teams: Tailor molecular generation models quickly to prioritize specific chemical properties using weighted sample corrections.
Authors
Maksim Bobrin, Maksim Zhdanov, Dmitry Dylov
Abstract
Adapting a pretrained generative model to an arbitrary preference expressed as a utility function underlies reward alignment, guided design, and constraint satisfaction, enabling diverse applications. Existing fine-tuning methods trade off generality against computational cost: they either restrict the family class of supported preferences to keep optimization simple or preserve generality at the expense of efficiency. We introduce Fenchel Tilt Flow Control (FTFC), which decouples utility optimization from generative-model fitting. FTFC first optimizes for a target distribution by jointly fitting an effective reward and density-ratio weights on pretrained samples. Method combines the utility's variational structure with Fenchel duality, supporting general $f$-divergence penalties that determine how rewards are transformed into an distribution-correction weights. These weights are then frozen and used to modify a diffusion or flow model in a single stage of importance-weighted denoising or flow matching, without differentiating through sampling trajectories. We establish exact duality for concave utilities under suitable conditions and show that weighted fitting reproduces the optimal target distribution for a given utility. Across image and molecule generation benchmarks, FTFC improves over baselines on diverse preference functions, while also being up to $20\times$ more efficient. roposed method enables adaptation beyond expected-reward maximization without complex optimization, while preserving robustness for more general class of the utility functions compared to baselines.