Evolution strategies move models toward reward-weighted centers
Center-of-Mass Interpretation of Evolution Strategies
Artificial IntelligenceComputation and Language
Summary
Evolution strategies are a way to improve models by trying many small changes at once and then moving towards the changes that worked better. The authors explain this process by imagining each model guess as a particle with weight based on how well it did. The model then moves toward the center of mass formed by these weighted guesses, making updates clearer and easier to visualize. This view works exactly for any number of samples and helps understand why subtracting the average reward changes the update direction.
What this means in practice
- •For machine learning engineers: Visualize and confirm model update steps in evolutionary algorithms by interpreting reward-weighted shifts geometrically.
- •For reinforcement learning developers: Design clearer reward-based update mechanisms by using the center-of-mass view to understand the impact of different reward transformations.
A theory result. No direct application yet.
Authors
Akshat Gupta
Abstract
Evolution strategies (ES) update a model by sampling a population of perturbed copies, scoring each one, and moving in the reward-weighted direction of the perturbations. This update is usually presented as an estimate of a smoothed gradient. We present a geometrical interpretation of this update instead. Treat each perturbed model as a particle and its reward as the particle's mass. Before scoring, every particle has the same mass and the population balances at its centroid. Once rewards are assigned, it balances at its reward-weighted center of mass instead. We show that the mean-subtracted ES update is exactly this displacement, scaled by a scalar factor that depends on the mean reward, for any finite population. The physical reading needs nonnegative rewards, which can always be arranged by adding a constant to every reward without changing the update. In the limit of a large population or under antithetic sampling, the centroid sits on the model itself, so ES has an even simpler geometric interpretation: from its current location, the model moves towards the reward-weighted center of mass of its own population. The same picture explains what subtracting the mean reward does: it changes the point from which the displacement is measured, from the model to the sampled centroid. Our contribution is an intuitive, geometric way to visualize an ES update, exact for any finite population.