Generalization gaps in reinforcement learning depend on action rules and baseline scores

What Does a ProcGen Generalization Gap Measure? Action Rules, Convergence, and the Missing Random Floor

Machine LearningArtificial Intelligence

Summary

When evaluating how well AI agents trained on some video game levels perform on new unseen levels, there is usually a difference called a generalization gap. This paper points out that to understand this gap properly, you need to compare agent scores not only to training scores but also to how a completely random player would perform on those levels. The authors show that how you pick the agent’s actions during testing (random sampling versus always choosing the best action) changes the measured performance a lot. They also find that measuring randomness in the agent’s choices is unclear because some different actions have the same effect. These insights suggest ways to better measure and report AI generalization in such games.

What this means in practice

  • For reinforcement learning engineers: Use appropriate reference baselines and specified action selection during testing to obtain more accurate measures of agent generalization on procedural generation benchmarks.
  • For machine learning experiment managers: Improve evaluation protocols by reporting random policy baselines and clearly stating test-time action rules to avoid misleading conclusions about agent performance.

Authors

Abhisek Keshari

Abstract

A generalization gap in reinforcement learning (return on training levels minus return on held-out levels) is usually reported without a reference point. We argue that the missing reference is a measured random floor: the return of a uniform-random policy on the same levels under the same harness. On ProcGen, the floor changes what several standard numbers mean. On identical checkpoints and levels across eight environments, switching between sampled and greedy (argmax) test-time actions moves held-out return in both directions, and greedy evaluation takes three environments to or below the floor: in miner, the sampled policy scores 4.9x the floor on held-out levels while its argmax scores below it. Raw policy entropy places seven of eight environments short of convergence, but 35-65% of that entropy is spread across actions with identical effects; after merging them, one to three remain short, and against the floor only heist has learned nothing that transfers. An audit of twelve prior ProcGen codebases finds that all eleven with held-out evaluation sample test-time actions for their policy-gradient agents, nine by default rather than explicit choice, and six report running in-loop averages rather than evaluating a fixed checkpoint. Applied to our own case study, the same checks grade down a statistically significant encoder effect and rule out a within-encoder train-vs-test CKA statistic. We recommend that every reported gap state its action rule, use a matched and seeded protocol, and report the random floor on both level sets.
View PDFOpen arXiv