Observation-safe coupling reduces noise in imperfect-information learning
OSCC: Certified Observation-Safe Coupling Optimization for Gradient-Noise Control in Imperfect-Information Learning
Artificial IntelligenceComputer Science and Game TheoryMachine Learning
Summary
Learning to make decisions when you don’t have all the information is tricky because it’s hard to compare what might have happened if you made different choices. The authors identify problems with usual ways to simulate alternatives and introduce observation-safe counterfactual coupling (OSCC), a method that carefully links these alternative scenarios without revealing hidden details or mixing up random events. This approach lowers the noise in estimating how good different choices are, helping the learning process be more stable and reliable.
What this means in practice
- •For game ai developers: Improve learning stability for AI strategies in games with hidden information by safely coupling rollout simulations to reduce noise.
- •For reinforcement learning engineers: Enhance multi-action policy optimization by selecting rollout coupling methods that lower gradient noise and computational cost effectively.
Authors
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Rui Chen, Daren Zha, Jun Xiao
Abstract
Coupled rollouts can reduce the noise of counterfactual action comparisons, but two issues prevent standard common-random-number constructions from serving as a general learning primitive in imperfect-information environments. First, an invalid coupling may expose hidden state, synchronize endogenous policy randomness, or misalign chance events after counterfactual histories diverge. Second, in multi-action policy optimization, lower return-contrast variance is not by itself the relevant objective: the optimizer depends on the return covariance matrix after projection through the local policy-gradient geometry. We introduce observation-safe counterfactual coupling (OSCC), a framework that defines an admissible class through marginal preservation, information-state safety, branch-local policy randomness, semantic event alignment, and trace-before-oracle replay. We derive a gradient-aware coupling criterion showing that, for marginal-preserving couplings, policy-gradient noise changes are determined by policy-Jacobian-weighted off-diagonal return covariance. This motivates OSCC-Select, a calibration-only selector that chooses among independent, root-only, continuation-only, and fully coupled rollouts using separate safety and gain certificates. Its gain target combines projected gradient noise with measured physical sampling cost and falls back to independent sampling whenever a simultaneous lower confidence bound does not certify improvement. On 100,000 fixed-root Leduc comparisons, the fully coupled CP-GRPO instantiation reduces return-contrast variance from 41.1158 to 18.1441, a 55.87% reduction, while preserving the declared branch marginals. With three actions, OSCC-Select chooses continuation coupling and attains gradient-noise trace 0.0783 versus 0.0917 for return-variance selection. Increasing calibration from 64 to 2,048 groups raises certification from 0.327 to 0.995.