Approximate prediction limits under feedback in universal agents
Approximate Universal Prediction under Interaction
Machine Learning
Summary
Predicting future events in complex environments often uses idealized mathematical models called Bayesian mixtures. This paper studies what happens when an agent uses an approximate version of these models for planning actions, where the agent’s decisions influence the data it sees next. The authors analyze how errors in prediction can grow depending on factors like how rare the tasks and chosen actions are. They also provide guarantees on how prediction and value errors behave for such agents, clarifying conditions needed for building practical approximations of ideal universal predictors.
What this means in practice
- •For reinforcement learning engineers: Understand theoretical limitations on prediction error when planning with approximate models that influence future data in reinforcement learning systems.
- •For ai safety teams: Use bounds on value and prediction errors to evaluate risk in AI agents that learn from interactive environments with feedback loops.
A theory result. No direct application yet.
Authors
Richard Cornelius Suwandi
Abstract
Bayesian mixtures, and Solomonoff induction in particular, provide an idealized model of prediction learned by minimizing population log loss, and they underlie universal agents such as AIXI. An agent that plans with an approximation of such a predictor decides where that predictor is evaluated, and a planner can be drawn toward branches on which its model is optimistic. We study which approximation guarantees survive this feedback. For normalized finite-horizon causal mixtures, the transfer coefficient factors into a policy likelihood ratio and task evidence available before each prediction, so task rarity and action-branch rarity can compound. Within one fixed universal mixture, infinitely many computable environments admit excess prediction error of order $κη\,2^{K(μ)}$, where $η$ is the training KL budget and $κ$ measures policy coverage. This order is sharp, and for each target a computable predictor attains it at a suitable finite horizon, so computability alone does not remove the penalty. For bounded returns, value error is controlled by the probability of entering a poorly covered branch plus a square-root term in $η$. Both terms are necessary, and finite maximum coverage ratios are not required. Motivated by the mentor override of Golden Handcuffs, we derive margins for preserving deferral decisions and conditional KL budgets with an unavoidable inverse-history-probability penalty, and a counterexample separates agreement of deferral decisions from task-wise event avoidance. The guarantees apply to any predictor architecture that meets explicit population approximation and coverage conditions, which gives concrete targets for implementing idealized universal agents.