Policylift improves complex action choices from observational data
Generative Policy Improvement from Observational Data
Artificial IntelligenceMachine Learning
Summary
Making better decisions when actions are complicated like texts or pictures is hard because it’s tough to evaluate and change existing decision methods. The authors introduce PolicyLift, which first simplifies the complex actions into easier codes, then learns a better way to choose these codes, and finally turns the codes back into full actions. They prove that improving choices on these codes improves overall decisions, and show results that PolicyLift consistently makes better text actions in experiments.
What this means in practice
- •For customer service teams: Enhance automated text responses in customer support by improving policies from past conversation logs without costly retraining.
- •For content generation developers: Improve automated content creation tools by learning better action strategies from historical text outputs, enabling more effective text generation.
Authors
Jonas Schweisthal, Dennis Frauen, Stefan Feuerriegel, Emma Brunskill
Abstract
We study how to improve decision policies from observational data to generate better actions when actions are complex, structured objects, such as text messages or images. This task is subject to two major challenges: (i) action-level overlap is often too weak for reliable off-policy evaluation and improvement, and (ii) generating new structured actions often requires expensive training or fine-tuning (e.g., of an LLM) and can be ill-defined due to information loss in action representations. To address both, we propose PolicyLift, a three-stage framework for generative policy improvement with structured actions from observational data. In Stage 1, PolicyLift learns a small set of discrete action codes that capture high-level strategies in the observed action space and are tailored to support policy improvement under limited overlap. In Stage 2, it learns a context-dependent policy over these action codes. At deployment time, Stage 3 "lifts" the learned code-level policy back to the original action space by generating actions from the historical behavior system in a code-conditioned way. We show theoretically that improving the policy over action codes guarantees improvement of the induced policy in the original action space. We demonstrate PolicyLift across experiments in synthetic and real-world text generation tasks with consistent policy improvements over the behavior policy.