Principal agent testing expands to ongoing experiments with adaptive stopping
Principal-agent hypothesis testing under adaptive experimentation
Computer Science and Game Theory
Summary
This paper looks at how to fairly pay someone (an agent) who provides data to another person (a principal) for testing a claim over time, not just once. The authors extend existing methods to situations where data comes in continuously, and the agent can choose when to stop the process based on what they see. They identify contracts that prevent agents with no real effect from gaining by quitting early and find the best way for agents to decide when to stop to maximize rewards. They also propose simpler ways to do this requiring less detailed knowledge from the agent.
What this means in practice
- •For product testing teams: Design reward schemes for ongoing user experiments that discourage premature termination of tests.
- •For online marketplace operators: Create incentives for data providers that prevent gaming the system when rewards accumulate over time.
A theory result. No direct application yet.
Authors
Ben Chugg, Aritra Ghosh, Aaditya Ramdas
Abstract
Principal-agent hypothesis testing studies statistical contracts in which a principal rewards an agent according to data the agent generates. Existing work focuses on the single-shot setting, where the agent produces a single dataset. We extend the framework to sequential settings, where evidence accumulates over time, rewards may be deferred to the end or disbursed incrementally, and the agent may stop the contract at a strategically chosen stopping time. We call a contract null-deterring if no ''null agent'' (those whose data-generating distribution lies in the null hypothesis) can profit in expectation under any stopping rule, and our main conceptual result shows that such contracts are characterized by e-processes. We then turn to the agent's best response and identify the optimal stopping rule through the Snell envelope of the reward process. We develop simpler, more interpretable approximations that often require the agent to have only partial knowledge of its distribution.