Streamdecisionbench assesses language model decisions over time in live streams
StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams
Computation and Language
Summary
When language models help programs make decisions based on ongoing information, sometimes they keep using an old decision even after new information arrives. This can cause errors that aren’t noticed by traditional tests that check only the final decision. The authors created StreamDecisionBench, a way to check if the current decision at every moment is correct, figuring out if mistakes are because of bad judgment, slow updates, or both. They tested it on different language models and scenarios, showing how accuracy changes over short time intervals and how reasoning affects performance.
What this means in practice
- •For software engineers: Evaluate how language model decisions hold up over time within stream-based applications to improve real-time responsiveness.
- •For ai system integrators: Benchmark different hosted language models on evolving input streams to select optimal models balancing judgment and update latency.
Authors
Jhen-Ke Lin, Chung Chun Wang
Abstract
As natural language drives more applications, language models increasingly run inside programs as decision components: the program sends them the current state and acts on the returned decision until a newer one arrives. When evidence changes during inference, a decision correct for its own state can stay in force after that state has passed, as when a call recorder keeps running after a customer starts reading out a card number; untimed (offline) accuracy counts such an error as correct. We introduce StreamDecisionBench (SDB), which evaluates the decision in force at every instant and attributes every erroneous instant to judgment, latency or both. Its scenarios stream evidence in four application families, with reference decisions computed from public rules by executable code. We summarize in-force accuracy across update intervals of 1-5 s by its normalized area under the curve on a logarithmic time axis, giving equal weight to equal multiplicative ranges. Across six settings of four hosted models, this score stays within 2.9 points of the scenario-wise product of untimed accuracy and an oracle's integrated timing score. Reasoning improves judgment, but at low effort latency costs Luna and Terra, two GPT models we also evaluate without reasoning, 38.8 and 42.8 points relative to untimed accuracy; a faster component with weaker judgment attains a similar integrated score to Terra without reasoning. The aggregate and family curves show where these tradeoffs change, making the evaluation's time-scale dependence visible.