Code driven framework improves handling of evolving information in conversations
EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural-Language Rendering
Artificial IntelligenceSoftware Engineering
Summary
Sometimes, when information changes over time, earlier facts get updated or erased, making understanding the full story harder. The authors created EvolveScaler, a method that first defines these changing facts using clear, computer-readable rules before turning them into natural language stories. This approach helps test and train computer models better by simulating events and checking answers precisely. Their tests show this method creates challenging examples and helps improve model performance on new situations.
What this means in practice
- •For conversational ai developers: Generate and verify complex evolving dialogue scenarios with explicit state management for better model training and evaluation.
- •For software testing teams: Create detailed executable simulations of dynamically changing system states to improve automated testing coverage and accuracy.
Authors
Ziliang Zhao, Zenan Xu, Shuting Wang, Zhao Wang, Bowen Cao, Minda Hu, Lincheng Li, Pluto Zhou, Zhicheng Dou
Abstract
In persistent interactions, long contexts may encode an evolving process rather than a fixed record: later events can revise or revoke earlier information, changing what remains valid and what conclusions follow. We call this setting information evolution (IE). Solving IE requires identifying valid records, applying updates in order, and reconstructing the query-relevant state from the event history. Existing text-first synthesis pipelines make such data difficult to verify because state transitions and answer logic remain implicit. We introduce EvolveScaler, a code-driven framework that defines information evolution before rendering it as natural language. Human-authored operational specifications define state transitions, record validity, difficulty controls, and executable answer logic; a strong LLM then synthesizes a self-contained simulator from each specification. Executing validated simulators produces natural-language multi-turn event histories, while deterministic replay computes reference answers and atomic checklists. We instantiate EvolveScaler with 117 task prototypes and 159 final-question operators across five difficulty levels spanning approximately 7 to 1,200 events per instance, yielding about 35,100 training examples and 585 validated evaluation instances. On the very_long tier, the strongest model reaches 59.3% avg@5, while six models score below 10%. Training an internal A3B model on 6,000 EvolveScaler examples improves performance over its base checkpoint on all eight independently constructed out-of-distribution benchmarks, with a 5.25-point average gain. These results show that code-driven IE synthesis provides both challenging evaluation and transferable training supervision.