Large language models struggle to follow complex action sequences accurately
Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generation
Artificial Intelligence
Summary
It is unclear if large language models (LLMs) truly understand how sequences of actions depend on each other, or just mimic common patterns. The authors examined this by testing LLMs on simple games like Rock-Paper-Scissors and sequence prediction tasks. They found that just looking at longer histories doesn’t help models figure out underlying strategies or rules. Even when LLMs seem to act realistically, their method for generating sequences may be incorrect. This means LLMs might not reliably simulate behaviors that require understanding of detailed step-by-step dependencies.
What this means in practice
- •For game developers: Design simulation opponents that require carefully tested sequence following beyond superficial pattern matching.
- •For dialogue system engineers: Improve dialogue agents by recognizing that longer conversational context may not guarantee correct understanding of sequential rules.
Authors
Jerry Wang, Zhengxiang Wang, Ting Yu Liu, Hsin-Ling Hsu, Yi-Cheng Lai, Tengfei Ma
Abstract
Large language models (LLMs) are increasingly used as interactive agents and simulators, yet it remains unclear whether they can recover latent sequential structure beyond surface action frequencies. This distinction is critical for behavioral simulation, where actions are often shaped by prior context rather than marginal frequencies alone. We study this question using controlled two-player Rock--Paper--Scissors interactions and a one-player stochastic n-gram continuation task. Across these experiments, we test whether LLMs can identify latent strategies, follow simple Markov rules, and sustain higher-order conditional dependencies. Our framework separates distribution matching from conditional rule following. Results show that longer context does not improve identification, correct recognition does not ensure faithful simulation, and higher-order dependencies substantially degrade rule recovery. Apparent behavioral fidelity can therefore mask incorrect generative mechanisms.