Allentown simulation tests language models in crime surveillance tasks
Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks
Artificial Intelligence
Summary
It’s hard to know how well language models can help with crime investigations because real cases don’t have a perfect answer to check against. The authors made a virtual city called Kierston, copying a real place with fake people and records, to test these models fairly. They created a set of tasks to see if language models can piece together clues and find truth from many different types of records. Most models struggled, but some advanced ones did better and avoided guessing when unsure. Still, none can replace a human investigator yet.
What this means in practice
- •For law enforcement analysts: Test language models' abilities to analyze multiple surveillance datasets in simulated investigations before real deployment.
- •For security software developers: Develop and evaluate AI tools for linking and interpreting diverse records in crime and surveillance scenarios using the KierstonBench simulation.
Tested on simulated data.
Authors
Andrew Semenov, Svyatoslav Dorofeev
Abstract
Large language models are candidates for collecting and analyzing records in criminal investigations and mass surveillance. Their capability and accuracy in this work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records that surveillance and investigative systems collect about them. These records derive from a common behavioral timeline and are distributed across sources representing separate providers. Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator.