Urban models improve decisions by comparing activity to local baselines
Measuring Decision-Scale Use in Tool-Augmented LLMs: A Contrastive Urban Benchmark
Information Retrieval
Summary
It's often hard to know if an event happening in one neighborhood is unusual compared to another because places have different typical activity levels. The authors created a test called URBANCONTRASTIVEQA to see if language models with tools can tell when an activity level is unusual for its own area rather than just bigger than another place. They tested six different models using data from cities like New York, Chicago, and Seattle. They found models usually just say the bigger number, missing the local context. Providing models with information about what is normal for each place helps them make better comparisons.
What this means in practice
- •For city planners: Support decisions by identifying unusual activity compared to local norms across urban areas using tool-augmented language models.
- •For transportation analysts: Improve assessments of mobility patterns by integrating local baseline context into model outputs for different city zones.
Authors
Ray Chen, Vivian Wong, Christan Grant
Abstract
Urban decision-support often asks whether activity is unusually high or low for a specific place, not which place has the larger raw count. Twenty pickups in a quiet neighborhood can be more abnormal than 180 at an airport. We introduce URBANCONTRASTIVEQA, a benchmark that asks whether tool-augmented language models can make this baseline-relative comparison. Each item pairs two urban situations from public mobility data in NYC, Chicago, and Seattle, labeled by how far current activity deviates from that place's historical baseline. We evaluate six instruction-tuned models under five tool-output formats. With only raw counts, models often pick the larger number even when it is less abnormal for its zone. Server-computed baseline scores and ordinal labels raise accuracy, but gains vary by model. For heterogeneous urban feeds, tool interfaces need to expose local baselines, not just activity volumes. We release the pair bank, labels, scoring scripts, and data card.