Trafficimag benchmark tests counterfactual roadside traffic video generation
TrafficImag: A Benchmark for Counterfactual Roadside Traffic Video Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Road cameras capture how traffic moves, but changing one vehicle's actions and predicting what happens next without breaking real-world rules is hard. The authors created TrafficImag, a new test set and method to check how well computer programs can change a car’s behavior in a roadside video and still keep everything nearby looking right and logical. TrafficImag includes many videos with detailed labels and a way to measure if the changes follow road rules and keep other cars unchanged. They found current AI models still struggle to perfectly generate these 'what-if' traffic videos, pointing to areas needing improvement.
What this means in practice
- •For traffic simulation developers: Use TrafficImag’s benchmark to test and improve AI models that predict how altering one vehicle’s behavior affects future traffic videos realistically.
- •For autonomous vehicle software teams: Evaluate and refine algorithms that simulate alternate traffic scenarios to enhance planning and safety validations using TrafficImag’s counterfactual video framework.
Authors
Xiangyu Li, Tianyi Wang, Zhihao Dou, Christian Claudel, Zhaomiao Guo
Abstract
Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video generation, in which a selected actor is modified and the generated future should remain consistent with road topology and unrelated traffic. We introduce TrafficImag, the first benchmark for counterfactual roadside traffic video generation. TrafficImag combines a large-scale roadside dataset (9,022 annotated images, 7,043 deduplicated video clips, and 31,145 actor-centered history-future samples) with an executable protocol that supports behavior reasoning, intervention-aware image editing, and conditional video generation. Each intervention is represented as an actor-level program describing the target actor, intended behavior, legal route, interaction order, and temporal constraints, enabling a unified evaluation interface across heterogeneous foundation models. TrafficImag evaluates four complementary validity dimensions: initial-state correctness, route and behavior validity, interaction consistency, and non-target preservation, and considers an end-to-end counterfactual successful only when all four are satisfied. Across state-of-the-art foundation models, the strongest reasoner reaches 80.4% macro F1, the complete condition interface raises end-to-end success from 23.3% to 55.0% for the best generator. Oracle studies further show that conditional video execution is the primary remaining bottleneck. TrafficImag provides a reproducible benchmark for evaluating and diagnosing counterfactual traffic video generation beyond perceptual video quality.