Checking test state identity improves vision language action model comparisons
Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models
Robotics
Summary
Testing AI models that understand images and language while taking actions relies on running them in the same test conditions to compare fairly. This paper shows that just matching test identifiers might not guarantee identical starting states because of how simulations reset. The authors offer a framework that logs and checks detailed setup information to ensure test states truly match before comparing models. This helps make sure comparisons are accurate and fair.
What this means in practice
- •For ai model evaluators: Verify initial test states match exactly before comparing AI models in simulation to ensure reliable performance comparisons.
- •For simulation software developers: Incorporate integrity checks for test state consistency in simulation platforms to avoid misleading evaluation results.
Tested on simulated data.
Authors
Rafiqul Islam
Abstract
Simulation-based VLA evaluation often treats matching task, seed, and episode identifiers as sufficient evidence that two policies faced identical test conditions. Under specific vectorized reset and autoreset conditions, different termination timing can change subsequent initial-state assignment, so the same episode index need not mean the same test state. We present an evaluation-integrity framework for paired comparison - invariants I1-I5, provenance logging, optional scene fingerprints, and an automated checker - and a controlled LeRobot 0.6.1 LIBERO harness study with emulated controllers. Four evidence levels are kept separate: (A) mixed-horizon Sync wave-1 IDs [20, 5, 22, 7] vs [4, 5, 6, 7]; (B) homogeneous separate-job Sync control (16 cells, 0 ID/fingerprint mismatches); (C) Async+Freeze batch-size map differences (12 I3 mismatches; bs=8 incomplete; mechanism unisolated); (D) a synthetic McNemar illustration only. Controllers were emulated and no real-VLA shared-vector experiment was completed. We recommend checking declared/resolved init-state identity and provenance before paired statistics. We do not claim that published VLA leaderboards are generally invalid.