Summary
Understanding how far an object slides after being pushed depends on hidden properties like weight and friction, which aren't clear from just one picture. The authors show that common tests of visual AI systems often fail to tell if these systems really understand physics or just rely on obvious cues in perfect scenes. They introduce a new test called CALIPER that checks if a model can use information from multiple strikes to predict sliding distance in more realistic, varying environments. Their findings suggest many AI vision models do not truly learn physical properties but instead depend on constant camera views or background setups. This work points out the importance of testing AI perception in diverse and changing real-world conditions.
pretrained visual encodersphysical inferencesimulationlinear probecamera viewpointmass and frictionmodel calibrationvisual representationworld modelsmanipulation tasks
Abstract
How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as the perception front end of world models for manipulation, and their physical competence is assessed with perturbation benchmarks and linear probes, almost always in a clean, fixed-camera scene. We show that these assessments cannot distinguish an encoder that infers physics from one that does not. CALIPER (calibrate, then predict) is a direct test: an object of unknown mass and friction is struck twice at known speeds, a third strike is shown only up to the moment of contact, and a linear readout on frozen features must predict how far the object slides. Swapping in another object's calibration clips checks that the evidence is actually used. Across 2,000 simulated episodes and eight representations, from V-JEPA 2 to a randomly initialised ViT and raw pixels, calibration adds +0.50 R^2 and the swap removes it. Yet in the clean scene every representation lands within 0.02 R^2 of the ceiling set by true simulator state, because a fixed camera exposes the object's displacement directly in pixel coordinates. Resampling camera, lighting, and clutter for every clip spreads the same representations across 0.50 R^2; when the readout chooses a push speed for a goal distance, V-JEPA 2 misses by 4 mm and the random ViT by 20 mm, no better than ignoring the object. Linear probes track none of this: a change in frame aggregation moves a probe more than pretraining does, and erasing the probed mass direction from the same representation costs nothing in one scene and 0.35 R^2 in the other. Whether a benchmark can rank models is an empirical property, and we give three checks that establish it.