When cross-view consistency matters for robot vision control tasks

When Does Cross-View Consistency Matter in Multi-View Appearance Augmentation? A Controlled Study for Visuomotor Policies

RoboticsComputer Vision and Pattern Recognition

Summary

Keeping images from multiple cameras consistent is common in robotic vision, but it’s unclear which types of consistency help robots perform better. The authors studied how different changes in camera views affect robot control in simulated tasks, finding that appearance changes usually have little effect, while inconsistencies in geometry between views hurt success more. They highlight that preserving geometric alignment and spatial correspondence between camera views is more important than keeping appearance exactly the same. This suggests robot vision systems can sometimes simplify appearance constraints without losing performance.

What this means in practice

  • For robotics engineers: Design visuomotor policies that prioritize geometric consistency across cameras over appearance consistency for better manipulation task performance.
  • For computer vision system developers: Optimize multi-camera augmentation pipelines by relaxing appearance coupling constraints where possible to reduce computational costs without harming control accuracy.

Authors

Zixuan Jiang

Abstract

Multi-view visual augmentation often seeks consistency across cameras, but it is unclear which forms of consistency matter for downstream control. We use simulation-state replay to manipulate cross-view coupling while preserving the per-camera perturbation distributions, physical trajectories, and action labels. Across seven LIBERO manipulation tasks, marginal-matched changes in global appearance, ghost injection, and target color have small average costs for a late-fusion Diffusion Policy: point estimates are near zero or negative, and the largest upper endpoint of the 90% posterior credible intervals is 2.55 percentage points. In contrast, geometrically anti-coupled observations reduce success by 3.30 percentage points relative to coupled observations (90% CrI [0.64, 6.28]), with displacement magnitude, direction marginals, and the number of affected views matched. Camera interventions reveal substantial dependence on wrist-view geometry, while feature analysis shows appearance sensitivity patterns associated with the factors randomized during training. ACT shows a higher target-color cost estimate than DP, with uncertainty that includes zero and sensitivity to task composition. Within the tested tasks and policy configurations, these findings motivate selective consistency constraints: appearance coupling can sometimes be relaxed, whereas task-relevant geometry and cross-view spatial correspondence warrant protection.