Wrist mounted event cameras improve robot hand manipulation under bad lighting
ECHO: Event-Augmented Context with Hindsight and Outlook for Wrist-Only Manipulation
Computer Vision and Pattern RecognitionRobotics
Summary
Manipulating objects with robot hands using regular cameras often fails in very bright or dark conditions because the camera images become unclear. The authors show that event cameras, which detect changes in light instead of just capturing images, can help. They developed a method called ECHO that uses a wrist-mounted event camera combined with smart prediction of past and future events to guide robot hands better. Their method works better than traditional cameras, even when lighting is very poor, both in simulation and real-world tests.
What this means in practice
- •For robotics engineers: Use wrist-mounted event cameras with ECHO to improve grasping and manipulation under varying lighting conditions for robotic arms.
- •For automated warehouse operators: Enhance robotic pick-and-place reliability in warehouses by employing wrist-only event camera systems that handle poor lighting better than conventional RGB cameras.$Commercial implications: Enables more robust robotic automation products for warehouses operating 24/7 under variable lighting, improving efficiency and reducing errors.
Authors
Xinyue Wang, Yicheng Jiang, Zesen Gan, Junhao He, Jiaxu Wang, Junhao Li, Jingtao Zhang, Tianlun He, Jianan Wang, Isabel Guan, Qiming Shao
Abstract
Learning-based manipulation policies relying on RGB cameras often suffer from degraded observations under extreme exposure. Event cameras mitigate this degradation by asynchronously detecting pixel-level intensity changes to offer a high dynamic range. However, their observations heavily depend on camera placement, as fixed cameras miss static scene content while wrist-mounted camera motion causes previously visited regions to leave the field of view. To address these spatial-temporal limitations, we present ECHO (Event-augmented Context with Hindsight and Outlook), a wrist-only latent world action model that encodes wrist events into compact motion representations to provide temporal and spatial context for policy reasoning. Specifically, ECHO utilizes a pretrained event encoder to explain visual-feature changes between frames. Its hindsight module preserves the gripper trajectory with past event stream as addressable off-camera context. Concurrently, the outlook module introduces learnable event foresight queries supervised to anticipate the event window for future actions, enabling the policy to predict upcoming scene changes. Evaluated on wrist-only RLBench tasks, ECHO outperforms RGB and RGB+event baselines by 20.6 and 12.0 percentage points under normal lighting, and by 14.6 and 11.3 points under severe exposure drops, respectively, while also surpassing RGB references using a third-person camera. Real-world experiments with a wrist-mounted event camera validate that ECHO outperforms RGB-only and RGB+event baselines across multiple tasks under both nominal and severely dark lighting. Project page is at https://echo-wam.github.io/.