LIME: Learning Intent-aware Camera Motion from Egocentric Video
2026-07-02 • Robotics
RoboticsComputer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors focus on teaching robots how to move their cameras based on natural language instructions, like "look around the corner" or "inspect this object." They created a system called LIME that predicts where the camera should move next to best fulfill these instructions. To train LIME, the authors used egocentric videos, extracting different camera motions linked to intentions described in language. Their approach helps robots actively adjust their viewpoint to better understand and interact with their environment.
autonomous robotscamera motionnatural language processingegocentric videoSE(3) posevision-language navigationactive perceptionflow matchingauto-regressive models
Authors
Boyang Sun, Jiajie Li, Yung-Hsu Yang, Chenyangguang Zhang, Tim Engelbracht, Sunghwan Hong, Cesar Cadena, Marc Pollefeys, Hermann Blum
Abstract
Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, language-conditioned camera motion remains comparatively underexplored as a first-class action. We formulate language-conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. This task is inherently non-trivial: viewpoint changes are driven by latent perceptual intentions, and a valid motion may operate at different semantic granularity, from entering a room to looking around a corner, inspecting a visible object, or revealing an occluded detail. To model this structure, we mine multi-intention camera-motion supervision from egocentric video, pairing plausible intents and observation-gain descriptions with relative SE(3) target poses. We propose LIME, a vision-language camera-motion generator that combines an auto-regressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across experiments and downstream robotic tasks, we show that LIME can learn to actively choose camera poses from passive human video, turning ordinary egocentric recordings into supervision for intent-aware active perception.