Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation
2026-06-22 • Robotics
Robotics
AI summaryⓘ
The authors developed See2Act, a method that helps robots learn to both look around and act when objects are partly hidden. Instead of assuming the robot can always see everything, their approach trains the robot to decide where to look next based on previous views, improving its ability to deal with occlusions. They tested this on simulated tasks and showed better results than older methods, and also successfully transferred it to real-world pick-and-place tasks without extra training. This shows that learning where to look is important for robots to handle situations with limited visibility.
imitation learningocclusionviewpoint refinementaction denoisingoffline demonstrationssim-to-real transferpick-and-place taskspartial observabilitydigital twinrobot manipulation
Authors
Kuancheng Wang, Vaibhav Saxena, Shuo Cheng, Yotto Koga, Danfei Xu
Abstract
Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability.