Beyond Point-Attached Semantics: Object-Centric Semantic Fields for Generalizable Manipulation
2026-07-03 • Robotics
Robotics
AI summaryⓘ
The authors address the challenge of helping robots understand important parts of objects, like handles or openings, in 3D space to improve manipulation. Instead of relying on point clouds that change depending on the view, they create a continuous semantic field that gives stable descriptions of object parts at any 3D location. This method is trained on objects with labeled parts and helps robots perform better in both simulations and real-world tasks compared to older methods. Their approach makes robot manipulation more reliable by focusing on consistent object part information.
robot manipulation3D point cloudsemantic segmentationcontinuous semantic fieldfunctional object partspart-aware embeddingsmanipulation policiesRoboTwin simulationbimanual manipulationobject-centric representation
Authors
Zheng Sun, Lerong Zhang, Zhihao Li, Zhuo Li, Quentin Rouxel, Fei Chen
Abstract
Generalizable robot manipulation requires stable 3D understanding of functional object parts, such as handles, tool heads, openings, and graspable regions. Raw point clouds provide geometry but lack explicit part semantics, and their sampled points vary with viewpoint, sensor configuration, and object instance. Existing 2D feature lifting and discrete 3D point-wise features enrich point clouds with semantics, but the resulting features remain attached to observation-dependent samples. We propose an object-centric continuous semantic field that conditions on an object point cloud and reads part-aware semantic embeddings at explicit 3D query locations. The field is trained from part-annotated object models and then frozen to generate semantic point clouds as object-level conditioning for manipulation policies. Experiments on RoboTwin simulation tasks and real-world bimanual object manipulation show that our representation provides more stable functional-part cues and improves policy performance over raw point-cloud, 2D feature lifting, and 3D point-wise feature baselines. Project Page: \href{https://zainzh.github.io/beyond-point-attached-semantics}{https://zainzh.github.io/beyond-point-attached-semantics}.