ParticleSplat learns 3D object particles for better scene understanding
ParticleSplat: Self-supervised Object-centric Latent Particle Splatting
Computer Vision and Pattern RecognitionRobotics
Summary
Understanding scenes as collections of objects in 3D space is important for tasks like robot manipulation. The ParticleSplat method improves on prior work by representing scenes with 3D particles that capture object position and appearance, allowing the system to reason about objects in three dimensions. The method learns this representation from multiple images of a scene without needing labeled data. This results in the ability to separate objects visually and edit them in 3D, which also helps improve robot manipulation performance.
What this means in practice
- •For robotics engineers: Use learned 3D object particles to improve robot grasping and manipulation in cluttered environments.
- •For 3d graphics developers: Enable interactive 3D scene editing by manipulating object-centered particles in an unsupervised latent space.
Authors
Lyuxing He, Daniel Guo, Elizabeth Terveen, Deepak Pathak, David Held, Tal Daniel
Abstract
We present ParticleSplat, a self-supervised object-centric learning method that decomposes scenes into a set of latent ''particles'' representing semantic entities through feedforward 3D Gaussian Splatting. Building on the Deep Latent Particles (DLP) framework, which represents images as a set of particles with attributes such as position, scale, and visual appearance, we address a key limitation of DLP: its inherently 2D nature, which prevents explicit 3D spatial and geometric reasoning that are critical for downstream tasks such as robotic manipulation. Leveraging the structural similarity between latent particles and 3D Gaussian primitives, we introduce a 3D latent particle space trained with a novel view synthesis objective. Our model jointly encodes multiple views with camera poses into a shared 3D object-centric latent space, then transforms particles into particle-aligned 3D Gaussians whose composition reconstructs the full scene. On simulated and real-world datasets, we show that this formulation inherently learns object masks without supervision and supports controllable 3D scene editing, such as moving objects by modifying particles in the latent space. We further establish that the learned 3D representation improves downstream performance on robotic manipulation tasks.