Compositional dataset enables modeling of multiple interacting objects
COSMI: COmpositional Synthesis of Multi-object Interactions
Computer Vision and Pattern Recognition
Summary
Modeling how people interact with multiple objects is hard because datasets usually capture interactions with one object at a time, making multi-object data rare. The authors noticed that many multi-object interactions can be broken down into simpler single-object parts. They created a large dataset by combining these single-object interactions in realistic ways to simulate multiple objects being handled together. Using this dataset, they trained a model that can generate realistic multi-object interactions from text descriptions, even for new object combinations it hasn't seen before.
What this means in practice
- •For virtual reality developers: Generate realistic multi-object human-object interactions to improve immersion in VR environments with varied object combinations.$Commercial implications: This enables more dynamic VR experiences by providing generated realistic hand-object behaviors, allowing VR companies to create richer interactive content.
- •For robotic manipulation engineers: Train robots with simulated human interactions involving multiple objects by leveraging the compositional dataset for varied scenarios.
Authors
Daniel Eskandar, Ilya A. Petrov, Gerard Pons-Moll
Abstract
Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities. We compose them: contact-consistent clips of single interactions, mirrored to balance the hands, transfer between bodies, and a language model and geometric checks admit only the pairings that are plausible, semantically and physically. Therefore, the dataset grows combinatorially with the clips rather than recording time. The COSMI dataset holds 222k sequences and 275 hours with up to five objects, nearly thirty times the largest multi-object capture, and can be extended by adding datasets or even hand-object recordings. On this data we train the COSMI method, a text-to-interaction diffusion transformer that follows how the data is built: weight-shared object slots generate a variable number of objects, predicted relative to the body parts that move them. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to the unseen combinations. COSMI outperforms baselines in text alignment and contact accuracy, where its margin is largest on the unseen object. Code, models, and the dataset pipeline will be released on the project page: https://ptrvilya.github.io/cosmi.