ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement
2026-08-17 • Computation and Language
Computation and LanguageComputer Vision and Pattern Recognition
AI summaryⓘ
The authors created ReRef-3D, a test set for putting objects in 3D scenes based on language instructions. It has thousands of instructions describing where to place things in different ways. Instead of picking one spot, their method checks if the chosen placement fits the language and physics rules in the scene. They tested several models and found that some do better at following relationships than ensuring physical correctness, and certain language cues are harder for the models. Changing how instructions are worded didn’t change results much.
3D placementlanguage-guided placementCLEVR scenesrelation satisfactionphysical validitynatural language instructionsbenchmark dataset3D scene understandingfine-tuning modelsobject placement
Authors
Mary Lynn Martin, Yifei Zhang, Martha Palmer, Maria Leonor Pacheco
Abstract
We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.