Brain signals and camera paths reconstruct 3D room layouts accurately
DRAFT: Decoding and Reconstructing 3D Room Layouts from fMRI and Camera Trajectories
Computer Vision and Pattern RecognitionMachine Learning
Summary
Understanding the shape and size of rooms just from brain activity while a person watches video walkthroughs is very challenging. This paper introduces a method called DRAFT that combines brain signals with information about camera movement to recreate the 3D layout of unseen indoor spaces accurately. The approach works better than previous methods that used brain data alone or camera paths alone. This advance shows it is possible to recover detailed spatial layouts people see using brain signals together with known motion paths.
What this means in practice
- •For augmented reality developers: Create virtual environments that adapt to user brain signals combined with device movement to enhance spatial understanding without visual input.
- •For brain-computer interface engineers: Develop systems that reconstruct spatial layouts from brain activity and known movements to assist navigation or spatial memory tasks.
Authors
Chi-Wei Lee, An-Kai Chen, Cheng-Han Chiang, Yi-Chun Chang, Po-Chih Kuo
Abstract
While functional magnetic resonance imaging (fMRI) decoders reconstruct images and 3D objects, recovering the metric geometry of rooms from a moving viewpoint remains an open challenge. We introduce, to our knowledge for the first time, the task of reconstructing 3D layouts of held-out scenes using only the known camera trajectories and the fMRI signals of participants watching indoor walkthroughs, without any visual frames. The task is hard because of the temporal mismatch between rapid visual changes and blurred fMRI signals. Our method, DRAFT (Decoding and Reconstructing 3D room lAyouts from fMRI and Trajectories), extracts a compact layout representation from the early visual cortex and fuses this neural code with the geometric free space derived from camera motion. To evaluate it, we propose layout r, the Pearson correlation between predicted and ray-cast log depths. On ten held-out scenes (20 participants), DRAFT reaches a layout r of 0.480, outperforming the camera path alone (0.446) and seven fMRI decoders (at most 0.429). DRAFT also works as a plug-in that improves all seven decoders without retraining, raising their layout r from 0.171-0.429 to 0.487-0.493. Brain activity and camera motion can thus recover the metric layout of unseen rooms, a first step toward reconstructing the 3D spaces people see.