Sparse-view 3D animal behavior models improve brain data analysis
Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding
Computer Vision and Pattern Recognition
Summary
Understanding how the brain works needs accurate ways to describe animal behavior. The authors created a new method called SABLE that builds 3D views of animals’ actions from just two video angles without needing exact 3D labels. This method keeps important details that simpler pose-tracking methods miss and produces clear 3D information unlike complicated video encodings. It works well on various animal videos and helps link behavior directly to brain activity, even for animals it has never seen before.
What this means in practice
- •For neuroscience research labs: Analyze neural activity more precisely by using detailed 3D behavior data reconstructed from minimal video angles.
- •For wildlife monitoring teams: Reconstruct 3D behaviors of animals in the wild from sparse camera views without needing extensive 3D calibration.
Authors
Xinming Dai, Qihang Jin, Tianshu Tan, Baiyuan Chen, Hanrui Lyu, Lenny Aharon, Kyle Daruwalla, Xun Helen Hou, Matthew R. Whiteway, Liam Paninski, Yizi Zhang
Abstract
A deeper understanding of brain function requires a precise, structured characterization of behavior.Yet, extracting behavioral representations from video in a form suitable for scientific analysis remains a fundamental challenge. Many prior studies represent behavior via pose estimation or nonlinear video embeddings. However, pose tracking discards rich information beyond predefined keypoints, while nonlinear video embeddings lack interpretability. We address this limitation with SABLE (Sparse-view Animal Behavior Latent Embeddings), a self-supervised framework that leverages a geometric inductive bias to learn behavior representations.By augmenting a multi-view transformer with priors from monocular depth and pose estimation, SABLE reconstructs 3D animal behavior from extremely sparse views while learning explicit 3D latent structure. Without ground-truth 3D labels, it reliably recovers 3D behavior from two-view videos, whereas state-of-the-art (SOTA) methods fail or yield degenerate solutions. Across the International Brain Lab and Cheese3D datasets, we demonstrate that SABLE learns 3D representations that match or exceed prior SOTA performance in neural encoding and decoding. Once pretrained across animals, SABLE serves as an off-the-shelf model that generalizes zero-shot to unseen animals without animal-specific calibration or retraining. Our method establishes 3D-aware video embeddings that capture complex behavior, opening new avenues for studying brain-behavior relationships.