Block-sparse features improve robot control and data targeting
Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models
Robotics
Summary
Controlling how robots behave can be tricky because their brain-like software uses many complicated internal features. The authors show that by using a special method called block-sparse featurizers, they can identify meaningful chunks of these features related to things like sight, language, and movement. This lets them steer the robot's actions more precisely and find exactly which parts of the training data cause specific behaviors, like recovering from failure. Their approach offers a clearer way to understand and adjust how robots learn and act.
What this means in practice
- •For robot developers: Precisely adjust robot behaviors by manipulating interpretable internal features identified via block-sparse featurizers.
- •For machine learning engineers: Improve robot training by isolating small, critical subsets of data responsible for specific behaviors to refine datasets efficiently.
Authors
Aiden Swann, Mozes Jacobs, Mathilde Papillon, Siddharth Boppana, David Chanin, Dron Hazra, Usha Bhalla, Curt Tigges, Mac Schwager, Owen Lewis, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger, Thomas Fel, Matthew Kowal
Abstract
Can better feature discovery improve control over robot behavior through internal interventions and training data? Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace. We investigate whether recovering this structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn. We show that BSFs recover several multi-dimensional task-relevant features underlying the model's computation (e.g., perception, language, proprioceptive state, episode phase, and control variables). We causally validate these features through fine-grained steering: targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction. Finally, we use feature-based data attribution to isolate where a specific behavior (failure recovery) arises in the training dataset, identifying fewer than $1.4\%$ of training tokens whose masking is sufficient to remove the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior.