Vision language models improve robot actions with better time frequency coding
Time-Frequency Geometric Cross-Attention for Chunked Vision-Language-Action Models
Artificial IntelligenceRobotics
Summary
Robots need to plan many steps of movement at once, but current models treat these as simple sequences of tokens, missing details in how actions change over time. The authors designed a new method that breaks down movements by their smooth trends and quick corrections and also better connects different action phases that usually look very different to the model. This method combines standard similarity with a new way to detect very different but related motions, improving robot task performance especially when facing new situations. Their approach can be added to existing robot models without starting from scratch.
What this means in practice
- •For robotics engineers: Improve robot manipulation tasks by integrating a time-frequency attention module to better capture the structure of multi-step action sequences.
- •For autonomous vehicle developers: Enhance planning systems to recognize and respond to distinct motion phases for smoother trajectory execution in autonomous driving.
Authors
Shengye Dong, Haochen Niu, Hao Liu, Peiwen Lin, Chuang Wang, Shanmin Pang
Abstract
Modern vision-language-action (VLA) policies predict a whole chunk of actions: one to two seconds of coordinated motion emitted in a single forward pass. Yet an action chunk is essentially a short multivariate trajectory, but inside these models it is a sequence of generic per-timestep hidden tokens decoded by a linear head. This under-serves two motion structures. First, frequency: a chunk superimposes a smooth global trend and fine corrective motion across time scales, and a single token entangles them. Second, cross-phase geometry: motions of different phases (reach, contact, grasp adjustment, settling) unfold along very different, near-orthogonal directions in representation space, yet are tightly related for the task and arise across the time axis. Dot-product attention scores alignment by an inner product, so it favors aligned tokens and is least sensitive near orthogonality, leaving such relationships for the network to recover through a detour. We introduce Time-Frequency Geometric Cross-Attention (TFGCA), a drop-in module repairing both blind spots. TFGCA uses a per-dimension learnable stationary wavelet transform to decompose the action chunk into time-frequency tokens, and each time token retrieves information from them via a cross-attention that fuses the dot product (similarity) with the wedge-product magnitude (sensitive to near-orthogonality) through a learnable weight. A zero-initialized residual reproduces the base behavior at initialization, so it can be dropped onto a pretrained VLA and fine-tuned jointly. Relative to the same-source base, TFGCA improves in-distribution LIBERO by +1.5 on average, the OOD LIBERO-Plus by +6.3, the randomized average under RoboTwin domain randomization by +28.5, and the overall success rate on three real-robot AgiBot A2 tasks by +11.67 points, with larger gains out of distribution.