Summary
This paper looks at whether a computer program that controls actions with images and language can get better by predicting what will happen next in the visual information it processes. The authors combine a vision-language model with a special action decision system and use features from a fixed visual predictor. They find that using current prediction features helps control success, but trying to predict further into the future does not improve results with the data they used. They also test different ways of adapting parts of the system but find no clear benefits. Their work shows the importance of testing prediction methods by how well the system controls things, not just by measuring internal prediction quality.
What this means in practice
- •For roboticists: Use vision-language fused features with predictive architectures to improve robot action success rates in visual control tasks.
- •For autonomous system engineers: Evaluate latent prediction techniques through closed-loop control success rather than internal prediction loss for system design choices.
Authors
Rufeng Xiao, Ruihao Xiao, Wei Chen, Xinhai Sun
Abstract
Joint-embedding predictive architectures (JEPAs) learn to predict representations rather than pixels, offering a compact way to model how visual scenes evolve. We ask whether this predictive objective improves closed-loop control when a vision-language-action (VLA) policy already uses V-JEPA features. We call our JEPA-conditioned flow policy family SaiVLA-JEPA. It combines a vision-language model with a diffusion transformer action expert and receives features from a frozen V-JEPA encoder. Two lightweight auxiliary heads predict future representations during training and are omitted at inference, leaving the action sampler unchanged. We compare policies without JEPA features, with current JEPA conditioning, and with additional future supervision on 10,030 paired LIBERO-Plus episodes. Current JEPA conditioning reaches about 82.5% success versus 71.6% without JEPA, while future supervision provides no detectable aggregate gain at the evaluated training budget. We separately evaluate SaiVLA-JEPA-TCA, which adapts latent context, and SaiVLA-JEPA-TPA, which adapts selected parameters. The context-retry gain is a system-level result under a larger attempt budget; parameter adaptation shows no consistent advantage over the frozen policy and target controls. These results motivate assessing latent-prediction objectives through paired closed-loop rollouts rather than latent loss alone.