AI summaryⓘ
The authors focus on teaching robots to better understand detailed actions people do when assembling things by breaking down each action into parts like what is done, what object is involved, and what tool is used. They improve vision-language models by fine-tuning them with questions and answers about these action parts, making the system very precise. To learn well even with little data, they developed a special training method that lets different parts of the model focus on different action elements without interfering with each other. They also made new datasets from assembly videos to test their method, which outperformed existing approaches and gave clear predictions about each part of the action.
Vision-Language ModelsFine-tuningAction RecognitionHuman-Robot CollaborationMulti-task LearningLow-rank AdaptersCompositional ContextDataset CreationVideo Question AnsweringAssembly Actions
Authors
Hao Zheng, Jinyi Huang, Tiantian Zheng, Xun Xu, Tuka Alhanai
Abstract
Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.