AI summaryⓘ
The authors explain how large language models (LLMs) have grown from just generating text to becoming central parts of intelligent agents that can understand tasks, plan steps, and even control robots. They describe different methods where LLMs work with robots by understanding instructions, using visual information, and creating action plans. For example, models like SayCan and VoxPoser combine language understanding with robot skills or 3D mapping to help robots perform tasks. The authors also mention advanced models that link language, vision, and actions to improve robot behavior. Overall, these LLM-based agents interpret instructions, consider the environment, and generate plans that robotic systems can follow to complete tasks.
Large Language ModelsIn-Context LearningTask DecompositionRobot Task PlanningVision-Language ModelsEmbodied AgentsAffordancesMotion PlanningSkill Libraries
Authors
Jiawei Liu, Jiacheng Guo, Tian Zhang, Yiwei Xu, Juan Wang, Jinlin Fan, Bowen Xiao, Chi Guo, Keyan Guo, Hongxin Hu
Abstract
Large Language Models (LLMs) have demonstrated capabilities in in-context learning, task decomposition, step-by-step reasoning, and code generation, driving their gradual evolution from text generation models into the core of agents capable of perceiving environments, invoking tools, and executing tasks. Traditional LLM Agents typically obtain information through webpages, documents, databases, or external tools and generate corresponding invocation sequences according to user goals; when this technology is further integrated with robotic systems, large language models begin to undertake functions such as task understanding, high-level planning, and behavioral decision-making. SayCan combines the task reasoning capability of language models with the affordances of robotic skills, while Code as Policies and ProgPrompt generate robot task plans through policy code and programmatic prompting, respectively, and VoxPoser uses language models and vision-language models to construct three-dimensional value maps to guide robotic manipulation \cite{6,7,8,9}. Vision-language-action models such as PaLM-E, RT-2, and GR00T N1 further strengthen the connection among language, visual perception, and robotic actions \cite{10,11,12}. In such LLM-driven embodied agents, the model not only needs to understand user instructions, but also needs to combine scene states, object attributes, spatial relations, and execution feedback to complete task grounding, and then hand the generated action plan to skill libraries, motion planners, or controllers for execution.