Summary
Tracking food and nutrition in a kitchen over time is harder than just recognizing food in pictures. This paper introduces NutriBench-Kitchen, a set of tests made from real cooking videos that evaluate how well an AI can remember ingredient changes and plan cooking steps using nutrition knowledge. The authors found that current AI models often struggle with estimating quantities, keeping track of ingredients over long periods, and making decisions based on nutrition rules. They also present Nutri-Vgent, an AI system that keeps separate memories for episodes, food states, and recipes, which improves performance. These tools help measure progress in making kitchen-assistants smarter by letting them remember and reason about food and nutrition as cooking happens.
Embodied AIFood state trackingNutrition managementVision-language modelsKnowledge-grounded planningRecipe understandingMemory managementLong-term planningIngredient estimationKitchen assistant
Authors
Yulin Wei, Xiangchen Wang, Jianhui Pan, Jinyu Xiao, Zheng Tan, Ruozai Tian, Guanhua Chen, Feng Zheng
Abstract
An embodied kitchen assistant must do more than recognize food in isolated frames. It must track ingredient states over time and integrate visual observations with recipe and nutritional knowledge to support constraint-aware decision-making. We formalize this capability as \emph{Embodied Nutrition Management}: perceiving nutrition-relevant events, maintaining a persistent food state, and using it for knowledge-grounded planning. Existing benchmarks evaluate static food understanding or embodied cooking actions, but do not measure whether an agent can continuously update and use nutrition-relevant states in dynamic kitchens. To fill this gap, we introduce \textbf{NutriBench-Kitchen}, a benchmark containing 1,500 manually verified question--answer pairs from 160 cooking videos. It covers five task families: Ingredient Entry, Memory Management, Recipe Query, Long-Term Planning, and Short-Term Planning, spanning food-state construction, maintenance, knowledge retrieval, and decision-making across different planning horizons. Evaluations of proprietary and open-source large vision-language models reveal a substantial gap from human performance, particularly in quantitative ingredient estimation, long-term state tracking, and reasoning under interacting constraints. We further introduce \textbf{Nutri-Vgent}, a diagnostic long-video agent with separate episodic, food-state, and recipe memories. Its consistent improvements demonstrate the value of explicit state representations and structured memory for nutrition management. Together, NutriBench-Kitchen and Nutri-Vgent provide a testbed for studying persistent state tracking and knowledge-grounded reasoning in dynamic kitchens. Code is available at https://github.com/V1ol1n/NutriBench-Kitchen.