AI summaryⓘ
The authors discuss how evaluating AI models used in physical robots is less developed compared to evaluating foundational AI models. They build on a previous framework (DeepInsight I) and focus on the physical, embodied parts of AI like navigation, manipulation, and whole-body control. Their new work (DeepInsight II) tests these AI controllers both in simulations and on real robots, comparing results directly to better understand the differences. They also introduce ways to diagnose and fix problems observed during physical robot tests using specific labels linked to repair actions. Overall, the authors aim to bridge the gap between simulation tests and real-world robot performance with a consistent evaluation approach.
Physical AIFoundation modelsNavigation benchmarkManipulation benchmarkWhole-body controlSimulation-to-real gapDeepInsight frameworkRobot evaluationTrace localizationRepairability
Authors
Siyi Li, Yuchen Kang, Wuliang Wang, Zhengjie Zhang, Jiangpin Liu, Jianhao Yao, Jie Chen
Abstract
Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment actually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions---task, resource, and result---but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight II keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured repairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis.