NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning
2026-06-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created NoRA, a new test using videos where AI must suggest next actions and explain them with clear facts and reasons. Unlike past tests that only ask AIs to choose from given options or just write about judgments, NoRA makes AIs identify good actions from scratch based on what they see. They tested 12 AI models and found that while the models often find plausible actions and related facts, they have trouble making a full set of reasonable choices and linking actions to the correct reasons. This new test helps measure how well AI systems can not only pick but also properly explain good actions in real settings.
Large Language Models (LLMs)agentic systemsnormative competencevisual first-person video benchmarkfact-reason-action support graphaction alignmentfactual groundingsupport bindingmultimodal systemsprompting regimes
Authors
Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar
Abstract
LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior. However, existing approaches either assess normative judgment in text alone or reduce it to choosing among a fixed set of candidate actions. We argue both are insufficient. In practice, agents are never handed a menu of options; they must identify a reasonable action from scratch, grounded in visible facts and supported by inspectable reasons. We introduce NoRA, a visual first-person video benchmark that requires models to generate candidate next actions and justify each through an explicit fact-reason-action support graph. The benchmark comprises 1,420 annotated video clips, including HumanGold-190 and LLMSilver-1230 splits. Each instance is evaluated through action alignment, factual grounding, and support binding, aggregated into a single grounded reasonableness score. We benchmark 12 multimodal systems under direct, deliberate, and structured prompting regimes, finding that current VLMs frequently recover plausible actions and relevant scene facts, but consistently struggle to construct the full reasonable action space and bind selected actions to the correct local support. NoRA makes this gap measurable, shifting the evaluation question from whether a model can pick an action to whether it can justify an appropriate action for the right visible reasons.