AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment
2026-08-17 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created a virtual cockpit environment called AeroCopilot Operational Environment (ACOE) to test how well AI agents can follow aviation procedures and stay safe during flight tasks. They also developed AeroCopilotBench, a two-level benchmark for evaluating AI on both aviation knowledge and real-time emergency task execution. Their tests show that even when AI knows the facts, it can struggle to carry out complex procedures safely. They found that successful AI needs to be aware of cockpit states, manage long tasks, and avoid safety violations. This work provides tools to better evaluate and improve AI agents helping pilots in interactive, safety-critical settings.
large language modelsaviation proceduresvirtual cockpitemergency tasksstate transitionssafety constraintsoperational safetytask executionAI evaluation benchmarks
Authors
Yuchen Yuan, Zhenghuang Wu, Yuangan Li, Liang Ma, Ke Li
Abstract
Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on static knowledge do not support systematic testing of procedural execution and safety compliance in interactive environments. This paper presents the AeroCopilot Operational Environment (ACOE), a reproducible interactive virtual-cockpit test environment, and AeroCopilotBench, a two-tier aviation agent evaluation benchmark. Tier-1 evaluates aviation knowledge using 1,200 multiple-choice questions, while Tier-2 comprises 73 emergency and abnormal tasks derived from the manufacturers' Pilot's Operating Handbooks (POHs) and instantiated in ACOE. ACOE converts natural-language procedures into executable state transitions, final-state goal conditions, and hard safety constraints, enabling models to interpret cockpit state, diagnose faults, and operate aircraft systems through standardized tool interfaces. We establish a safety-gated evaluation framework in which a trajectory succeeds only when all task goals are achieved without violating any hard safety constraint, while safe goal progress and trajectory safety are measured separately. Across 12 models, the highest Tier-2 success rate is 72.6%, while static knowledge performance does not consistently translate into procedural execution. Analysis of 451 failed episodes from 3 representative models identifies recurring failures in procedural completeness, use of state feedback, and long-horizon execution management. These findings motivate state-aware agent orchestration, joint assessment of task completion and trajectory safety, and repeated regression testing. ACOE and AeroCopilotBench provide a reproducible foundation for testing knowledge application, interactive execution, and operational safety in aviation agents.