PilotBench: A Benchmark for General Aviation Agents with Safety Constraints
2026-04-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created PilotBench, a test to see how well language models (like AI that reads text) can predict the paths and attitudes of airplanes, focusing on safety. They found that traditional forecasting methods are more precise in numerical predictions, while language models follow instructions better but are less accurate. The language models struggled most during complex flight phases like climbing and landing, showing limits in understanding physics deeply. The authors suggest combining both approaches to build safer, smarter AI for real-world tasks.
Large Language Modelsaviation telemetryflight trajectory predictionsafety compliancemachine learning benchmarkingmean absolute errorsemantic reasoningembodied AIsymbolic reasoningnumerical forecasting
Authors
Yalun Wu, Haotian Liu, Zhoujun Li, Boyang Wang
Abstract
As Large Language Models (LLMs) advance toward embodied AI agents operating in physical environments, a fundamental question emerges: can models trained on text corpora reliably reason about complex physics while adhering to safety constraints? We address this through PilotBench, a benchmark evaluating LLMs on safety-critical flight trajectory and attitude prediction. Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction through comparative analysis of LLMs and traditional forecasters. We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance. Comparative evaluation across 41 models uncovers a Precision-Controllability Dichotomy: traditional forecasters achieve superior MAE of 7.01 but lack semantic reasoning capabilities, while LLMs gain controllability with 86--89% instruction-following at the cost of 11--14 MAE precision. Phase-stratified analysis further exposes a Dynamic Complexity Gap-LLM performance degrades sharply in high-workload phases such as Climb and Approach, suggesting brittle implicit physics models. These empirical discoveries motivate hybrid architectures combining LLMs' symbolic reasoning with specialized forecasters' numerical precision. PilotBench provides a rigorous foundation for advancing embodied AI in safety-constrained domains.