Large language models struggle with car safety risk assessments
SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Computation and Language
Summary
The authors found that large language models (LLMs) can write believable descriptions of potential car hazards but have trouble correctly classifying the safety risks according to official standards. They created a big test dataset with 3,000 real vehicle hazard cases to check how well these models do in following car safety rules. They also identified specific reasons why the models fail, such as missing important details and misjudging how controllable risks are. This work helps show where human experts still need to be involved when using AI for car safety checks.
What this means in practice
- •For automotive safety engineers: Test and compare language models' ability to assist with hazard analysis and risk classification in vehicles according to ISO 26262.
- •For industrial ai developers: Use the SAFARI dataset to improve language model training on functional safety tasks in regulated industries beyond automotive.
Authors
Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu
Abstract
Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive Risk Inference), the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified industrial HARA cases and evaluates two coupled tasks: open-ended hazard analysis and standards-grounded risk assessment. To evaluate open-ended HARA artifacts, we propose the first reference-anchored LLM-as-a-judge protocol with high expert correlation. Experiments with nine frontier LLMs show that models often produce plausible hazard narratives but remain weak at ISO 26262 risk classification, with the best ASIL macro-F1 reaching only 0.261. Chain-of-Thought prompting provides limited benefit and often degrades categorical risk assessment. Error analysis further localizes major failures to scenario-critical context omissions during hazard generation and to controllability misjudgments during risk assessment, indicating where expert oversight should be concentrated. The dataset can be obtained from https://github.com/xixi47520-hash/HARA.