Language models show promise and limits in Hong Kong fire code tests
Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry
Computational Engineering, Finance, and ScienceArtificial Intelligence
Summary
Understanding if computer programs called Large Language Models can read and follow fire safety rules is important for building safety. The authors made a test specifically for Hong Kong’s fire safety laws using questions from experts. They found the best models can get most multiple-choice questions right, but when asked open-ended questions, they struggle more. This means even the best models still need work before they can fully help with legal safety checks.
What this means in practice
- •For construction compliance teams: Evaluate and improve the use of language models to assist in interpreting Hong Kong fire safety regulations for ensuring building code adherence.
- •For legal advisory firms: Provide better insights into the strengths and weaknesses of language models in automating interpretation of specialized regulatory documents.
Authors
Le Qiu, Zhuan Anne Zhang, Hok Ming Adeline Chan, Emmanuele Chersoni
Abstract
Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.