Summary
Accident data changes a lot from state to state, making it hard to reuse computer models designed to predict injury severity in crashes. The authors adapted a language model that reads crash details as text to work on Tennessee's crash data and tested different ways of picking data for training and testing. They found that models trained on random or county-based samples performed well overall but missed rare severe crashes, while models trained on balanced data were better at recognizing serious crashes more evenly. This shows that how you choose and split data makes a big difference in understanding a model’s true performance.
What this means in practice
- •For traffic safety analysts: Use adapted language models to better estimate and classify injury severity in state-specific crash databases for targeted safety improvements.
- •For insurance risk assessors: Employ text-based crash severity classifiers fine-tuned to local data to improve risk evaluation and claims handling accuracy.
Tested on one dataset.
Authors
Abhilasha Saroj, Pranav Govindu, Bharat Sharma, Usman Ahmed
Abstract
State crash databases differ in structure, coding, and injury-severity distributions, limiting direct reuse of predictive workflows across jurisdictions. This study adapts the SafeTraffic Copilot large language model (LLM) crash-severity workflow to a three-year Tennessee inventory of 624,392 crashes. Tennessee crash, roadway, vehicle, and person attributes were harmonized and converted into textual prompts while unavailable values were preserved rather than inferred. Llama 3.1 8B was fine-tuned using low-rank adaptation to classify five injury-severity categories. Random, county-based, and severity-balanced sampling strategies were evaluated using separate in-sample and unseen-test experiments; unseen-test experiments used 70/15/15 training, validation, and test splits. On their respective unseen test sets, random and county-based models achieved weighted F1-scores near 80%, but macro F1 remained below 47% and fatal-crash F1 below 27%, showing that strong aggregate performance can mask weak recognition of rare outcomes. Within its balanced evaluation population, the severity-balanced model achieved weighted and macro F1-scores of 57.7% and a fatalcrash F1-score of 65.9%, yielding more even class-level performance. Because each sampling strategy used a different test subset, crossstrategy differences are descriptive rather than controlled rankings. The results highlight the importance of interpreting LLM crashseverity performance together with sampling design, class balance, class-level metrics, and evaluation-population composition.