Large language models repair web app access errors with growing success
Repairing Broken Access Control in Web Applications with LLMs: An Empirical Benchmark Study
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
Summary
Broken access control is a common and serious security problem in web applications, where users might see or change things they shouldn't. Traditional automated methods struggle to fix these complex logic errors because they require understanding how the app works. The paper’s authors created a benchmark with real-world examples to test how well large language models (LLMs) can fix these errors. They found that LLMs, especially when guided with certain strategies, can fix many access control problems, sometimes even improving on original patches. This shows promise for using AI to help secure web apps better.
What this means in practice
- •For web application developers: Automatically generate patches for access control bugs in web applications using large language models guided by repair strategies.
- •For security operations teams: Validate and fix access control vulnerabilities more efficiently by integrating LLM-based patch suggestions into vulnerability management workflows.
Authors
Mingrui Liu, Liang He, Xinchu Wang, Yeting Li, Dongxu Han, Yunpeng Li, Yuling Liu
Abstract
Broken access control (BAC) ranks among the most severe web-application vulnerability classes, yet its automated repair remains largely unexplored. Unlike memory-safety vulnerabilities, BAC is a logic vulnerability whose repair demands a deep understanding of an application's access-control model and business semantics, which traditional automated vulnerability repair techniques handle poorly. Large language models (LLMs) open a promising avenue for this problem, but no standard benchmark exists for studying how well they repair BAC vulnerabilities. We construct BAC-Bench, the first benchmark for BAC repair, consisting of 46 real-world BAC vulnerabilities drawn from six widely used web applications spanning four programming languages. Each vulnerability is equipped with a sandboxed runtime environment, a PoC validation program and a functional test program. We further propose a BAC repair-strategy framework that characterizes any BAC patch along four dimensions, Position, Coverage, Mechanism, and Content. Evaluating eight base LLMs and two commercial LLM agents on BAC-Bench, we find that single-turn base LLMs repair 37.68% of the vulnerabilities on average, while two minimal scaffolds, Apply-Feedback and Navigate-and-Locate, raise this average to 66.03%, with the best base LLM reaching 75.36%, close to the 78.99% of the best agent. Notably, some LLM-generated patches surpass the official patches, and one incidentally closed a zero-day vulnerability that had lain dormant for three years. Our findings delineate the current capability boundary of LLM-based BAC repair and offer guidance toward its practical adoption.