Typed template fuzzing improves software testing across many languages
Typed Template Fuzzing
Software EngineeringProgramming Languages
Summary
Testing software with complicated inputs is hard because those inputs need to follow special rules. The authors created a new method called Typed Template Fuzzing that uses language models just once to prepare templates, then quickly mutates these templates to create many valid tests. This approach works well for many programming languages without extra language-specific tools, finds many bugs, and keeps almost all tests valid. Their tool Typos showed good results by finding many new bugs and maintaining high test validity.
What this means in practice
- •For software testers: Generate many valid and semantically correct test inputs across multiple programming languages to find bugs efficiently.
- •For compiler developers: Use typed template fuzzing to test compiler frontends with diverse language inputs without needing language-specific parsers or checkers.
Authors
Lu Maltsis, Dominik Winterer
Abstract
Fuzzing software with complex inputs has been a longstanding challenge. Traditionally, fuzzers either programmatically encode language constraints, requiring significant engineering effort, or rely on users to declare grammars and constraints. Recent LLM-based fuzzers address these drawbacks but introduce new issues. Some use LLMs for test generation, sacrificing throughput and producing invalid inputs when the LLM makes mistakes. Others rely on language-specific parsers, static analysis, and AST manipulation libraries. We introduce Typed Template Fuzzing (TTF), a lightweight LLM-based fuzzing methodology. The key idea is to use LLMs in a preprocessing step to transform seeds into typed templates. The typed templates then enable semantically aware, mutation-based fuzzing without language-specific parsers, type checkers, or further LLM queries during fuzzing. By exploiting the correspondence between seeds and templates, TTF can moreover repair LLM mistakes. We have implemented TTF in our tool Typos, which we evaluate across 21 input languages, including C, C++, Go, Rust, SystemVerilog, Scala, and Java, as well as libraries such as PyTorch and Qiskit. In a six-month fuzzing campaign, we reported 135 bugs, 128 of which were previously unknown. Perhaps most notably, on average 96% of Typos's mutants are valid, coming close to the validity rates of domain-specific fuzzers. Our evaluation further shows Typos's benefits over the state of the art.