Rust security tools get better tests with new evaluation methods
Building Sound Evaluation Practices for Rust-Based Security Frameworks
Cryptography and Security
Summary
Rust is a programming language designed to be safe, but security problems still happen in Rust programs. The authors found that testing tools made to find these problems often have flaws in how they are evaluated. They developed a new method to test Rust security tools more reliably, including ways to automatically check coverage of risky code and measure how varied the test programs are. They tried this method on popular Rust testing tools and showed it works well with real Rust projects.
What this means in practice
- •For rust security engineers: Use the new evaluation method to more accurately test Rust security tools and ensure better vulnerability detection.
- •For software testing teams: Adopt this method to improve the assessment of security testing tools for Rust applications, helping prevent missed bugs.
Authors
David Paaßen, Lucas Davi
Abstract
Background: Rust is a modern programming language known for its focus on usability, concurrency, and security. However, empirical studies reveal that Rust is still affected by software vulnerabilities, with hundreds of memory safety issues found in Rust programs, including Rust's standard library. As a result, additional tool-assisted security testing is crucial to enhance the security of Rust code. Evaluating such tools, however, is highly complex. While there is substantial research on evaluating C/C++ security frameworks, including test sets, serious issues in published evaluations have been identified, undermining their scientific soundness and reproducibility. In practice, selecting an inadequate implementation (e.g., a fuzzer) can lead to missed vulnerabilities ultimately compromising a program's security. Aims: Our methodology aims to help researchers to conduct scientifically sound evaluations in order to prevent similar issues with the evaluation of security tools and frameworks in the Rust ecosystem. Method: We systematically analyze existing evaluation literature to develop the first sound methodology for Rust based evaluations. Our guidelines provide enough flexibility to support a variety of use cases. To quantify the performance of a security tool, we introduce an automated approach to inject unsafe code coverage oracles, which require no forward porting, avoid overfitting, and allow updating test sets in the future. We also present methods to quantify the diversity of a Rust test set, ensuring that the results are applicable to a large set of existing Rust programs. We practically apply our methodology in our evaluation of the most popular Rust fuzzers. Results: We show the practical applicability of our design and conduct an evaluation with a diverse test set of 20 Rust projects. [...]