We’re thrilled to announce the general availability of Spark, an AI Test Agent that lowers the entry barrier to white-box fuzz testing. In this blog, we explain how Spark works and share the main results from its beta testing that prove its effectiveness.
Why AI is needed in fuzz testing
White-box fuzz testing has proven highly effective in finding critical bugs and vulnerabilities. Tech giants like Google and Microsoft uncover thousands of issues using this method. But why doesn’t every company adopt white-box fuzz testing as part of their testing strategy? The main barrier is the high level of manual effort and the extensive time required to properly set it up and maintain it. You first need to identify what you should test, then implement corresponding fuzz tests, run them, and address the identified bugs.
However, with LLMs' rapidly improving coding and reasoning capabilities, a large part of the process can now be automated.
How Spark automates fuzzing from start to findings
Spark by Code Intelligence is the first AI test agent that autonomously identifies bugs and vulnerabilities in unknown code without human interaction. It’s the first AI Agent to find a real-world vulnerability by automatically generating and running a test for a widely used open-source software.
With Spark, this is how you can do white-box fuzzing now:
- Set up a testing goal - how much code coverage you want to achieve.
- Launch Spark.
- See issues and vulnerabilities hidden in your code.
In less than 2 minutes, Spark generated 3,000 test cases for five targets (fuzz test candidates), found two issues, and reached 79.51% of code coverage
When you launch Spark, it operates as follows under the hood:
1. Analyzes your codebase and identifies the most important functions and APIs to fuzz. Each fuzzing entry point is scored based on its expected impact using four key metrics.
2. Generates fuzz tests (harnesses) for each function by leveraging static code analysis to extract the correct context needed to test the target function and large language models (LLMs) to create and optimize the fuzz test.
3. Runs and validates the generated tests to ensure they can be built and run correctly and achieve high code coverage.
4. Flags all findings, providing details such as the exact lines of code where issues occur, stack traces, and triggering inputs to assist in root cause analysis.
“We were truly impressed by the abilities of Spark to enhance our fuzz testing workflows. By reducing the manual effort for creating and integrating fuzz tests, we are able to bring our cycle time down and further improve the quality of our embedded software.”
Andreas Lackner, Senior Software Development Engineer at Vector Informatik
How Spark uncovered a real-world vulnerability and reached up to 45% higher code coverage in 1 hour
During its beta testing, we used Spark to fuzz 8 open-source projects for 1 hour each. These projects have been continuously fuzzed in OSS-Fuzz. As a result of 1 hour of autonomous fuzzing with Spark, the achieved code coverage was higher up to 44.7%, and three issues were identified. For 2 of the projects, code coverage was higher than 70%.
Code coverage results achieved in OSS-Fuzz vs. by 1 hour fuzzing with Spark
One of the uncovered and fixed vulnerabilities was in WolfSSL, an open-source cryptography library widely used in developing embedded devices and IoT systems. The vulnerability was a heap-based use-after-free. The only human involvement in finding it was launching the spark command; analyzing the code, generating a relevant test case, and running it was done autonomously.
How to start using Spark by Code Intelligence
If you’re looking to incorporate white-box fuzzing into your codebase and simplify the process for your security and development teams, book a free call with our experts for a tailored demo of Spark.
To start using Spark, you’ll need to:
- Install CI Fuzz in its enterprise package.
- Ensure access to large language models (LLMs).
During the call, Code Intelligence’s experts will guide you through the setup process, share industry best practices, and provide insights on pricing options.