Or how to replace 309 unit tests with fuzzing and reach 74% code coverage in 25 seconds
While unit testing is crucial for improving code quality and reducing later testing time, it consumes at least 15% of developers' time. Developers can utilize automated fuzz tests to allocate more time for developing new features. They replace negative test cases, constituting around 30% of unit tests. In a recent analysis of a Java project using a fuzzing platform, a single fuzz test was equivalent to potentially 309 unit tests, achieving 74% code coverage within just 25 seconds.
Why software engineering teams need unit testing
According to The State of Developer Ecosystem 2023 Report, unit tests are the most common type of tests developers use—64% of projects have unit testing. The goal of unit testing is to verify that individual components or units of software behave as expected and meet the specified requirements. An individual component may be either an individual function or a procedure.
Developers usually conduct unit testing at an early stage of development before the code is integrated and tested as a whole system. This allows developers to quickly identify and fix any issues early in the development process, improving the overall quality of the software and reducing the time required for later testing.
Testing Pyramid
In SDLC or V Model, unit testing is the first level of testing done before integration testing.
Two types of testing
There are two types of tests — positive ("test to pass") and negative tests ("test to fail").
Positive testing verifies that a particular functionality or feature behaves as expected when provided with valid input or when the system is in an expected state. These tests focus on validating the intended behavior of the software under normal or expected conditions. Positive unit tests typically include scenarios where inputs are within the valid range and expected outcomes are known.
Negative testing validates how the software handles unexpected or invalid inputs, exceptional conditions, and error scenarios. By intentionally providing invalid or unexpected input, these tests aim to uncover potential vulnerabilities, weaknesses, or bugs in the software.
Negative tests in unit testing include scenarios such as passing null values, out-of-range inputs, or malformed data to the component under test. Negative testing aims to ensure that the software handles errors, avoids unexpected crashes, and provides appropriate error messages or error-handling mechanisms.
While there's no strict rule for the exact split between negative and positive test cases, a commonly suggested guideline is the 70-30 rule, where around 70% of test cases are positive (valid input) and around 30% are negative (invalid or edge cases). However, this split can vary depending on the specific requirements and characteristics of the software being tested.
How much time is needed to write unit tests
The time spent on unit testing varies by project, company size, and testing strategy. Studies and forums confirm that the time invested is different, but they all agree that testing requires a significant time investment.
Developers have to think about what potentially could go wrong, identify all the relevant unit test cases, find missing ones without duplicating or overlapping existing ones, and finally, write tests. Then, time is needed to maintain unit tests. Every code change may require existing unit tests to be updated or rewritten to reflect the changes in the codebase.
On top of that, developers have to do regression testing to prevent the emergence of old bugs by introducing new changes. As a result, the number of tests is constantly growing.
A 2019 survey showed that software engineers spend 15% of their time on testing.
Source: How Much Time Do Developers Spend Actually Writing Code?
On forums, developers share that they spend 1-3 days writing unit tests for every feature:
“In a company I used to work for, executives insisted that the code coverage with unit tests must be 99% or more. This resulted in writing more tests than code. It took us literally 3 days to write tests for a single class that took a day to implement”. Source
“In our product group we target 50-70% code coverage from unit tests and 90%+ coverage from unit tests and test automation combined. Typical time budgeted on writing unit tests is about 1 day for every feature that takes 3-4 days of heads down coding. But that can vary with a lot of factors”. Source
“We spend 70% of time development time in writing unit test(c++ gtest), Are we right?” Source
How to replace negative test cases with fuzz testing
Tech companies like Microsoft and Google were early adopters of fuzzing technologies to test their own systems. Since launching in 2016, Google's OSS-Fuzz, a free fuzzing platform for critical open-source projects, has helped fix over 8,800 vulnerabilities and 28,000 bugs across 850 projects.
Fuzz testing is a form of negative testing as it investigates how a program should definitely not behave, by automatically generating a ton of invalid inputs and seeing if any of them will trigger undesired behavior.
Fuzz testing can take on checking negative test cases, so developers can focus more on positive cases in unit testing and free some time for developing new features.
In April 2024, Code Intelligence released a testing overview dashboard in their fuzz testing platform - CI Sense. New dashboard allows developers to monitor the performance of fuzz tests in real time, including such KPIs as the number of findings, code branches tested, auto-generated inputs, code coverage, and unit test equivalents.
“Unit Test Equivalent” clearly indicates the time saved on writing and maintaining unit tests by running a single fuzz test. It calculates the number of specific unit tests needed to check all distinct paths in the code tested. These unit tests need not be written or maintained as long as developers execute fuzz tests.
In a recent Java project analyzed by the fuzzing platform, it was found that one fuzz test was equivalent to 309 unit tests, achieving 74% code coverage within 25 seconds. On average, fuzz testing can replace approximately 30% of unit tests, particularly those designed for negative testing.
"With this feature, every user of Code Intelligence's fuzz testing platform can easily measure cost savings by reducing the need for unit tests. They now require less effort to achieve the same, or typically higher, code coverage," says Werner Krahe, VP of Product at Code Intelligence.
Tracking test performance also allows developers to quickly spot testing gaps and ensure deep testing is done for critical parts of the software even before the standard pipeline tests are executed.
Fuzzers can work as a JUnit extension, so that it behaves like a JUnit-based test.
As fuzz testing by Code Intelligence analyzes source code, it identifies critical issues early in the testing process and pinpoints them to the exact code line in the repository. This helps to avoid the classic debugging problem: “I know something is broken, but I don’t know where.”
All uncovered bugs are pinpointed to the exact line of code in the repository
All discovered issues are consistently subjected to regression testing as a standard practice. When an issue is identified, there's no need for additional unit tests. Code Intelligence automatically checks to ensure the same problem won't recur, alerting you if it does.
Discover more about fuzzing and Code Intelligence
Read more about fuzz testing or book a free consultation to discuss how fuzzing can fit into your testing strategy.