All software has bugs, and some can be difficult to find or reproduce. However, not all approaches to bug-finding need to be difficult to use!
Fuzzing is an undeniably effective approach to finding security issues and bugs in software projects, however, tools can be complex to set up and execute. CI Fuzz (open-source), automates the parts that make fuzzing complex, giving its users the look and feel of a unit test. Actually, it integrates into popular unit testing frameworks such as JUnit, allowing you to call a fuzz test by simply using @FuzzTest instead of @Test.
What to Expect:
- Why Fuzz Java?
- What Kind of Bugs Can Fuzzing Find in Java?
- Example: Out-of-Memory (OOM) Exception Found Using CI fuzz
Why Fuzz Java?
Fuzz testing approaches are very popular among the C/C++ community, where manual memory management issues can cause significant errors and bugs, such as program crashes or accidental access to memory. These issues can have significant impacts, particularly in embedded systems where software written in C/C++ controls hardware (e.g., industrial machinery).
However, this does not mean that C/C++ is the only example where fuzzing is useful. While most memory corruption issues are not a problem for memory-safe languages such as Java, there are a variety of other threats that fuzzing can help to eliminate. These bugs can be frustrating and have a significant impact on the usability of applications. In some cases, they can even be dangerous, as attackers could use them to shut down services and cause damage to software and reputation. There are plenty of examples of heavy-hitting Java issues that were found with fuzz testing, such as the infamous log4j, or the RCE that was recently discovered in the HyperSQL database (CVE-2022-41853).
What Kind of Bugs Can Fuzzing Find in Java?
Java is popular mainly because of two reasons: Java code is effectively cross-platform (code can be written and executed on most main operating systems) and it has automatic memory management. While these two features greatly help in testing and debugging Java applications, there are still often very severe defects found in Java code.
Issues such as stack overflows, out-of-memory exceptions and many other severe security vulnerabilities can be very problematic for Java applications. Finding these issues can sometimes be tricky, even with good unit testing practices. This is where fuzz testing tools such as CI Fuzz can help.
Open-Source Fuzzing With CI Fuzz
Fuzz testing and unit testing are complementary testing methods that should be used together to ensure that an application functions correctly and securely. Unfortunately, fuzz testing can be complex and generally requires specialized knowledge, making it difficult for developers to incorporate it into their testing workflow.
To make fuzz testing more accessible, we developed CI Fuzz, a tool that integrates with popular unit testing frameworks like JUnit. This integration allows developers to run fuzz tests using @FuzzTest, similar to @test in unit tests. By using this integration, developers can easily incorporate fuzz testing into their existing unit testing setup, ensuring that their code is thoroughly tested and free of bugs and vulnerabilities.
This brings security testing right into the environments that devs use to build their code and makes it just as easy as a unit test. Developers can now test their code for both security and functionality issues in the same place, using a few simple commands. Below, I added a short example to give you an idea of how it works.
Example: Out-of-Memory (OOM) Exception Found Using CI fuzz
Let’s start with a simple example to illustrate how CI Fuzz works. Consider this Analyzer class, which is part of a Java Maven project:
public class Analyzer {
public void complexProcessing(int[] input) {
ArrayList<Integer> integers = new ArrayList<>();
// copy the array
int pos = 0;
while (pos >= 0 && pos < input.length) {
integers.add(input[pos]);
pos = input[pos] + 1;
}
// process the copy
}
}This class has a single method, complexProcessing, that takes an array of integers, makes a copy of it, and does some complex processing with the copy. This is a fairly simple example, but it is indicative of how particular bugs may appear in code in subtle ways.
In the above example, an array of integers is the method’s input, and a copy is made using an ArrayList type. There is also a small but significant bug. Can you see it?
Let’s try writing a fuzz test using cifuzz to try to find this bug.
Step 1: Initialize cifuzz
Assuming that we have installed cifuzz, we start by initializing cifuzz for this project, by running the following command from the CLI:
cifuzz initThis command will create a configuration file called cifuzz.yaml at the root of our project. This configuration file can be used for various settings, but for now, we’ll use the default configuration provided. Since our example is part of a Maven project, running cifuzz init will also provide dependencies we can add to our Maven pom.xml:
Please make sure to add the following dependency to your
pom.xml to enable fuzz testing:
<dependency>
<groupId>com.code-intelligence</groupId>
<artifactId>jazzer-junit</artifactId>
<version>0.13.0</version>
<scope>test</scope>
</dependency>
We highly recommend using cifuzz with JUnit >=5 to ensure
easy IDE integration. You can add it with the following
dependency to your pom.xml:
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.9.0</version>
<scope>test</scope>
</dependency>
For Jacoco coverage reports, please make sure to include
jacoco in your pom.xml, for more info see:
https://www.jacoco.org/jacoco/trunk/doc/maven.html
Also, please add the following profile in your profiles:
<profile>
<id>cifuzz</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<formats>HTML,XML</formats>
<outputDirectory>${cifuzz.report.output}</outputDirectory>
</configuration>
</plugin>
</plugins>
</build>
</profile>After updating the pom.xml, we can create a test.
Step 2: Create a Fuzz Test
Using CI Fuzz, fuzz testing can be done within the JUnit testing framework, similar to unit tests. Let’s create one with this command:
cifuzz createCalling this will create a fuzz test class that looks like this:
package .;
import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;
class MyClassFuzzTest1 {
@FuzzTest
void myFuzzTest(FuzzedDataProvider data) {
// Call the functions you want to test with the provided data and optionally
// assert that the results are as expected.
// If you want to know more about writing fuzz tests you can checkout the
// example projects at https://github.com/CodeIntelligenceTesting/cifuzz/tree/main/examples
// or have a look at our tutorial:
// https://github.com/CodeIntelligenceTesting/cifuzz/blob/main/docs/How-To-Write-A-Fuzz-Test.md
}
}There are two important points to note about this generated class. First, the Maven dependencies added when initializing cifuzz are present here in the two import statements near the top of this class. Second, this code has one test method called myFuzzTest that starts with no logic but makes use of these dependencies.
Let’s remove some unnecessary lines of code, update the name of the test to accurately reflect our test case, and add code to fuzz the complexProcessing method:
import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;
class AnalyzerFuzzer {
@FuzzTest
void fuzzComplexProcessing(FuzzedDataProvider data) {
Analyzer underTest = new Analyzer();
underTest.complexProcessing(data.consumeInts(10));
}
}This is a straightforward fuzz test with two important lines added, one that creates an instance of the Analyzer class for testing and another line that makes use of the data FuzzedDataProvider to generate fuzzy inputs for testing. We’ve also renamed the class from MyFuzzTest1 to AnalyzerFuzzer, which makes the class clearer in its intent.
Step 3: Run the Fuzz Test
Now let’s run the test using:
cifuzz run AnalyzerFuzzerRunning the test provides one finding. In this case, the finding is an Out-of-Memory Exception being thrown by the complexProcessing method. This finding is traced back to the line:
pos = input[pos] + 1;This line causes the out-of-memory issue, as it should be
pos = pos + 1;This bug is subtle, but CI Fuzz is able to fuzz this class effectively to find this issue. What’s even more interesting is that while this bug appears somewhat obvious, it is very unlikely that it would have been found by basic unit testing. Consider this unit test:
@Test
public void checkSizes(){
Analyzer underTest = new Analyzer();
List expected = Arrays.asList(0,1,2); List actual = underTest.complexProcessing(new int[]{0,1,2});
Assert.assertEquals(expected.size(), actual.size());
}This test checks whether the processed list has the same size as the inputted list, and it passes. This unit test may give an impression that the complexProcessing method has no major issues, but it turns out that there is a severe bug. Fuzzing finds this directly.
Conclusion
Fuzz testing is undisputedly effective at uncovering security issues. Due to its complexity, operating a fuzzer used to require tons of experience and domain knowledge. The motivation behind CI Fuzz was to make fuzz testing as unit testing. By bringing fuzz testing into JUnit, most of the complexity that was hindering people from fuzzing their code is basically alleviated. So feel free to give it a spin and let us know how it went