Build Secure Java Applications with Fuzz Testing

Google Found 25 000 Bugs in Chrome With Fuzzing

Fuzzing (or fuzz testing), which can be seen as a complementary approach to unit testing, is currently gaining a lot of popularity in the open-source community. With the same testing approach, engineers at Google have already found more than 25,000 bugs in Google Chrome. The Chromium team has nearly fully scaled and automated its security testing with this testing approach. 

Google recommends open-source developers to use fuzz testing to secure java software supply chains, as this approach has proven particularly effective in detecting memory issues and memory-safe bug classes, such as code execution, cross-site scripting, and regex injections. 

I will now show how to apply fuzz testing to your own code.
But let's start with the basics.

What Is Fuzz Testing, and How Does It Help Secure Java Applications?

Fuzz Testing is a dynamic testing method that can be used to secure java software and to find functional bugs. 

During a fuzz test, a program or function under test is executed with millions of random, unexpected, or invalid inputs in order to crash the application, like in an automated pentest designed to test the reliability of your code. 

Feedback-based Fuzzing

In contrast to other secure coding practices in Java, such as static application security testing, the code actually gets executed. On top of that, the fuzzer receives detailed feedback about the structure of the code and the states reached inputs.

With this information, the fuzzer can adjust and mutate its dynamic inputs to produce additional test cases that are highly likely to trigger even more errors and crashes in the system under test. This entire process is fully automated and produces almost no false positives, as the fuzzer always provides the error source and the malicious input that crashes the application.

In summary, this is the perfect testing method to increase the quality and security of your code with next to no effort.

What Are Good Use Cases for Fuzz Testing? 

In general, any library or application that handles untrusted or complicated data is well suited for fuzz testing. For example:

• Media Codecs
• Server-side web development
• Mobile development
• Cryptography
• Compression
• Regular Expression Matchers
• Text/UTF processing
• Database transactions/ETL Processing
• Text Editors/Processors

What Bugs Can You Find With Fuzzing in Java Applications 

With fuzzing, you can find all sorts of bugs and vulnerabilities that can cause your application to crash, for example: 

Availability Issues and Regressions

• Denial of Service (DoS) 
• Infinite Loop
• Security Misconfiguration 
• Insecure Deserialization 
• Using Components with Known Vulnerabilities 
• Uncaught Exceptions

Data Validation Errors

• Injection 
• Exposure of Sensitive Information to an Unauthorized Actor 
• XML External Entities (XXE) 
• Sensitive Data Exposure
• Generation of Error Message Containing Sensitive Information 
• Cross-Site Scripting XSS 

Logic Issues

• Logic issue: bypass security features 
• Audit/Logging Errors
• Logging of Excessive Data 
• Insufficient Logging
• Insufficient Logging & Monitoring
• Broken Authentication
• Broken Access Control

Cookie Issues

• Sensitive Cookie with Improper SameSite Attribute 
• Sensitive Cookie Without 'HttpOnly' Flag 
• Sensitive Cookie in HTTPS Session Without 'Secure' Attribute

With fuzzing, you can secure java applications by uncovering and fixing issues such as Injections, Cross-Site-Scripting, and other bugs from the OWASP Top 10, but you may have to feed your fuzzer with additional information about the structure of your code before running your first fuzz test. 

For example, if a fuzzer generated SQL statements that call trusted methods from the Java classpath, your application would potentially be vulnerable to remote code executions. This scenario is quite common for web applications, but the inputs that trigger this vulnerability do not necessarily cause the application to crash. So, how can a fuzzer know which inputs are allowed to call trusted methods? 

To reliably trigger this vulnerability with a fuzz test, you would actually need to tell the fuzzer which inputs are good and which inputs are potentially dangerous. 

Fuzz Testing Tools for Secure Java Applications

If you are new to fuzz testing, I would recommend starting with an easy-to-use open-source fuzzer like CI Fuzz. This tool will make it easy for you to write fuzz smaller projects even if you don't have a ton of experience with fuzzing. If you know how a unit test works you'll be able to use it.

Fuzz Testing in JUnit to Secure Java Applications

Now that you have learned about the basics of fuzz testing, let's have a look at how to perform fuzz testing in practice. In the next chapter, I will show you how to set up and run a fuzz test on a Java application with CI Fuzz. 

CI Fuzz is an easy-to-use fuzzing tool that enables you to integrate fuzz tests into the JUnit testing framework to fuzz your code directly from your command line. Similar @Test CI Fuzz only requires you to call @FuzzTest within Junit to start fuzzing. From there on, it only takes three simple commands.

# Initialize fuzzing
$ cifuzz init

# Create your first fuzz test
$ cifuzz create my_fuzz_test

# Run fuzz test and find bugs
$ cifuzz run my_fuzz_test

CI Fuzz runs on Linux, macOS, and Windows and supports integrations with common build systems and IDEs. If you want to test a Java application with CI Fuzz, the only prerequisites are Java JDK (e.g., OpenDesk or Zulu) and a build system of your choice. 

sh -c "$(curl -fsSL https://raw.githubusercontent.com/CodeIntelligenceTesting/cifuzz/main/install.sh)"

In the README, you will find detailed instructions on how to install the tool.

Fuzzing Use Case: How to Find Bugs in a Java Library

This example library has multiple branches that are reached under different conditions. It contains a potential remote code execution, where an attacker could inject a string and overwrite the settings.

ExploreMe.java
package com.example;
public class ExploreMe {
  // Function with multiple paths that can be discovered by a fuzzer.
  public static void exploreMe(int a, int b, String c) {
      if (a >= 20000) {
          if (b >= 2000000) {
              if (b - a < 100000) {
                  // Create reflective call
                  if (c.startsWith("@")) {
                      String className = c.substring(1);
                      try {
                        Class.forName(className);
                      } catch (ClassNotFoundException ignored) {
                      }
                  }
              }
          }
       }
   }
}

Let's take a look at how to write a fuzz test with CI Fuzz that triggers this remote code execution.

How to Set Up a Fuzz Test in 3 Easy Steps

How to set up CI Fuzz CLI in Gradle

CI Fuzz commands will interactively guide you through the needed options and give you instructions on what to do next. You can find a complete list of commands with all options and parameters by calling cifuzz command --help.

1. Initialize the Project for CI Fuzz

The first step is to initialize the project you want to test with CI Fuzz. 

This step will vary depending on which build system you are working with, as you will need to modify relevant build files, either pom.xml for Maven projects or build.gradle for Gradle projects. Creating a fuzz test is the same for each build system. 

To initialize a Java Maven project, first run cifuzz init in the root directory of the project. This will create cifuzz.yaml in the root directory of the project, the primary configuration file for CI Fuzz. 

CI Fuzz will also list the dependencies you need to add to your pom.xml for your project:

<dependency>
  <groupId>com.code-intelligence</groupId>
  <artifactId>jazzer-junit</artifactId>
  <version>0.13.0</version>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter-engine</artifactId>
  <version>5.9.0</version>
  <scope>test</scope>
</dependency>

2. Create a Fuzz Test

To create a fuzz test, cifuzz create java. This will create a fuzz test stub in the current directory. You can also use the -o flag to specify a path to create the fuzz test. 

You can put the fuzz test anywhere, but we recommend you keep it close to the tested code as you would a regular unit test. In the example maven project in the cifuzz repository, the fuzz test is created in src/test/java/com/example/FuzzTestCase.java.

Here is that fuzz test:

FuzzTestCase.java
package com.example;
Import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;
public class FuzzTestCase {
    @FuzzTest
    void myFuzzTest(FuzzedDataProvider data) {
        int a = data.consumeInt();
        int b = data.consumeInt();
        String c = data.consumeRemainingAsString();
        ExploreMe.exploreMe(a, b, c);
    }
}

If you are familiar with unit tests in Java, this should look mostly familiar. A few things to note about this fuzz test:

• The fuzz test is in the same package as the target class
• The @FuzzTest annotation identifies this method as a fuzztest method
• The FuzzedDataProvider is part of the Jazzer API package and makes it easy to split fuzzing input into multiple parts and various data types.
• The name of this fuzz test, as cifuzz recognizes it, is com.example.FuzzTestCase. So when you want to run this fuzz test, then run cifuzz run com.example.FuzzTestCase from the project directory.
  
  

3. Run the Fuzz Test

Start the fuzzing by executing cifuzz run FuzzTestCase. CI Fuzz now tries to build the fuzz test and starts a fuzzing run.

$ cifuzz run FuzzTestCase 
[...]

Use ‘cifuzz finding <finding name>’ for details on a finding. 

💥[awesome_gnu] Security Issue: Remote Code Execution in exploreMe (com.example.ExploreMe:13)

Note: The crashing input has been copied to the seed corpus at: 
    src/test/resources/com/examples/MyClassFuzzTestInputs/awesome_gnu

It will now be used as a seed input for all runs of the fuzz test, including remote runs with artifacts created via ‘cifuzz bundle’ and regression tests. For more information, see:
    https://github.com/CodeIntelligenceTesting/cifuzz#regression-testing

Execution time: 3s
Average exec/s: 316880
Findings: 1
New seeds: 5 (total: 5)

Coverage Reporting and Debugging

What you can do now is take a closer look at the findings. cifuzz findings shows you a list of all findings and cifuzz finding [name] will show you detailed information about a particular finding, which is useful to understand and fix a bug.

Here we have the stack trace for the remote code execution:

[awesome_gnu] Security Issue: Remote Code Execution in exploreMe (com.example.ExploreMe:13)
Date: 2022-11-10 13:31:07.94532426 +0100 CET

 == Java Exception: com.code_intelligence.jazzer.api.FuzzerSecurityIssueHigh: Remote Code Execution

Unrestricted class loading based on externally controlled data may allow
remote code execution depending on available classes on the classpath.
 at jaz.Zer.<clinit>(Zer.java:54)
 at java.base/java.lang.Class.forName0(Native Method)
 at java.base/java.lang.Class.forName(Class.java:375)
 at com.example.ExploreMe.exploreMe(ExploreMe.java:13)
 at com.example.FuzzTestCase.myFuzzTest(FuzzTestCase.java:13)
 == libFuzzer crashing input ==
 MS: 0 ; base unit: 0000000000000000000000000000000000000000

0x40,0x6a,0x61,0x7a,0x2e,0x5a,0x65,0x72,0x0,0x1,0x0,0x40,0x0,0x0,0x0,0x75,
 @jaz.Zer\000\001\000@\000\000\000u
 artifact_prefix='./'; Test unit written to 
 ./crash-f29a1020fa966baaf8c2326a19a03b73f8e5a3c9
 Base64: QGphei5aZXIAAQBAAAAAdQ==

We also get a report of the CLI, how many lines and functions were found, and how many of the branches were covered.

Coverage Report
               File |  Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found
  FuzzTestCase.java |         2/2 (100.0%) |     9/9 (100.0%) |       0/0 (100.0%)
src/explore_me.java |         1/1 (100.0%) |   23/23 (100.0%) |       8/8 (100.0%)
                    |                      |                  | 	
                    |  Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found                                      
              total |                  3/3 |            32    |                8/8

If you can, optimize your fuzzer for code coverage. The idea behind this recommendation is to enable the fuzzer to explore your application by taking as many different paths through your code as possible. This way, the fuzzer will be able to learn if a specific data structure or inputs are required to execute the code, and the more information the fuzzer gets about the structure of the code, the more effective your fuzz tests will be. 

Closing Thoughts

Getting started with fuzzing can be challenging, as many open-source fuzzers require a lot of setup and knowledge to get your first fuzz test up and running. But truly believe that CI Fuzz and its JUnit integration will make it easier for you to get started with fuzzing, as it's really one of the most effective (and fun) ways to test for integrity issues, reliability issues, and other vulnerabilities.