Jazzer is a coverage-guided fuzzer for the Java Virtual Machine (JVM). It works on the bytecode level and can thus not only be applied directly to compiled Java applications, but also to targets in other JVM-based languages such as Kotlin or Scala.
Jazzer consists of two main components:
- The Jazzer driver is a native binary that links in libFuzzer and runs a Java fuzz target through the Java Native Interface (JNI).
- The Jazzer agent is a Java agent that runs in the same JVM as the fuzz target and applies instrumentation at runtime. The information obtained through this instrumentation is fed back to libFuzzer just as it would be for a compile-time instrumented native binary.
https://github.com/CodeIntelligenceTesting/jazzer/
Taken together, the driver and the agent make it seem to libFuzzer as if it were fuzzing an ordinary native binary. Our goal is to make all of libFuzzer's features work with Jazzer out of the box. In the following, we want to highlight some of the most interesting aspects of the engineering effort required to achieve this goal.
Update: Google integrated Jazzer into OSS-Fuzz. Now open-source projects can use Google's infrastructure and computing power to secure their Java libraries. Read the full release note in the Google Security Blog.
Update 2: Jazzer is now running in CI Fuzz, a fuzzing solution that lets developers fuzz their code with a few simple commands, straight from the command line. CI Fuzz integrates with the JUnit testing framework, making fuzz testing as easy as unit testing.
Instrumenting JVM Bytecode for Fuzzing
The JVM bytecode of a compiled Java class is much higher-level than the machine code of an equivalent C/C++ binary. This makes it possible to instrument a JVM application at runtime rather than at build-time, which means that neither the code nor the build script requires any changes. The Jazzer agent, which is written in Kotlin, instruments the classes right as they are being loaded by the JVM.
To obtain coverage information from the JVM fuzz target, the Jazzer agent inserts bytecode instructions together with a unique ID at the beginning of every basic block. When we enter a block, the previous basic block’s shifted ID is XORed with the current ID to obtain an identifier for the control flow edge between the two blocks. This idea to obtain edge-level coverage information from basic block instrumentation originated in AFL and has already featured in the early AFL-based Java fuzzer Kelinci. By relying on memory shared directly between the JVM and the Jazzer driver through a direct ByteBuffer as well as implementing a branch-free saturating counter increment, we found this approach to coverage instrumentation to be very efficient in practice (take a look at AFLCoverageMapInstrumentor.kt if you want to know how this is done).
Since coverage is not the only type of information that is used by libFuzzer to guide its exploration of the fuzz target, Jazzer also instruments other JVM constructs (see TraceDataFlowInstrumentor.kt):
- bytecode-level compares, such as the
lcmp,if_*, andif*opcodes
- higher-level method-based compares, such as
String#equalorArrays#compare
switchstatements (corresponding to thelookupswitchandtableswitchopcodes)
- integer divisions (corresponding to the
idivandldivopcodes and thedivideUnsignedmethods)
- constant array indices (corresponding to the
*aloadopcodes andget*methods on array-like objects)
In all these cases, interesting observed values are forwarded to libFuzzer callbacks such as __sanitizer_cov_trace_cmp4 (for comparisons of 32-bit integral types) and __sanitizer_weak_hook_strcmp (for string comparisons). The values populate libFuzzer's table of recent compares and will be incorporated into the fuzzer input bytes via random mutations.
Value Profile Support
The table of recent compares is usually quite effective at letting the fuzzer progress through simple compares in which the required input can be obtained directly by inserting bytes reported via the compare hooks. But real-world fuzz targets are rarely that simple. For example, a function might encode the input into Base64 or apply a simple, non-cryptographic checksum function, followed by a comparison. In these cases, the relation between the input required to explore the fuzz target further and the bytes reported via callbacks is highly non-trivial.
libFuzzer's value profile, which can be enabled with the runtime flag -use_value_profile=1, can be very helpful in this situation. In this mode, for every comparison between two values, an extra set of coverage counters will be incremented for every matching bit position in A and B. This allows the fuzzer to track its partial progress in getting A and B to match, thereby simplifying the problem from brute force (exponential time) to a linear search (linear time).
Since Jazzer builds on libFuzzer and reports string comparisons using __sanitizer_weak_hook_strcmp, basic value profile support is automatic. Allowing the value profile to track comparisons between integral types is a more difficult task as the relevant fuzzer hooks such as __sanitizer_cov_trace_cmp4 do not take the address of the compare instruction as an argument. Instead, these hooks retrieve the return address from the stack via the __builtin_return_address compiler intrinsic.
While this is a very natural approach for instrumented native binaries, our calls to the callbacks all go through the JNI and will thus have the same native return address somewhere in the JNI dispatcher. This would mean that the fuzzer would not be able to distinguish between different compare instructions, which would render the value profile largely useless for non-string types. We initially considered submitting a patch to LLVM that would add variants of the compare callbacks that take a call site address as an additional argument, but then found a way to provide full value profile support even without upstream changes.
A small piece of inline assembly in the Jazzer driver implements a "trampoline" that can invoke any of the libFuzzer callbacks with an arbitray fake return address addr, e.g., a unique ID assigned to a bytecode instruction (see sanitizer_hooks_with_pc.cpp for the implementation). The trampoline first pushes an address pointing to the addr & 0xFFF-th entry in a "sled" of 0xFFF + 1=4096 ASM ret instructions to the (native) stack and then performs a direct jump (also called a "tail call") to the sanitizer callback. The callback will see the address of the ret corresponding to the lower 12 bits of the synthetic address addr as the return address via __builtin_return_address and associate the bit-wise progress made in the comparison with it in the value profile map. Since libFuzzer only uses the lower 12 bits of an address for the purpose of the value profile, this allows for fast and complete value profile support also for integral types.
FuzzedDataProvider
Both libFuzzer and Google’s Python fuzzer Atheris provide convenience functions that transform the raw byte input received from the fuzzer into useful primitive types, such as strings, integers, and arrays. While working on Jazzer's analogue of such a FuzzedDataProvider, we found the two prior implementations to differ in quite a few relevant ways. Adding our own experience with building fuzzers to the mix, we arrived at the following design decisions:
- Jazzer's
FuzzedDataProviderturns raw fuzzer input bytes into valid JVM strings by carefully applying the least number of bit changes required to turn the input bytes into valid UTF-8. This differs from the approach taken by Atheris, which randomly chooses between ASCII, UTF-16 and UTF-32. We found the fuzzing to be more stable if a consistent, ASCII-compatible encoding is used throughout theFuzzedDataProviderand the libFuzzer callbacks. This however comes at the cost of added complexity in the implementation of the string generation functions: The JNI internally uses an encoding that historically predates UTF-8 and is best described as "UTF-16 code points encoded using UTF-8, with special handling of zero bytes". We found this additional complexity to be worth it based on the increased fuzzing efficiency.
- The provider distinguishes between "data byte" and "choice bytes". More complex data which is likely to be parsed by the fuzz target, such as strings or arrays of primitive types, is consumed from the beginning of the fuzzer input. Small primitive types such as single integers, floats, and booleans, which are more likely to guide control-flow, are consumed from the end of the fuzzer input. This idea goes back to libFuzzer's
FuzzedDataProviderand has two key benefits: It makes the consumed fuzzed data more stable under the typical mutations applied by libFuzzerand also ensures that corpus entries preserve the structure of valid input files (e.g., JSON, PDF, or JPEG files) as much as possible.
- Reproducing crashes in fuzz targets using a
FuzzedDataProviderwould be challenging if it always required running the crashing input through the driver. For this reason Jazzer, in addition to the crashing input dumped by libFuzzer, creates a Java-only reproducer for every crash. This reproducer contains all values consumed from theFuzzedDataProviderin serialized form and can be run like any other Java program without any native dependencies.
Custom Method Hooks
Experience shows that roughly 65% of all serious security bugs found in projects using languages without memory-safety guarantees stem from incorrect memory handling. For the most part this high percentage is likely explained by how difficult memory handling can get if memory-safety is not built into the language from the ground up. However, there is another part to this story: With the likes of AddressSanitizer, MemorySanitizer, and UndefinedBehaviorSanitizer, there is excellent tooling available to detect such bugs with fuzzing.
While there is no need to detect violations of memory-safety while testing Java applications (at least those without native libraries), other vulnerabilities such as SQL injections, path traversals, or infinite loops occur in JVM-based languages just as often as they do in C/C++. One of the core ideas behind Jazzer is that the higher-level structure of JVM bytecode compared to native binaries coupled with suitable "sanitizers" should make these bugs detectable via fuzzing. For this purpose, Jazzer includes a flexible annotation-based mechanism to hook methods, regardless of whether these are defined in user-supplied code, a dependency, or even the Java standard library. Jazzer itself uses this hooking mechanism to instrument not only bytecode-level compare instructions, but also standard library functions such as String#equals and String#replace.
The hooking mechanism is not only useful for providing feedback to the fuzzer, but can also be used to create tailor-made sanitizers. For example, the following method hook could serve as the basis for a "path traversal sanitizer":
class PathTraversalSanitizer {
@MethodHook(type = HookType.AFTER, targetClassName = "java.io.File",
targetMethod = "<init>", targetMethodDescriptor = "(Ljava/lang/String;)")
public static void fileConstructorHook(MethodHandle method, Object thisObject, Object[] arguments, int hookId, Object returnValue) {
File file = (File) thisObject;
String pathname = (String) arguments[0];
try {
// Check whether the canonical path of `file` lies inside a known list of allowed paths.
if (!file.getCanonicalPath().startsWith("/expected/path")) {
// If not, throw a distinctive exception that is reported by Jazzer.
throw new PotentialPathTraversalException();
}
} catch(IOException e) {
}
}
We believe that the design of sanitizers for the JVM is an interesting space to explore and welcome contributions of any kind to the Jazzer repository on GitHub.
CI Fuzz Enterprise
Aside of open-source tools there are also enterprise solutions, such as CI Fuzz, which aim to solve enterprise problems and integrations relevant to working in development teams. Such features as reporting, CI/CD and dev tool integration, WebAPI fuzzing, OWASP vulnerability detection enable highly productive work in the development process (DevSecOps). If you're interested in fuzzing more complex applications, reach out to one of our security experts. We will walk you through the product and answer your questions.
