Understanding Out-of-Bounds Memory Access Vulnerabilities and Detecting Them with Fuzz Testing

Out-of-bounds memory access, also known as buffer overflow, occurs when a program tries to read from or write to a memory location outside the bounds of the memory buffer that has been allocated for it. This type of vulnerability is particularly dangerous because it can lead to various issues, including crashes, data corruption, sensitive data leaks, and even the execution of malicious code. 

The recent global IT outage caused by a faulty update from CrowdStrike is an excellent example of the severe consequences these vulnerabilities can have. Crowdstrike reported that problematic content in Channel File 291 triggered an out-of-bounds memory read, leading to a Windows operating system crash (BSOD). Let’s dive into how out-of-bound memory reads occur and how you can eliminate them in C/C++ code with fuzz testing. 

Contents

• What is an Out-of-Bound Memory Access
• What Causes Out-of-Bound Access Bugs
• How to Detect Out-of-Bound Reads
• What is Fuzz Testing
• Why Fuzz Testing is Effective in Detecting Out-of-Bound Reads
• Example: Detecting Out-of-Bound Reads with Fuzz Testing

What is an Out-of-Bound Memory Access 

At a high level, think of a buffer as a fixed-sized container that you can read data from or write data to. If a program mistakenly tries to access data outside of this container, it can cause unpredictable behavior. These bugs are especially prevalent and critical in memory-unsafe languages such as C/C++ because they make it possible to access data from other buffers or data structures potentially corrupting or exposing their content.

There are two main types of out-of-bounds access:

1. Out-of-bounds read: When the program reads data beyond the boundaries of a buffer, it potentially exposes sensitive data or causes a crash.
2. Out-of-bounds write: When the program writes data beyond the buffer's limits, which can overwrite other critical data or control structures, leading to more severe security vulnerabilities like arbitrary code execution.

In the Root Cause Analysis report from August 6, 2024, Crowdstrike states:

On July 19, 2024, a Rapid Response Content update was delivered to certain Windows hosts, evolving the new capability first released in February 2024. The sensor expected 20 input fields, while the update provided 21 input fields. In this instance, the mismatch resulted in an out-of-bounds memory read, causing a system crash.

Out-of-bounds memory access can lead to a wide range of problems, which can be catastrophic as the Crowdstrik incident clearly showed. Here are some concrete examples of what can go wrong:

1. Program Crashes: Accessing memory outside the allocated space can cause the program to crash or behave unpredictably, potentially leading to denial of service (DoS) by rendering the vulnerable software unusable.
2. Data Corruption: Reading from memory that the program does not own can result in corrupted data being read and used by the program, leading to incorrect behavior.
3. Remote code execution: An out-of-bounds write can lead to buffer overflow vulnerabilities, which attackers often exploit to inject malicious code. The classic example is the exploitation of a buffer overflow to inject and execute malicious code on a system, giving the attacker control over the program’s execution
4. Information Leakage: Out-of-bounds reads can expose sensitive information that resides in adjacent memory. For example, reading beyond the bounds of an array could allow access to passwords, encryption keys, or other confidential data inadvertently left in nearby memory locations. The Heartbleed vulnerability in OpenSSL, which was caused by an out-of-bounds read, is one of the best examples of this risk. 

What Causes Out-of-Bound Access Bugs

Out-of-bounds access bugs are typically caused by errors in how programs handle memory. These errors can occur in various ways, leading to memory being accessed outside of its intended boundaries. Here are some of the most common causes:

1. Array Index Errors: Arrays in C/C++ are zero-indexed, meaning that if an array has `n` elements, valid indices range from `0` to `n-1`. Accessing an index outside this range leads to out-of-bounds access.
2. Pointer Arithmetic Errors: Pointers allow direct memory manipulation in C/C++. Incorrect pointer arithmetic can lead to pointers that point outside the intended memory region, causing out-of-bounds access.
3. Incorrect Boundary Checks: Functions that copy memory, such as `memcpy`, `strcpy`, or `strncpy`, require careful handling to ensure that the source and destination buffers are large enough. Failing to properly check boundaries before copying data can result in out-of-bounds access.

Below is an example of a buffer overflow in C++ that leads to an out-of-bound memory read. This example demonstrates how writing beyond the bounds of an array can cause subsequent reads to access unintended memory locations. 

#include <iostream>
#include <cstring>
int main() {
    char buffer[10];
    const char* longString = "This string is too long for the buffer";
    // Buffer overflow: writing more data than the buffer can hold.
    std::strcpy(buffer, longString);
    // Out-of-bound memory access by trying to access the 16th element of a 10-element array
    std::cout << "Out-of-bound read: " << buffer[15] << std::endl;
    return 0;
}          

How to Detect Out-of-Bound Reads

Fuzz testing, or fuzzing, is the most effective method for detecting out-of-bounds memory access vulnerabilities and memory corruption bugs in general. Crowdstrike has now added fuzzing to its testing strategy to prevent similar incidents in the future. 

Extract from Crowdstrike’s Preliminary Post Incident Review

What is Fuzz Testing

Fuzz testing, also known as fuzzing, is a software testing technique that involves feeding a program with a large amount of randomly generated or intentionally malformed data to find bugs and vulnerabilities. The idea is to expose the program to unexpected or extreme inputs that it might not normally encounter, with the goal of causing the program to behave in ways that reveal underlying issues, such as crashes, memory leaks, or security vulnerabilities. 

White box fuzzing, also known as feedback-based fuzzing, is a type of fuzz testing where the source code is instrumented for code coverage and bug detection capabilities (using dedicated sanitizers). This makes it possible to automatically generate test cases that systematically explore your program, maximize code coverage, and effectively detect a wide spectrum of issues, including memory corruption bugs. 

You can learn more about fuzzing technology in this guide, “What is Fuzz Testing?”

Why Fuzz Testing is Effective in Detecting Out-of-Bound Reads

White box fuzzing with code instrumentation and sanitizers is particularly effective in finding bugs for several reasons:

1. Maximized Code Coverage: By coverage feedback for each test case, the fuzzer can generate inputs that are more likely to explore the less commonly executed paths, including edge cases that are often where bugs hide.
2. Early Detection of Vulnerabilities: Sanitizers can immediately detect and report bugs like out-of-bounds memory access, memory leaks, and other issues as soon as they occur, even if they don't cause a crash right away. This early detection helps developers catch and fix bugs before they become serious problems. By integrating fuzzing into your continuous integration pipeline, you can ensure that any code change is tested right away, and you uncover bugs that may remain undetected for a long time.
3. Easier and Faster Remediation: When fuzzing detects a bug, it provides a specific input, i.e., reproducer, that reliably triggers the bug. It also provides a detailed stack trace showing the sequence of function calls leading up to the error and pinpointing the exact location in the code where the problem lies. This is invaluable for developers to quickly understand the root cause and provide a fix.

Why Fuzz Testing is Effective in Detecting Out-of-Bound Reads

White box fuzzing with code instrumentation and sanitizers is particularly effective in finding bugs for several reasons:

1. Maximized Code Coverage: By coverage feedback for each test case, the fuzzer can generate inputs that are more likely to explore the less commonly executed paths, including edge cases that are often where bugs hide.
2. Early Detection of Vulnerabilities: Sanitizers can immediately detect and report bugs like out-of-bounds memory access, memory leaks, and other issues as soon as they occur, even if they don't cause a crash right away. This early detection helps developers catch and fix bugs before they become serious problems. By integrating fuzzing into your continuous integration pipeline, you can ensure that any code change is tested right away, and you uncover bugs that may remain undetected for a long time.
3. Easier and Faster Remediation: When fuzzing detects a bug, it provides a specific input, i.e., reproducer, that reliably triggers the bug. It also provides a detailed stack trace showing the sequence of function calls leading up to the error and pinpointing the exact location in the code where the problem lies. This is invaluable for developers to quickly understand the root cause and provide a fix.

Example: Detecting Out-of-Bound Reads with Fuzz Testing

Let’s look at another critical example of triggering out-of-bounds reads. The Heartbleed vulnerability affected the OpenSSL library. Heartbleed allowed attackers to steal sensitive data such as secret keys, user names, passwords, and business-critical documents. The root cause was an out-of-bound memory read (heap read buffer overflow) that went undetected for over two years. This video below demonstrates a fuzz test by CI Fuzz in action and how quickly it can reveal such vulnerabilities.

Crowdstrike’s incident and the Heartbleed vulnerability underscore the importance of making fuzz testing an integral part of your software testing strategy to uncover critical issues that might otherwise remain hidden. Watch the full recording and see live demos of detecting out-of-bound memory access bugs and similar vulnerabilities in C and C++ projects.