Top 5 reasons to fuzz embedded systems

One of the most effective security testing methods for embedded systems is fuzz testing. It’s the fastest way to identify memory corruption errors and their root cause. It enables a shift-left testing approach, recommended by many industry standards, and reaches up to 100% code coverage. Read on for the details. 

Contents

• What is fuzz testing?
• 5 reasons to fuzz embedded systems
• Best practices for embedded software security testing

What is fuzz testing 

Fuzz testing, also known as fuzzing, involves feeding invalid or random data, known as "fuzzy" data, into the software under test and observing how it behaves, aiming to uncover vulnerabilities or crash the application. 

Similar to unit testing, the software is tested under various scenarios. The key difference is that unit testing is deterministic, meaning that the expected outcome is known in advance, while fuzz testing is non-deterministic or exploratory, meaning that the expected outcome is not necessarily known ahead of time. This makes fuzz testing useful for uncovering bugs and security vulnerabilities that may not have been identified through other testing methods.

There are protocol (aka black-box) and source code fuzz testing (white-box) solutions. Learn their differences in the blog post “Protocol Fuzzing vs. Code Fuzzing.”

Code Intelligence’s fuzz testing platform does source code (white-box) testing for higher code coverage and issue reproducibility.

5 reasons to fuzz embedded systems

Due to increasing connectivity and dependencies, modern embedded applications are constantly growing and becoming more complex. The knock-on effect is that an enormous amount of effort is required to keep software testing up to speed with the increasing complexity. One of the most effective security tests for embedded systems is fuzz testing. Here is why:

1. Fuzzing detects memory corruption issues & other critical bugs

By feeding embedded software with unexpected or invalid inputs, modern fuzzing tools can uncover dangerous edge cases and behaviors related to communication with memory. These include buffer overflows, memory corruption, out-of-bound errors (which caused the IT outage caused by Crowdstrike’s update), and other issues relevant to memory-unsafe languages such as C/C++. See the full list here. 

Practical example

Here is an example showing how a buffer overflow can be uncovered by Code Intelligence’s CI Fuzz but is missed by static analysis, another security testing approach. 

In this example, Static analysis reports an issue in line 68 as an issue, indicating that the parameter ‘dest' may point to deallocated memory. This can only happen if the option RESET_DESTINATION is activated. However, this option is always deactivated in the projects setting, so the dest buffer is never deallocated. This false positive is because static code analysis does not know the runtime environment of the tested software and makes imprecise assumptions about it.Line 75 contains a real security issue (buffer overflow) that can be used to cause crashes and even inject malicious code. Static code analysis misses this bug as it is hard to follow the dynamic computations performed to construct the dest buffer. So, for static analysis, it's a false negative. On the other hand, fuzz testing was able to quickly find this bug and produce a concrete input that triggered the bug. This proved very useful for a developer to provide a fix.

2. Shift-left: fuzzing uncovers issues as early as you have executable code

Fuzz testing that analyzes source code can be integrated into the development process to test your code automatically as soon as you have an executable program. By leveraging fuzzing early, you can find bugs and vulnerabilities even before the software is ready to run on the target hardware.

If we look at the V-model, fuzz testing can be applied at the unit, integration, and system testing stages.

Static Analysis and Source Code Fuzzing in the V-model

3. Fuzzing aids in identifying the root cause

White-box fuzzers analyze source code and show exactly where an issue occurs. They pinpoint the exact line of code where the bug is hidden and share the input that triggered it. This makes the bug easy to reproduce, allowing developers to identify the root cause in minutes.

CI Fuzz provides the exact line of code where the bug is hidden alongside crashing input and stack trace

As fuzz testing analyzes the running code, every flagged issue represents an actual problem. This ensures your team doesn't waste time on false positives.

4. Fuzzing aids in identifying the root cause

Source code fuzzers leverage feedback about the software under test to achieve the highest code coverage. Subsequent executions automatically generate new test cases to detect additional paths, thereby increasing code coverage.

Besides reaching critical parts of the tested code more reliably, this also generates highly useful reports that inform development teams about how much of their code was executed during a test and which parts are dead or unreachable. This reporting helps identify which parts of the code need inspection or additional review.

Example of a code coverage report. Lines colored blue were hit by fuzz testing

5. Industry standards and regulations advocate for fuzz testing

Fuzz testing is recommended by many norms and standards, including but not limited to: 

• ISA/IEC 62443 Security for industrial automation and control systems – Part 4-1: Secure product development lifecycle requirements  
• ISO/SAE 21434 Road Vehicles — Cybersecurity Engineering
• Automotive SPICE for Cybersecurity Guidelines
• Guidance for Industry and Food and Drug Administration Staff “Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions”
• MDCG 2019-16 Guidance on Cybersecurity for Medical Devices
• AAMI TIR 57:2016 Principles For Medical Device Security - Risk Management
• IEC 81001-5-1 Health software and health IT systems safety, effectiveness and security. Part 5-1: Security — Activities in the product life cycle.
  
  

Best practices for embedded software security testing

Broadly speaking, there are two kinds of embedded testing tools: static and dynamic code analysis. The best security practice is to leverage both. 

Static analysis is a method of evaluating source code without executing it. There are two main types of static analysis: linters, which check the quality of the code, and static application security testing (SAST), which focuses on security analysis.

Dynamic Analysis tools test software for bugs and vulnerabilities by executing the code and evaluating its behavior during runtime. Dynamic testing complements static analysis, and the best security practice involves doing both. One of the most effective dynamic testing methods for embedded systems is fuzz testing.

The free guide “Best practices for embedded software security testing” explains the advantages and disadvantages of static and dynamic testing approaches. If you work with embedded systems, read it to learn the top 7 challenges of embedded testing, how to overcome them, and dive deeper into static and dynamic testing methodologies.