Embedded Testing Tools: A Comprehensive Guide

Due to increasing connectivity and dependencies, modern embedded applications in many industries including automotive, medical devices, aviation, and even automated cow brushes (no joke) are constantly growing more complex. This complexity comes with implications for embedded testing tools and requires plenty of manual effort, depending on the toolchain. From an operational perspective, many embedded industries are tightly staffed and work in long cycles with strict deadlines. In addition, embedded software systems often have hard dependencies on third-party applications and integrations that introduce extra complexity for testing. When things get tight, this can cause time-critical matters to be prioritized over software testing.

Common Issues in C/C++-Based Software

When talking about embedded programming languages, the first one that comes to most developers' minds is usually C/C++, as it’s one of the largest embedded ecosystems, mainly because C/C++ are low-level languages that can access hardware components directly, e.g., microcontrollers and system memory. This low-level control is both C/C++’s strength and its weakness, as this means there is a large class of issues with memory management and resource overflows. From a security perspective, testing an embedded application is vital since the device running the application is often part of a critical system.

Embedded Testing Tools and Approaches

Broadly, there are two main types of embedded testing tools: static analysis and dynamic analysis. Static analysis approaches examine and test source code before it is executed, whereas dynamic analysis approaches examine code while executing it. Both approaches complement each other and are not mutually exclusive.

Static Analysis Approaches and Tools

Static analysis is a method of evaluating source code without executing it. There are two main types of static analysis: linters, which check the quality of the code, and static application security testing (SAST), which focuses on security analysis.

Static analysis is particularly useful for embedded C/C++ code, as the system under test does not need to be compiled or deployed in production. Advantages include:

• Works from the IDE and gives immediate feedback loops as no compilation is required
• Helps achieve compliance with code styles and guidelines, e.g. MISRA guidelines
• Finds bad practices, code smells, and syntax errors by parsing the code
• Provides code coverage data.

While static analysis has undeniable advantages, it also has a few limitations. 

• False negatives: some bugs and vulnerabilities cannot be identified through static analysis.

There are specific categories of issues in C/C++ code that may be challenging or impossible to detect solely through static analysis. Some examples include Memory Buffer Overflows, Runtime or Dynamic Behavior Issues, and Input-Dependent Flaws. 

• The fatigue among developers due to false positives. 

Whenever you check your code for security issues with static analysis, you get many false positives (code that does not actually contain vulnerabilities) and duplicates (previously assessed findings that reappear). In large projects, it's common to encounter hundreds of thousands of warnings. When teams turn off some checkers or security rules to get fewer alarms, the risk of missing crucial bugs increases.

As a result, alternative testing technologies such as penetration, dynamic, or fuzz testing continue to reveal bugs and vulnerabilities in projects previously scanned by static code analysis.

Dynamic Analysis Approaches and Tools

Dynamic Analysis tools test software for bugs and vulnerabilities by executing the code and evaluating its behavior during runtime. Dynamic testing complements static analysis, and the best security practice involves doing both. 

There are two types of dynamic application security testing: white-box and black-box. 

	Black-box testing	White-box testing
Typical Users	Security Teams	Developers and Testers
Application Knowledge	Little Knowledge Analyzes the functionality of an application without a thorough knowledge of the internal design.	Full Knowledge  Analyzes the internal structure of a program knowing the code of the software system.
Access to source code	Not needed	Required
Code coverage	Unknown and random	High, up to 100%
Ability to uncover complex issues	Low Needs human experts with deep knowledge and experience.	High The process can be automated.
Issues reproducibility	No It’s hard to understand issues due to missing source code connection.	Yes Showing the exact location in the source code, a trigger, and a stack trace.
Integration into the development process	Difficult Misses developers' acceptance.	Full Automatically testing every single code change with high developer acceptance.

Table 1. Black-box vs. White-box dynamic testing

What is fuzz testing? 

One of the most effective dynamic testing methods for embedded systems is fuzz testing. Fuzz testing involves feeding invalid or random data, known as "fuzzy" data, into the software under test and observing how it behaves, aiming to uncover vulnerabilities or crash the application. 

Similar to unit testing, the software is tested under various scenarios. The key difference is that unit testing is deterministic, meaning that the expected outcome is known in advance, while fuzz testing is non-deterministic or exploratory, meaning that the expected outcome is not necessarily known ahead of time. This makes fuzz testing useful for uncovering bugs and security vulnerabilities that may not have been identified through other testing methods.

There are protocol (aka black-box) and source code fuzz testing (white-box) solutions. The differences between them are the same as between black-box and white-box testing, as mentioned in the Table above. Code Intelligence’s fuzz testing does source code (white-box) testing for higher code coverage and issue reproducibility. 

Learn more about best practices for embedded software security testing and the top 7 challenges in embedded testing by reading the guide 'Best Practices for Embedded Software Security Testing.' 

Where Are Embedded Testing Tools Heading?

In the past, embedded systems usually were shipped and therefore couldn’t be updated. Now, many embedded systems can be connected to the cloud, meaning that updates and bug fixes can be deployed even to running systems. This means that modern software development approaches, such as Continuous Integration and Continuous Delivery, can be applied to embedded software. While this new form of software development poses many challenges for embedded testing tools, it also opens some doors, as it enables embedded software teams to integrate automated software tests into their CI/CD for continuous testing.

IoT devices also increase the need for security testing since now there are new security concerns with network connectivity on these systems. Fuzz testing can be a great technique to find and assess security vulnerabilities before shipping a connected device.

In light of the above, a great approach for securing embedded C/C++ applications is to combine the approaches of static analysis and fuzz testing. If you're new to fuzz testing, you can learn more about fuzz testing here.