How to detect more bugs in AUTOSAR Applications and enable SiL testing by using a simulator

Testing Classic AUTOSAR applications has long been a significant challenge due to the reliance on hardware-in-the-loop (HiL) setups, which are costly, complex, and hard to scale. Code Intelligence’s new lightweight AUTOSAR simulator revolutionizes this process by enabling entire AUTOSAR applications to run on x86 Linux systems, thus facilitating software-in-the-loop (SiL) testing. This enables security and functional testing at the system level, ultimately enhancing the robustness and security of automotive software.

Content

• Introduction
• Why test AUTOSAR Applications at the system level
• Classic AUTOSAR architecture
• Challenges of Testing Classic AUTOSAR Applications
• Code Intelligence AUTOSAR Simulator
• Why use the AUTOSAR simulator

Introduction

In the rapidly evolving world of automotive software, AUTOSAR (AUTomotive Open System ARchitecture) stands out as a pivotal framework designed to enhance the development of robust, scalable, and maintainable automotive applications. Established as a global development partnership of car manufacturers, suppliers, and other companies, AUTOSAR aims to standardize software architecture, ensuring reliability and interoperability across diverse vehicle systems.

AUTOSAR is primarily categorized into two variants: Classic AUTOSAR and Adaptive AUTOSAR.

1. Classic AUTOSAR is tailored for deeply embedded systems that demand high reliability and real-time performance. It is widely used in various vehicle functions, such as transmission systems and safety-critical systems like airbags and brake control. Version 1.0 of the classic platform was released in 2005.
2. Adaptive AUTOSAR caters to the dynamic and flexible needs of modern automotive applications. It supports high-performance computing platforms and is designed for applications requiring more processing power and adaptability, such as advanced driver assistance systems (ADAS), infotainment, and autonomous driving features. The first version of the adaptive platform was released in 2017.

Examples of AUTOSAR applications include electronic stability control (ESC), steering, electric parking brakes, park distance control, exterior lights, infotainment, anti-theft systems, and remote keyless entry. 

Why test AUTOSAR Applications at the system level

Testing AUTOSAR Applications at the system level is crucial. While unit testing is important for verifying individual components and catching bugs early, system-level testing validates the end-to-end functionality of integrated components in real-world conditions. It ensures that all components work together as intended, meeting the high standards required for automotive applications.

Even if you catch bugs at the unit level, only testing AUTOSAR applications at the system level can verify if these bugs can actually be triggered in an integrated system or exploited by external attackers.

However, testing AUTOSAR applications, particularly Classic AUTOSAR, in a Software-in-the-Loop (SiL) manner presents significant challenges. These include the need for specialized hardware and complex setups to replicate the real-time and reliability requirements of automotive systems.

We at Code Intelligence addressed these challenges by developing a lightweight simulator that enables software-in-the-loop testing. As Classic AUTOSAR was developed earlier and, thus, has wider adoption, we first focused on simulating the Classic platform. Let’s dive into Classic architecture and how Code Intelligence’s lightweight simulator enables developers to run and fuzz Classic AUTOSAR applications directly on an x86 Linux system in a completely software-in-the-loop manner.

Classic AUTOSAR architecture

The Classic AUTOSAR architecture is designed to create a standardized ECU (Electronic Control Unit) software architecture by separating hardware-independent application software and hardware-oriented basic software (BSW) using a software abstraction layer known as the Runtime Environment (RTE). The RTE facilitates the development of OEM-specific and competitive software applications on its upper side while ensuring the standardization and OEM independence of basic software on its lower side. This separation allows for scalable ECU software across different car lines and variants, the distribution of functional software modules across multiple ECUs, and the integration of software modules from various sources.

The basic software within the AUTOSAR architecture is divided into three layers: services, ECU abstraction, and microcontroller abstraction. The RTE manages the data exchange between these layers and the application layer, forming a component-oriented, hardware-independent software structure at the application level. This structure comprises software components (SWCs) as individual units, enabling the development of SWCs without specific knowledge of the hardware and allowing for flexible relocation of existing SWCs to different ECUs during development.

Figure 1: Classic AUTOSAR Architecture, Source

Challenges of Testing Classic AUTOSAR Applications

Testing Classic AUTOSAR applications presents significant challenges due to their nature as deeply embedded systems. Typically, this testing is conducted using a hardware-in-the-loop (HiL) approach with the target hardware, which ensures real-time performance and reliability. However, compared to software-in-the-loop (SiL) testing, HiL testing is hard to scale and integrate into development processes. This is mainly due to the following reasons: 

• HiL testing requires specialized hardware setups, which are often costly and limited in availability. 
• Scaling these setups to accommodate extensive testing needs can be prohibitively expensive and logistically complex. 
• The physical hardware constraints make it difficult to run multiple tests in parallel or rapidly iterate through different scenarios. This limitation poses a significant bottleneck in continuous integration and delivery pipelines, where quick feedback and iterative development are crucial.

In contrast, SiL testing offers a more scalable solution by running tests entirely in software, leveraging the flexibility and computational power of modern x86 systems. The main benefits of doing so: 

• It allows for the scaling of the testing setup, enabling multiple instances of tests to run concurrently without the need for additional hardware. 
• It reduces costs and accelerates the testing process, facilitating more frequent and thorough validation of AUTOSAR applications throughout the development cycle.
• This setup unlocks the potential for superior security testing approaches, such as whitebox fuzzing, which is currently the most effective practical method for finding bugs and security vulnerabilities in software.

To enable SiL testing for AUTOSAR applications, we developed a lightweight classic AUTOSAR simulator that allows entire AUTOSAR applications to run on x86 Linux systems. This enables developers to perform standard testing, e.g., unit, integration, and end-to-end system tests, as well as whitebox fuzzing entirely in software. By eliminating the need for specialized hardware, our simulator accelerates the development process and enhances the ability to identify and resolve bugs and security vulnerabilities early in the development cycle.

Code Intelligence AUTOSAR Simulator

The lightweight classic AUTOSAR simulator by Code Intelligence is designed to replace hardware-dependent layers, including the Microcontroller Abstraction Layer (MCal) and Operating System (OS), with implementations for Linux x86. This allows the upper layers of the AUTOSAR stack to run unchanged on a Linux system. The simulator consists of several main components:

1. Scheduler: This component allows for the registration and execution of AUTOSAR tasks and alarms. It also runs the OS for a specified amount of time, ensuring that all tasks are executed as they would be on the actual hardware. Additionally, tasks can be executed in response to events triggered by receiving specific inputs, such as CAN messages with certain IDs, making the simulation environment highly responsive and realistic.
2. Interface Implementation: The simulator implements various standard AUTOSAR interfaces such as CAN, SPI, DIO, and custom complex device drivers like I2C. These interfaces serve as entry points for external inputs into the system, making them ideal for feeding fuzzer-generated data. For example, the CAN interface is a crucial entry point for reaching code that can only be accessed through these interfaces.
3. Mutator: A simulator's key feature is enabling structure-aware fuzzing. The mutator generates input messages according to the application's expected format. For instance, CAN messages have a specific frame format, including an ID, length, and payload. Our mutator can generate inputs for all supported interfaces, ensuring that the fuzzing process is both effective and efficient.
4. Data Provider: This component offers an easy-to-use interface that allows the System Under Test (SUT) to consume data for specific interfaces from various sources, including fuzzer-generated inputs or user-provided data in E2E testing scenarios. This ensures seamless integration of the simulated environment with the actual application code.
5. Time Machine: This library is linked to the application to reset the global memory state before each test or fuzzing run. This is crucial for ensuring that each test starts from the same initial state, providing consistency and reliability in the testing process.

To use the simulator, software engineers need to write glue code that redirects calls from the original MCal functionality to the simulator's implementation using its API. We refer to this as adapters. The result is a fully runnable AUTOSAR application on Linux x86.

Figure 2: Code Intelligence AUTOSAR Simulator

Why use the AUTOSAR simulator?

One of the simulator's key advantages is that it interacts directly with standard interfaces like CAN, i.e., it generates inputs that are fed directly into the respective interface. Therefore, any bugs and vulnerabilities identified during fuzzing result in crashing inputs that can be replayed on the target hardware. This enables end-to-end fuzzing of AUTOSAR applications completely in software, leveraging the advanced fuzzing and bug detection capabilities available on Linux x86 systems.

In summary, the lightweight AUTOSAR simulator by Code Intelligence simplifies the testing process by allowing developers or security experts to run and fuzz their applications at the system level on a standard Linux x86 environment. 

As the simulator has demonstrated its effectiveness in fuzzing AUTOSAR applications in a SiL manner, some of our customers are now considering using it for their standard testing procedures (functional and integration tests). The goal is to reduce costs, enhance scalability, and accelerate the development and testing cycles, ultimately leading to more robust and secure automotive software.

“Code Intelligence’s AUTOSAR Simulator allows us to execute and fuzz code in a software-in-the-loop manner. It automates finding bugs as early as possible and ensures that all delivered components from Tier-2 suppliers don’t contain critical vulnerabilities as buffer overflows,” said Dr. Michael von Wenckstern, Product Cybersecurity Governance, Risk and Compliance Specialist, Continental AG. 

We are currently working on expanding support for additional interfaces based on customer requirements and enhancing overall performance. By incorporating more interfaces, we aim to cover a broader range of AUTOSAR applications, ensuring comprehensive testing capabilities. Performance enhancements will focus on optimizing the simulator to handle more complex scenarios and larger-scale testing setups efficiently.

If you are interested in seeing the AUTOSAR simulator in action, watch the recording of a live demo session or book a call with us.