For obvious reasons, civilian aerospace is steeped in safety regulations. Long-standing international governing bodies mandate and oversee the specification, design, and implementation of civil avionics such that failure conditions that could lead to safety hazards are identifiable, assessed and mitigated.
During FuzzCon Europe, Paul Butcher talked about considerations over why international aerospace regulatory bodies felt additional guidelines that combine aviation safety and security were needed in the form of an "Airworthiness Security Process".
Paul Butcher on fuzz testing in international aerospace guidelines.
What Do We Mean by Airworthiness?
The number one priority with civilian air travel is human safety. Everything else is secondary and, while other factors, including security, are important, none of them will ever be placed before human safety. In this context, human safety is focused on any persons involved in the operation of the air vehicle (i.e., passengers, the flight crew, the ground crew, etc).
Military Aviation AuthorityMaster Glossary (MAA 02 Glossary)
This regulatory-enforced approach to air travel is set out in international legislation and can be summarised by the term "Airworthiness". Operators wanting to fly their air vehicles need to gain airworthiness from the regulatory authority responsible for the airspace the operator wants to travel within. More specifically, organizations such as the Federal Aviation Administration (FAA) (as part of the U.S. Department of Transportation) and the European Union Aviation Safety Agency (EASA) (for air corridors across Europe), plus others.
Ensuring our air vehicles are airworthy, and therefore safe for flight is a challenge. However, a bigger challenge is convincing the regulatory authorities that the vehicles are safe! Here approaches like safety cases are used to document clear, concise, and convincing safety arguments. The goal of these arguments is to convince the certification authority that the risk of an air-vehicle systems failure (that could lead to a safety hazard) is as low as reasonably practical.
Fortunately, the regulatory authorities provide help in the form of "Advisory Circulars" (ACs) that stipulate that certain standards and guidelines are deemed as an acceptable means of compliance (AMC) with specific aspects of airworthiness. DO-178C, titled "Software Considerations in Airborne Systems and Equipment Certification", is a prime example and is often used to gain approval of the safe usage of commercial software-based aerospace systems.
It is also fair to say that, even considering the complex, thorough, and mandated regulatory safety processes, the industry is very good at achieving airworthiness certifications. This is good, and we should all sleep better knowing that these safeguards are in place!
Security Trends in Modern Civil Avionics
So, if civilian air travel is already very safe, why do we need an "Airworthiness Security Process"? This can be partially answered by a keynote address made by Robert Hickey:
“We got the airplane on Sept. 19, 2016. Two days later, I was successful in accomplishing a remote, non-cooperative, penetration” (Ref: Aviation Today)
What is maybe more worrying is that the report goes on to imply that the involved Avionics Original Equipment Manufacturers (OEMs) later declared they were aware of the exploit path, as well as many others.
Another interesting statement made by Robert Hickey during that keynote address in 2016 was around the estimated staggering cost to patch software in a deployed avionics system:
“The cost to change one line of code on a piece of avionics equipment is $1 million, and it takes a year to implement.”
Clearly, this emphasizes the obvious secondary need for the aircraft industry to construct safe and secure embedded systems.
Why the Need for Aviation Security Standards?
One reason this situation could have occurred is likely due to the terminology used within the existing safety guidelines. "Failure Conditions" are widely understood within the industry to be resulting scenarios that directly affect the vehicle and/or its occupants. These conditions are caused by internal failures, system errors, environmental operating conditions, extreme external events such as atmospheric conditions, and other scenarios such as bird strikes and baggage fires.
In order to gain airworthiness, all aircraft Failure Conditions need to be identified, analyzed and understood such that the resulting effect can be categorized, associated with any known safety hazard, and mitigated if appropriate.
The problem, however, is that there is no explicit consideration of cyber-threats acting as events that can lead to a failure condition. In order to address this shortfall two, new working groups within the RTCA and EUROCAE were formed and tasked with producing an Airworthiness Security Process.
This led to the birth of a set of standards and guidelines widely known as the ED-202A/DO-326A set, and early action of this joint committee was to bring a new term to the table, namely a "Threat Condition".
"A condition having an effect on the aeroplane and/or its occupants, either direct or consequential, which is caused or contributed to by one or more acts of intentional unauthorised electronic interaction, involving cyber threats, considering flight phase and relevant adverse operational or environmental conditions. Also, see failure condition." (Ref: ED-202A/DO-326A)
Here the terminology is deliberately succinct; a Threat Condition focuses on the effect of a cyberattack on the air vehicle. This also makes it very clear that the primary purpose of the Airworthiness Security Process is to ensure the safety of flight.
Airworthiness Security Process (AWSP)
The process comprises seven main stages broken down into sub-stages with identified stage inputs and stage outputs. The initial phase is known as the "Plan for Security Aspects of Certification" (PsecAC), and it is here that we set our security goals and how we intend to security test our application. Much like a "Plan for Safety Aspects of Certification", we need to ensure our regulatory authority accepts our plan before we commence with our development and test phases.
However, the narrative of the process should not be considered in any way linear. Instead, it tends to jump between sub-stages and loops around groups of stages as security assurance is reassessed, risk mitigation readdressed, and security development reworked. The process details are too complex to address in any detail within this blog post. However, one area of particular interest, where fuzz testing can play a crucial role, is within the "Security Effectiveness Assurance" stage.
Security Effectiveness Assurance
This phase aims to show compliance with security requirements and to evaluate the effectiveness of implemented security measures. More specifically, we need to verify that we have satisfied any explicit security requirements, demonstrate the effectiveness of any security measures to protect our identified security assets, and provide evidence to argue that our system is free of vulnerabilities. Note that in the context of ED-202A/DO-326A, the definition of vulnerability states that it has to be demonstrably exploitable.
Fuzz testing is one such means of meeting objectives within the Security Effectiveness Assurance phase due to three primary reasons:
- Fuzz testing can assess the effectiveness of a security measure
- Fuzz testing can identify vulnerabilities in the form of exploitable software bugs
- and therefore, fuzz testing can help identify security assets.
DO-356A / ED-203A and the Introduction of Security Refutation
The ED-202A/DO326A Airworthiness Security Process is supported by a set of guidelines stated within ED-203A/DO-356A, titled "Airworthiness Security Methods and Considerations". Here, the reader is introduced to the term "Refutation". The aim of the Refutation phase is to assess the security assurance of the system under test.
Refutation is all about refuting that the system is secure, and this negative take on standard verification-based testing (positive testing) is very deliberate. The intention is to direct the focus of the activity towards the mindset of an attacker. It is advised that multiple activities should be adopted to make up the Refutation testing phase, and the guidelines suggest that the following should be considered:
- Security penetration testing
- Fuzzing
- Static code analysis
- Dynamic code analysis
- Formal proofs
Fuzz testing is traditionally considered a negative testing capability and is therefore considerably well suited to refutation testing. Unfortunately, the guidelines around how to include a fuzzing campaign within a PsecAC are lacking. An industrial working group within HICLASS highlighted this gap and a clear appeal for a better understanding of the technology was made.
Guidelines and Considerations Arund ED-203A / DO-356A Security Refutation Objectives
In order to meet this industry need, AdaCore produced a technical paper to provide additional considerations and guidelines over how to include a security refutation activity (including fuzz testing) within a PsecAC. The paper is available via AdaCores tech papers website. One area of particular interest within this paper is the recommendation that a fuzzing campaign plan includes both a "starting criteria" and a "stopping criteria".
The starting criteria focus on the quality of a particular fuzz test’s starting corpus and argue that a good aim is to achieve 100% statement coverage. Once the starting criteria has been satisfied, we can commence the fuzzing campaign until the stopping criteria is met.
The stopping criteria guidelines state that a formula should be derived that determines the campaign duration. The formula's goal is to argue that the resulting duration is complementary to the level of security assurance the test is trying to achieve. Factors should include (but not be limited to):
- the average achievable test execution speed;
- the security assurance level of the targeted security measures;
- the cyclomatic complexity of the control flow of the application under test;
- the measured complexity of the test input data structure.
Final Thoughts...
Fuzz testing is not a traditionally used technique within aerospace. However, the emergence of security guidelines such as ED-202A/DO-326A forces the industry to think again. Mature software testing approaches now need to adapt to new regulatory requirements around cyber threats.
In addition, avionics software development life-cycle plans now need to include considerations around security assurance testing and identifying exploitable software bugs. Where other industries, such as automotive and medical devices, have been early adopters of fuzzing, aerospace is now playing catch up.
Want to learn more about fuzz testing?
Book a demo with Code Intelligence's experts to learn how fuzz testing can strengthen your security strategy and comply with industry standards.