During the last year, we went to many security testing conferences and fairs. In describing what we do and what our product is intended for, we have experienced some confusion. While the meaning of software testing is clear, the difference between functional testing and non-functional testing is not always evident. Many of the people we met were not sure if security testing falls under the category of functional testing or not. Therefore, we decided to take a look at the topic, and point out the differences between these two software testing approaches.
Functional Testing
Functional tests are conducted from the perspective of users. The main questions they ask are “Can the user do this?” (capability test), “Does the feature work?” (feature test) and “Does the software meet user expectations?” (UA/UI test) (source).
Thus, during functional testing, the product is tested as it is meant to be used, to assure that specific functions and activities of the code are working. This does not include security or reliability issues.
Functional testing approaches do not protect the software from unwanted third-party access. Their purpose is rather to ensure that users cannot produce misbehaviours or catastrophic failures. Not all those who use the product do so for the original purpose, and not all users have good intentions. Issues regarding security vulnerabilities may not be detected while taking in the perspective of common users.
Non-Functional Testing
Non-functional testing concerns software issues that are not necessarily associated with a particular function or user action. This includes performance, reliability and usability issues. Basically, non-functional tests analyze how a system deals with unexpected inputs. This also includes security testing. So to answer the initial question: Security testing is as a form of non-functional testing.
Security Testing
In fact, security testing is essential for software that processes confidential data to prevent system intrusion by attackers. But not just confidential data has to be protected. IoT and embedded devices are currently among the largest boom markets. Their omnipresent integration in everyday life, as well as in critical infrastructures and industrial facilities, makes security issues unpreventable.
However, practice shows that the code quality of software projects is often insufficient and companies lack testing experts to deal with it. Nevertheless, security testing should be a crucial part of every software development process.
Do Both!
Of course, functional and non-functional testing cannot be clearly distinguished. The lines get blurry when, occasionally, non-functional tests find functional issues or vice versa. However, this does not mean that one procedure is dispensable. The two testing methods should be conducted and evaluated independently, to ensure that all vulnerabilities are found, and a high-quality product can be offered.
One of the main reasons why these approaches are not always sufficiently applied, is the shortage of personnel. The lack of skilled professionals makes it difficult for companies to find highly qualified developers and testers. Available developer time is therefore primarily devoted to matters with more immediate consequences than software testing. Also, applying both testing approaches together requires a lot of time and slowed down release cycles. This is why testing is sometimes neglected, despite its high relevance, and its long-term profitability.
Continuous Testing Saves Time
Since most development teams have little time for security testing, it even becomes more important to have the right setup and testing processes in place. Best practices for security testing usually recommend running a number of different, testing approaches in a CI/CD pipeline, with the ultimate goal to perform an automated security test with each pull requests. Fixing a bug early in the development process, saves you time and efforts, as posed by the rule of ten. That's why companies, with large development teams, such as Continental (+19k developers), rely on special tools, that help them to speed up the development process.
➞ Learn how Continental managed to test a large safety module with 18,000 lines of code (LoC) within only one week.
