In web security, fuzz testing is becoming more popular by the day. RESTler expands the reach of fuzzing REST APIs by providing an intelligent, automated solution to do fuzzing given a Swagger/OpenAPI specification. In this article, Marina Polishchuk explains how RESTler works, how it is used and which bugs developers will find with RESTler.
How Important Is it to Fuzz REST APIs?
Web APIs are everywhere right now. Over the past several years, there has been vast growth in public web APIs and microservice architectures. These services, which are continuously evolving, imply an increased attack surface and thus increasing demand for API testing to keep up.
This video is an excerpt of Marina Polishchuk’s talk at FuzzCon Europe - WebSecurity Edition. View the full recording for an in-depth look at RESTler.
Stateful REST API Fuzzing
RESTler is the first stateful REST API fuzzer, developed at Microsoft Research. Our research was initially motivated by Microsoft-internal feedback about the need for deeper security testing for REST APIs. Previously, there were two main options: either using tools that required generating traffic that is then replayed and fuzzed, or loading up an API specification and fuzzing each request in isolation.
There are many ways to do traffic capture, but of course, the fuzzing coverage will depend on having representative traffic that can also be automatically updated when the service behavior changes. Fuzzing each request independently typically yields low coverage without a human in the loop to guide the fuzzer to improve coverage, e.g. by pre-creating the necessary resources. These solutions are simply too expensive to maintain with good coverage for large, frequently updated REST APIs.
Key Features of RESTler
First of all, RESTler is a stateful REST API Fuzzer, which means that a test case runs sequences of requests. Resources created by earlier requests are consumed by subsequent requests. In fact, requests are only fuzzed if their prerequisite resources can be created. This results in higher code coverage of the service because fuzzing is performed in a state where request paths are valid and there is a good chance that the unique service code that handles the request being fuzzed will be reached. The fuzzing algorithm also avoids redundant testing of invalid inputs that were properly handled by the service.
Secondly, RESTler is automatic, as it uses the Swagger/OpenAPI specification to generate the fuzzing grammar. It provides good coverage out-of-the-box for simple, well-documented APIs.
Last but not least, RESTler is extensible: it supports several security checkers that can be optionally enabled while fuzzing, and it also allows you to write your own security checkers and plug them into the fuzzing loop.
How RESTler works
RESTler first takes an OpenAPI specification and compiles it to produce a fuzzing grammar. The fuzzing grammar contains the information on how to fuzz each request type, and which resources must be passed between requests.
Then, the test engine executes requests against the service and analyzes test results. The engine has feedback loops to learn from past service responses and avoid redundant test cases. If bugs are found, they are bucketed and checked for reproducibility.
Classes of Bugs Found by RESTler
- API Specification
One use case for RESTler is validating the API specification. If the specification is either under-documented or inconsistent, RESTler will have low API coverage out of the box. To improve coverage, you will find that you can either use RESTler to work around these issues in specifications or fix them. In addition, to help with initial positive coverage, RESTler uses payload examples, if they are included with the specification. Similarly, low coverage may indicate that examples are incorrect, such as if they have diverged with the service implementation. -
Input validation
RESTler is very effective at finding input validation issues. The test oracle for these is getting 500 server errors instead of an expected 400.
-
Authentication
With RESTler, we found several types of bugs in the authentication. RESTler offers specific checkers to test whether you can access another user's resources without authorization. However, you can also just run RESTler with an invalid token across a large API. If it runs, and you see some 200s in the logs, this can be seen as an indicator that something might be going wrong.
-
Resource Management
RESTler offers specific checkers to test certain resource leak scenarios. However, service resource exhaustion can happen due to RESTler fuzzing, and can’t be detected by RESTler because it has no knowledge of the internal state of the back end service. If RESTler fuzzing causes resource exhaustion, this can be caught by service alerts and service incidents.
-
Data Leaks
Although RESTler does not yet have automation to find data leaks, some leaks have been found by examining the logs. Once such a leak is identified, a checker can be written to look for such leaks in responses across the entire API.
When to Do REST API Fuzzing
If you can run REST API fuzzing during development, a big problem is a time and the complexity of having a reliable test environment in CI/CD that will not break your builds. One strategy to mitigate this is doing RESTler testing during deployment validation, where a test environment is already available for another integration testing.
What Can We Expect From RESTler in the Future?
One thing we are driving toward is better coverage out-of-the-box. Today, if your service is relatively simple, the OpenAPI spec usually has enough detail to get near 100% coverage. If it is more complex, some initial configuration is required.
We have also created a self-hosted REST API Fuzzing-As-A-Service platform, which can help you bootstrap REST API testing at scale. We are working on enhancing this solution to support more CI/CD scenarios, as well as regression fuzzing.
We recently received some great community feedback about specific security checkers that we did not initially cover. If you are interested or have already implemented a new checker, and you would like to integrate it or share your results, feel free to contact us on Github.
To find out more about Stateful REST API Fuzzing, have a look at the recording of Marina Polishchuk’s talk at FuzzCon Europe - WebSecurity Edition.
About Marina Polishchuk
Marina Polishchuk is a Research Software Engineer at Microsoft Research. She is passionate about systematic testing complex software as well as teaching engineering best practices. Over the last 15 years, she has written testing tools for the C++ compiler and the SQL server database engine. In her current work, she is focused on developing tools for finding security and reliability bugs in cloud services.