Negative Testing for More Resilient APIs

Joyce

May 27, 2021

I’ve previously shared how to build resilient APIs with chaos engineering and security hacking with the Big List of Naughty Strings. I’ve also broken some hearts, so I have some experience with breaking things. In this post, let’s go beyond basic test automation in Postman, and learn why intentionally trying to break things increases the resilience of your applications. We’ll also explore some common negative test scenarios.

Happy path vs. unhappy path

During product design and development, stakeholders focus on intended user behavior. Product owners describe this happy path with business requirements. Engineers push code to enable users to do these things. A user who demonstrates this expected behavior follows the happy path.

These are examples of happy paths:

• User creates a new account
• User signs in with their new credentials
• User retrieves information about their account

But what happens when there’s unintended behavior, like when a user submits invalid inputs? These unhappy path scenarios are often neglected in design and development. But with enough traffic, unexpected inputs will inevitably come from both well-intentioned and malicious users.

These are examples of unhappy paths:

• User creates a new account without providing all required inputs
• User signs in with invalid credentials
• User pastes a SQL query that gets executed (maliciously or unintentionally)

A strong test plan covers both happy and unhappy paths. Positive test cases describe happy path scenarios, where errors result in failed tests. Negative test cases describe unhappy path scenarios, where expected errors result in passed tests.

Trying to break the Unbreakable API

In this “How to Break an API” livestream with my colleagues Trent McCann, quality engineering manager, and Evan Lindsey, lead SDET, I tried to break the Unbreakable API. This API was engineered by Evan to demonstrate the importance of validating inputs and handling errors.

Typically, architects and engineers of an API are very familiar with the vulnerabilities of that API. However, they’re focused on enabling happy path user stories. They may not be thinking through all possible scenarios a user might encounter when interacting with the API. In the next section, let’s explore some conventional ways to break an API.

A recommended testing flow

Unbreakable API Lite is a simplified version of Unbreakable API used in the “How to Break an API” livestream. Instead of managing authorization for multiple roles, the role of admin is removed and employee is renamed to user. In Unbreakable API Lite, once you create a user and set the returned token, you will have access to all available endpoints.

The collection Testing Flow for Lite includes examples of positive and negative test scenarios for the API referenced in Unbreakable API Lite. For engineers, Testing Flow for Lite can be a guide for input validation and error handling. And for testers, it can be a recipe for mischief.

1. Fork the collection: Fork the collection Testing Flow for Lite to your workspace. You may need to enable your public profile if you haven’t already.
2. Create a new user: Find the User create request, and update the values under the Body tab. Hit Send to create the new user, and also set a collection variable called userToken that can be used in subsequent calls.
3. Step through the collection: Explore the positive and negative test scenarios outlined in the remaining folders. Under the Authorization tab, notice the authorization method used for each request. Under the Pre-request Script and Tests tabs, notice code that runs before and after you send each request.
4. Run the collection automatically: This collection can also be run to automate your testing flow. This is done using the Runner in Postman, Newman from the command line, or Monitors on Postman servers. Remember to set up a new, unique user under the User create request before running the collection in its entirety.

Positive and negative testing

The examples in the Positive folder of Testing Flow for Lite describe happy path scenarios, such as:

• • Retrieve all movies
  • Create a new movie
  • Retrieve the new movie by ID

For positive test cases, you write a test asserting a successful response like receiving a 200 OK, or similar, HTTP status code. If there are errors in those assertions, the test should fail. For example, to test the endpoint of a user creating a new movie, write a Postman test and assertion like this:

pm.test("Status code is 200", function () {
  pm.response.to.have.status(200);
});

The examples in the Negative folder describe unhappy path scenarios, such as:

• Retrieve all movies, using an authorization token when not needed
• Create a new movie, using an invalid authorization token
• Delete the movie, when cascade is not set on database cleanup

For negative test cases, you write tests expecting certain errors. If your application doesn’t return expected errors or swallows exceptions, your application isn’t handling unexpected input gracefully. In these cases, your tests should fail if you don’t see the expected errors.

For example, what happens when a user creates a new movie and submits an invalid authorization token? We expect the server to return a 401 Unauthorized, or similar, HTTP status code. To test that case, send a token that you know is invalid, and write a Postman test and assertion like this:

pm.test("Status code is 401", function () {
  pm.response.to.have.status(401);
});

The importance of negative testing

With negative testing, you are asserting the application accommodates unexpected user behaviors. When unexpected user behavior crashes the whole system, engineering rushes to put fail-safes in place, adding tests to ensure they are handling that exact scenario correctly going forward.

With negative testing, you’re proactively exploring these edge cases to increase the resilience of your APIs in production. It’s important to establish a workflow and foster a supportive organizational culture to do this, before your system fails.

Let us know your favorite way to break an API in the comments below. And level up your testing in Postman with these additional resources:

• Test examples in Postman public workspace
• Acing your API tests — what you need to know for test automation blog and examples
• Chaos engineering to increase resilience blog and recipe
• How to break an API livestream
• Security hacking with the Big List of Naughty Strings livestream

Technical review by Evan Lindsey.