How to review Postman test results

When you review API test results in Postman, you want to know if the tests pass or fail. It’s also important to know what is being tested and why a test fails, so you can troubleshoot and debug unexpected API behavior.

Read on to learn more about the following topics regarding Postman tests:

• See what is being tested
• Know why a test is failing
• Debug and troubleshoot unexpected behavior
• Write more readable tests

Introduction to Postman tests

Lots of people use Postman to test their APIs. In addition to exploratory testing, you can also write tests in Postman to make assertions about your APIs. These types of tests confirm that your API is working as expected, that integrations between services are functioning reliably, and that any changes haven’t broken existing functionality.

Go to the request builder in the Postman interface, and find the Tests tab. This is where you’ll write your tests in JavaScript in the top panel. In the bottom panel is where you will see your results.

A test in Postman is defined by the pm.test() method that lets you add instructions for your assertions or expectations. The method accepts the following two parameters.

• Test name: The first parameter is a string representing the name of the test. This helps you identify the test in the Test Results section and also communicates the test’s purpose to the consumers of your collection.

• Test specifications: The second parameter is a function where you define the assertions on response data, using a readable Chai.js BDD syntax like the pm.expect() function. Your tests can include multiple assertions as part of a single test to group related assertions. If you use multiple assertions this way, the test will stop executing as soon as any assertion fails and the remaining assertions will not execute. You can also use pm.response.to.have.* and pm.response.to.be.* to build your assertions.

In this example, we can see the results summarized in the Test Results section. Three out of four tests have passed.

In the next section, let’s look at the failing test to see what’s happening.

Review test assertion errors

Let’s look at the failing test called “My Best Test” in the Test Results section of Postman. The test result is composed of the following elements.

• Test status: This label indicates whether the test has passed or failed. In our example, the “My Best Test” test has failed and the label is colored red with the text “FAIL.”
• Test name: This is the name of the test that corresponds to the test written under the Tests section of the request.

Assertion error: If there is an error resulting from the test specifications, the error message is displayed following the test name and a pipe symbol (|). The assertion error message corresponds to one of the assertions written in the Postman test. In our example, there’s only one assertion.

In the next section, let’s look at the test assertion itself to see what might be failing.

Review test assertions

The second parameter of the pm.test() function is a function where you define the assertions about response data, using a Chai.js BDD syntax like pm.response.to.have.* and pm.response.to.be.* to build your assertions. It is also common to use pm.expect(), as in our example here, to write predicate expressions that can be evaluated to a boolean value, i.e., true or false. An assertion succeeds if the predicate expression returns true.

pm.expect(pm.response.json()).to.be.an(“array”)

The JavaScript pm.response.json() is the syntax that allows us to parse the JSON response from our API. In the code above, we expect the response body to be an array data type. If this is true, our test would pass. If this is false, our test would fail.

In reviewing the assertion error message in our test results, we learn that our response body appears to be an object. To break it down further, pm.response.json() in the assertion corresponds to { info: { count: 126, …(3) }, …(1) } in the assertion error message. In our assertion, we expected that to be an array, but can see in the assertion error message that it is an object instead.

If the assertion error message syntax is unfamiliar or still unclear, let’s investigate further.

To inspect the response body more closely, we can either select the Body tab of the response or inspect the response in the Postman console. Inspecting the response like this allows us to verify the response is indeed an object. Our test was making an assertion that the response would be an array, but the response is an object!

In the next section, let’s talk about what to do now that we know what is causing the test error.

Resolve a test failure

API producers write tests to make sure their APIs are functioning as they expect. If you open a request and see that it has tests, those were added by the collection author to ensure the API is functioning as they expect.

API consumers also write tests to make sure the APIs they use are functioning as they expect.

When a test fails, the fault can lie with the producer or the consumer, and sometimes it’s a failure to align expectations between the two.

In one scenario, a failed test can mean the consumer needs to update the request in order to comply with the expectation. For example, during self-paced onboarding and training where you encounter failed tests, the collection author could verify the user has submitted an authorization token, included certain parameters, and so on. In this case, the test provides feedback to the user about the expected way to use the API.

In another scenario, a failed test can mean the producer needs to update the API, or the test itself, to align expectations with reality. This is like seeing a typo in the API producer’s documentation. In this case, you can optionally report the issue by leaving a comment in the producer’s main collection (not your own fork) or contacting the producer directly.

In any of these cases, the test is a communication tool to convey expectations about an API.

Ways to resolve a test failure

• API consumer can update the API request
• API producer can update the API or test
• Person relying on the test (either consumer or producer) can update the test

In our example from the previous section, we can’t control what comes back from the API server as a consumer. Our request is working properly, but the test itself needs to be corrected to expect the correct data type. Let’s update the test to assert that our response is an object data type.

Now the test passes. We can confidently build an integration and use this API knowing that we are working with an object, and not an array. We can also run this test in the future to ensure the API is continuing to work as expected. If this test ever fails, we are alerted that something has changed with the API and we might experience breaking changes with our integrations or applications using this API.

In the next section, let’s discuss tips for writing more readable Postman tests.

Tips for writing better Postman tests

This article was about reviewing test results. However, if you are also responsible for writing Postman tests, hopefully you have already gained some new perspectives into how you can more clearly communicate a test’s purpose and provide more visibility into a test’s failure.

Follow these tips for writing more readable tests.

• Group multiple assertions: You don’t need to write multiple assertions for every element that you test, but grouping together related assertions allows you to write more rigorous tests and keeps them logically organized for those who review the test results and need to debug issues.
• Use messages and console statements: Use console statements to provide visibility into key areas of your testing and validate conditional testing and execution order. You can also prepend custom messages in your assertions to provide more information and clarity to otherwise ambiguous assertion errors.
• Use descriptive, consistent, or dynamic test names: Be consistent with your naming conventions so reviewers don’t need to guess what’s really happening when a test fails. You can be descriptive and also use variables within test names to provide more detail, especially if the same test is used for multiple scenarios or iterations.

If you’re writing tests, it’s also important to make sure your tests fail. It’s easy to write tests that pass. Make sure your tests are checking what they’re supposed to be checking, and verify that they can also fail. For example, an API producer can input a mismatched set of parameters to ensure their API fails “properly” in certain conditions.

In the next section, explore some of these intermediate techniques in a hands-on exercise.

Check your knowledge

Fork the following sample collection to your own workspace if you want to follow along with these examples and try it out yourself. There are four additional practice exercises to do on your own to check your knowledge, and explore some of the more intermediate techniques described in the previous section of this article.

Fork the collection to your own workspace by clicking the Run in Postman button:

 

Watch the video to follow along: