How to write powerful automated API tests with Postman, Newman and Jenkins

Unless your development team is running on a six-month or an year-long cycle, you would be practicing Continuous Integration. From the ThoughtWorks definition of Continuous Integration:

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

While source control systems have made it trivial to set up shared repositories of code and view what people have been sharing, few teams are able to practice well the second part – verifying check-ins through an automated build process. Writing tests and especially tests for APIs, can be a tedious process. Nobody would disagree that having tests is good and we should be running them as often as possible. In practice, writing tests, verifying whether they are working, setting up the testing environment and then eventually plugging them with the build system is hard. Teams tend to skip this part accruing a ton of technical debt in the process.

We believe that in the context of API development, this process can be made a lot easier. This is where Postman departs from just being a REST client in your arsenal. Postman contains a full-featured testing sandbox that lets you write and execute Javascript based tests for your API. We won’t go into the specifics here but do check out this tutorial for more detail.

The next logical step is hooking up Postman with your build system. This is where Newman, Postman’s command line companion comes in. Think of Newman as Postman’s Collection Runner engine that sends API requests, receives the response and then runs your tests against the response.

In this tutorial, we are going to use Jenkins as an example. The same steps apply everywhere though. Jenkins is one of the most popular continuous integration servers available right now. Jenkins primary goal is to help you build and test software projects continuously. You can also monitor executions of externally run jobs too.

Newman and Jenkins are a perfect match. Lets start setting this up. We are using Ubuntu as a target OS as in most cases your CI server would be running on a remote Linux machine.

Install Jenkins: The process is straightforward. Just follow the instructions here.

Install NodeJS and npm. Newman is written in NodeJS and we distribute the official copy through npm. Follow the instructions for Linux to install nodejs and npm here.

Install Newman through npm install -g newman. This would set up Newman as a command line tool in Ubuntu.

Run a sample Postman Collection. We are assuming that you already have a Postman Collection with some tests. I am using a collection that sends two requests to echo.getpostman.com. You can download this sample collection if you want to follow the example. This is what the output looks in Postman’s Collection Runner:

Some of my tests are failing intentionally in the screenshot. We will fix this later in the tutorial.

To run this collection inside newman, use the command [command here]. If everything is set up nicely, you should see the output below.

Jenkins exposes an interface at http://localhost:8080. This is what it looks like:

Create a new job by clicking on the “New Item” link on the left sidebar. Then select a “Freestyle Project” from the options shown. I am calling my project Jenkins_Newman_Test.

Add a build step in the project. The build step executes a Shell command.

The command is:

newman -c jenkins_demo.postman_collection --exitCode 1.

Note here that we are using the newman command parameter “exitCode” with the value 1. This denotes that newman is going to exit with this code that will tell Jenkins that everything did not go well. Click the save button to finish creating the project.

Lets run this build test manually by clicking on the “Build Now” link in the sidebar.

Jenkins indicates that the build has failed with a red dot in the title. We can check why with the console output from newman.

Click on the “Console Output” link in the sidebar to see what newman returned.

Let’s fix these tests inside Postman and then try again. You can download the updated collection here.

I have updated my collection now. Running these tests inside Jenkins tells me that everything worked!

Jenkins indicates that the build succeeded with a blue ball. If you are wondering why it is not green, check out this article.

Let’s change the Jenkins build trigger as every 30 mins. You can do this by clicking on “Configure project” in the main project window and then scrolling down.=. The syntax for setting the frequency is H/30 * * * *

And we are all done! Jenkins will now run Newman every 30 mins and will tell you whether the build failed or succeeded. In a bigger set up, Newman will be part of your build process and probably not the entire process. You can set up notifications and customize Jenkins as per your needs.

You can use a wide variety of other configurations to make your collection more dynamic. Check out the other tutorials on our blog to see how to do this. Postman, Newman and Jenkins will give you a lot of power along with flexibility. We are continuously evolving the testing sandbox and let us know your suggestions on our Github issue tracker. Of course, do share this article on Twitter if you found it useful!