Get early access to Postman’s API performance testing: simulate real-world traffic
When it comes to APIs, you may have wondered some or all of these questions:
-
How will my APIs perform in real-world situations?
-
How will the response times change when multiple users are sending requests at the same time?
-
Will my users see acceptable response times when my system is under load or will they see errors?
-
How can I identify performance bottlenecks that may become major production issues?
If so, you’ve likely realized that as you scale, your API’s performance impacts the success of your products and business.
In today’s fast-paced digital world, providing a great user experience is essential to building a sustainable business and staying ahead of competitors. APIs have increasingly become the backbone of such modern businesses, and the quality and reliability of these APIs impact how customers experience a product. To ensure user success, you need to know that your APIs meet the expected functionality (through functional testing) and also that they can handle the expected real-world traffic (through performance testing).
API performance testing helps you gain confidence in the quality and reliability of your APIs, which helps you ensure a good user experience.
While the Postman API Platform has already been your trusted ally in the functional testing of your APIs, today, we are happy to announce the early-access release of API performance testing with Postman! You can request early access here.
Let’s understand more about API performance testing and how it can help you.
API performance testing: what and why?
API performance testing involves testing your APIs by simulating real-world traffic and observing the behavior of your APIs with the given traffic. It is conducted to evaluate how well an API meets performance expectations, such as response time, throughput, and availability under different loads and stress conditions. This will help you to:
-
Ensure your APIs can handle the expected load and check how the API responds to changes in the load (load is the number of parallel users hitting your APIs at the same time).
-
Optimize and improve the API performance for a better user experience.
-
Identify any bottlenecks, latency, and failures and determine the scalability of the system.
API performance can be measured by response times, throughput, and error rates under a simulated load. All of these attributes affect the satisfaction of your API’s users.
There are different kinds of API performance testing: load testing, stress testing, endurance testing, and spike testing. We will focus on load testing as part of this release.
What is load testing?
Load testing helps you test the performance of your APIs under a simulated load that resembles the expected real-world scenario. The key term is “expected.”
The goal of load testing is to let you observe and benchmark the performance of your APIs under expected real-world load. It also lets you identify performance bottlenecks and improve your API performance.
For example, a realistic load testing of Postman’s Public Get Collection API would be to test it with 100 parallel users sending requests continuously (producing anywhere between 100 requests per second up to 1000 requests per second).
Get early access to get started now
Don’t forget to request early access here.
You can use the Collection Runner to set up a performance test with Postman by following these steps:
Step 1. Select a Postman Collection or select an environment and click on Run:
Please note that all your pre-request and test scripts are also executed along with the requests.
Step 2. Select the Performance tab under Runner and specify the load settings, click Start:
Note that each virtual user executes the selected sequence of requests as represented on the runner (requests in the sequence are executed in serial order). Multiple virtual users execute this sequence this in parallel, creating a load for your API workflows.
Step 3. Observe the response times and throughput (requests per second) in real time:
Step 4: Once the run is completed, observe and identify bottlenecks in response times and requests per second:
Not able to see the feature while clicking early access as well from my account. Even from desktop it is not accessible.
Hi Harmeet,
The feature is now available on our Canary app which you can download from here .
Would love to hear your feedback!
This is so cool and useful. When does early access come into effect?
Hi there! I’m excited to sign up for the Performance testing feature on Postman. It’s great to see that you’re always improving your platform and offering new tools to help us streamline our API testing. Looking forward to using this new feature!
Hey, this will be very useful. When will the early access come into effect? (sorry, I might have written this comment once before, but I’m not able to find it)
Hi Alex,
The feature is now available on our Canary app which you can download from here.
Would love to hear your feedback!
Hello! Is there a troubleshooting guide? I did a SUPER quick try and a collection that runs successfully is failing with the performance run… Curious if there’s troubleshooting guide for authentication issues so far 😀
Hi Marianne,
We are planning to add some more details about the error rates so you can understand what type of http or non-http error you are receiving and to what extent. For the time being, I would suggest to update to the latest canary version (we rolled out a few fixed last week) and run the collection using Collection Runner and resolve the auth issues. Once, they are resolved- you can try to run the performance test. If you are still seeing issues, you can send me more details about the issues you are facing on [email protected] or setup a call using my calendly link.
Also attaching a how to guide of Performance Testing incase it helps.
Its a nice feature. Will it replace the need for the other load testing tools like JMeter, Loadrunner, NeoLoad etc. for API testing? Can you please explain how to leverage this feature for high volume load tests?
Hi Sridhar
I’m a big user of jmeter and previously loadrunner. I think at this stage the postman performance function would need to go a long way yet to replace jmeter. I definitely think though that the postman performance offering has a place for devs and test engineers for ramping up the load on APIs to capture early performance issues.
Hi Sridhar,
Thanks for asking the question. Our intent with Performance Testing is to make it available to our users faster and with very less learning curve. The current version of the product, simulates the load from your local machine and hence there is a limitation on the maximum volume of load that you can generate. We will incorporate your feedback to include a solution in our future roadmap to generate a higher volume of load. Would love to know more about your usecase though and if you are up for a discussion, here is a link to my calendar.
Thanks Again!
Malvika
Does it impact our collection runner limit count ?
Saravanan, Performance test runs doesn’t impact your collection runner usage in anyway.
What’s the max. Virtual users we can use? Do we have options to add internal/external load generators?
Hi Raju,
Currently, the canary app supports a maximum of 200 Virtual Users for all accounts (free and paid). We are working towards improving the app efficiency so we can support more virtual users. We do not have an option to add internal/external load generators yet. I would love to understand this better from you though. Would you be up for a discussion for this?
Thanks!
Malvika
Is there a limit on vusers per postman setup in a local machine ?
Any option available to add up/integrate multiple postman softwares installed in multiple machines in order to simulate huge load
Hi Tejaswar, thanks for asking this question. The performance test on Postman happens on your machine right now i.e. the virtual users are instantiated from your local machine. The Virtual Users require CPU and memory as basic resources. The higher the number of VUs, the more resources from your machine will be needed.
Therefore (and as you might have guessed already) the virtual users will be constrained by the specifications of your local machine such as amount of CPU and Memory available.
To simulate bigger load (not constrained by local machine) to do things like stress testing or spike testing, we are evaluating a few solutions and the future releases might incorporate this.
Is there a guide to use performance test feature? please advise.
Hi, You can read this how to guide on using the performance testing feature on Postman.
How to download the performance testing report
Hi Subash, thanks for the question. I am curious to know more about your usecase for downloading the report.
There are two use-cases for downloading the report that we have heard so far- (1) Sharing with other team members (2)Saving it in a place in order to generate insights from past reports.
Currently, your past performance testing report is saved in Postman under Collection-> Runs -> Performance Runs. You can invite more members to the workspace to share the reports with them.
I am curious to know more about your use-case.
My early access just got approved
Thanks for adding this new feature. I requested this feature last year through a feedback call happened last year. I am glad it’s available now.
I would like to know the VU limit available for free. How do we scale up the load to 1000’s of VU’s? Do we have to pay for it? How do we generate comprehensive test result reports? Do we have a schedular and alert/notification system available?
Awesome new feature! Is it accessible in command line via Postman CLI or Newman?
I already have the access to the Canary instance, however the performance functionality does not work. I get 100% of error and I don’t have information about errors. My results table looks like:
Request Total requests Requests/s Resp Time Min Max 90th Error%
POST 50 1.4 – – – – 100.00
GET 50 1.4 – – – – 100.00
Hi Diego
I get the same issue as well. I had a call with the guys in India and they are looking into the problem
I have signed up for it but still unable to see that option there for performance testing.
Hi Rushabh,
Could you drop me a note on [email protected] so I can understand your issue better. Also note that the feature is not available to our enterprise users yet. If you want to try it from a personal account, you should be able to see the option under “Runner”. This how to guide talks more about where the feature can be found and how to use the same. Looking forward to your response.
While the virtual users’ capability is very welcome and long-awaited, it will be good to understand the basic assumptions around this function. For example, is there an assumption that one API user must allow multiple concurrent sessions for the virtual user scaling to work?
Suppose the API only allows one concurrent session per user. Can we provide a list of usernames and passwords that can be used for creating an independent session (JSESSIONID cookie) for each virtual user (like in JMeter)?
This is a blocker for us, so any help is much appreciated!
When testing APIs it is often more important to set up throughput than using virtual users. I use Jmeter for my API testing and they have a great little feature that allows you set up the throughput per second/minute for example.
Found the auth issues, but fixed them by adding the auth at the start of the collection runner.
same as mentioned by “Marianne”
Looks nice so far, but i wonder if we will be able to save our runs report and read our errors in more details like endpoint response etc?
Hi, Your past performance test reports are saved in Postman and are available under Collection-> Runs -> Performance Runs.
We are working on adding more details about the error in the report so you can troubleshoot and/or get more information about your run. Thanks for your feedback. If you like to speak to us about this or any other suggestions, here is a link to my calendar.
tell me how to see what statuses are coming for errors in the tests?
Hi Roman,
We are working towards this improvement! We will soon be able to show you a breakdown of the http response codes so you can understand the error rate better and troubleshoot. Thanks for your interest in the feature! We are tracking it here.
Regards,
Malvika
So great news!
Do you have plans to expand this, so performance Postman collections can be ran using CLI?
For example, with Newman
Hi Leo,
Thanks for your response and for your suggestion.
We might look at this in the future product iterations. Currently, our focus is to give users this functionality as seamlessly to run performance tests locally from their machines.
We have a feature request logged in on our github issue tracker for enabling Performance runs on CLI.
Thanks!
Good
Downloaded postman canary , details listed below , but still not able to see the performance tab .
I read the post saying not for enterprise users , I didn’t sign in.
Version 10.13.7-canary230510-0719
Architecture x64
Platform win32 10.0.19044
Hi Surendran,
Request you to login using your personal email account and give it a try. You are right, we haven’t enabled the feature for Enterprise Accounts. However, you would not be able to access it without signing in either.
Please feel free to drop us a note at [email protected] if you need further assistance.
Hey Malvika,
Can you provide any documentation or any details on how to start the performance runs via CI/CD Pipeline. Is there any functionality to create a Jenkins pipeline currently ? If so, could you please explain in detail.
Hi Onkar,
Thanks for asking this question. Currently, we are only supporting running of performance tests using Postman UI. However, we are tracking “Running in CI/CD Pipeline” as a feature request here . We will have this in our future product roadmap.
Regards,
Malvika
Do you have a guide on how to read the report that was generated?
Hi January,
You can read our guide on using performance testing to understand better.
If you need further assistance or have questions, please feel free to drop me a note at [email protected]. I would love to know your feedback and how we can simplify the experience.
Regards,
Malvika
Works great – thanks for building this, useful to have everything in one place.
One question: what determines the number of requests per user per second? I seem to hover at around 50-60% of the number of users, e.g. if I have 30 VUs I get around 17 requests per second – why is that and how can I change that?
Hi Reiner,
Thanks for the interesting question.
The throughput(requests per second) generated depends on many factors including how optimized the process is, the nature of collection being run, logic inside the pre-request or test scripts, system specifications and also the server response times- to name a few. We have made some recent optimisations to the process and released the updates on our latest canary build (version 10.13.9-canary01). I would request you to update your app and give this a shot. Let us know your findings.
Regards,
Malvika
After a Performance run, how can I see more details of the error? if there is some % of error rate.
IMPROVEMENT: Please add export perf runs to Excel or pdf for all runs or per perf run.
Hi Satya,
Thanks for the feedback! We are adding the error rate breakdown on the product. This will be released in the next few days.
We are also tracking “Exporting of results” as an improvement here. This is on our roadmap too!
Regards,
Malvika
Hi Malvika,
I did fillowed your instruction also, i am using canary but i am not able to see performance tab when i hit run. I tried reachign you over email but no support received.
Hi Subhajeet,
We hope you received our response and are able to access the feature using your personal account. As mentioned, we haven’t enabled the feature for Enterprise Teams and hence you were unable to access this feature.
Regards,
Malvika
Hi,
Can you guide me on how can we download the performance test report, can we download it in PDF / Excel format will be much help.
Thank you
Hi Abhi,
Download functionality is not available currently. We are tracking this as a feature request here.
Regards,
Malvika
If anyone has issues with 100% error rate, remove any environment variables and try again. i was using a variable for part of the endpoint url and for an api token. once i removed these are set them manually it worked fine. if it also helps I could pass a variable in the body such as {{$randomFirstName}}, which worked fine. it seems to only have issues with environment/collection viriables or so. thank you 🙂
hi Chris,
This seems like an unexpected behavior. Performance testing feature is built to take all variables into account just like Collection Runner does.
We are tracking this as an issue here. If you can provide more information about your system specs and perhaps a screen recording or sample collection, it will help us investigate this issue further.
Regards,
Malvika
Is there an ETA on a public release? I put in a request for early access, but wasn’t selected. We’re currently in need of a load testing tool and this would be amazing.
Hi Kristian,
The public release is scheduled in the next 4-5 days. You can still try the feature on our canary version meanwhile by downloading the canary app from here.
You can drop me a note at [email protected] if you need any further help.
Regards,
Malvika
Hi Malvika,
‘Virtual users’. What users does it take to run?
Could you please elaborate on this one?
Can we provide a list of usernames and passwords that can be used as virtual users? And monitor performance of each user ?
Hey Sharan,
Virtual Users are parallel users that hit your requests creating a traffic for your apis. You can specify the number of Virtual Users that you would like to simulate while setting up the run. Read more about it on our documentation here.
Currently, you can’t provide usernames/passwords for Virtual Users. However, we are looking to add this as an improvement to the product soon. Tracking it here.
Please let me know if you want to discuss anything further, we can do this over a call. Please feel free to drop me a note at [email protected] if you are interested.
Regards,
Malvika
Hi Malvika,
I hope you are well. Is there any update on the issue of 100% errors when using the performance runner?
I did have a call with your product manager weeka ago but never heard anything back. We are big postman users so we are keen to get this resolved as soon as possible.
Hi Roger,
Most of the issues resulting into the error rate (such as environment resolution) has been fixed. You will need to download the production app to try the fixes. You can download the prod app from here: https://www.postman.com/downloads/
We have also added more details on error rates that are being observed for better troubleshooting. Read about it here: https://learning.postman.com/docs/collections/testing-api-performance/
Regards,
Malvika
Downloaded Postman Canary app and using the latest version “Postman v10.13.9-canary01”. I’m not able to see the ‘performance’ option under the runner tab. Under which version this option is available?
Hi Team,
Thank you for adding the feature.
What parameters do we have to add to use the performance feature in CLI mode when running the API tests from newman.
Request you to kindly suggest.
Regards,
Satish
Hey Satish,
Performance Tests are not yet added to newman/postman cli. We are tracking this feature request here: https://github.com/postmanlabs/postman-app-support/issues/12008
Feel free to share your thoughts/suggestions on the above github ticket. If you like to talk to me, you can drop me a note at [email protected]
Regards,
Malvika
hi,
How to perform load test Python API if parameter file path
Thank you for the detailed information, very much appreciated. We were much excited to test out the newly launched performance testing feature in the postman. To start with we wanted to test a single API, for a single user. We executed a 10 minute performance test, the average response time was 1 second and we didn’t add any wait times so for the 10 minutes duration we would expect 600 total requests but we observed only 300 total requests. we wanted to understand if there is any implicit wait times or any other limitations reducing the total requests that should be happened as per the configuration. Awaiting your response.
Hi Rajesh,
I am dropping you an email to discuss and understand this further. We do have some wait times (that can range from 50ms to a bit longer depending on whats happening on the client). Besides this, if there are scripts in your requests, that could take up some processing time too.
Regards,
Malvika
I am loving the Performance runner feature with Fixed & Ramp-Up, and have been using it everyday at work. My only ask, would it be possible to increase the number of Virtual Users (VU) to load test further. That would make this feature very powerful to test it in near-production environment.
Hi Giri, thanks for your feedback and glad to hear you are finding this feature useful. Performance Testing feature is now generally available on our Production app. You can download Postman prod app from here. In this you have access to a 100 Virtual Users on a free or basic plan and upto 500 Virtual Users on a Professional Plan. You can read more about our plans and pricing here.
If you see any concerns or would like to talk to me 1:1 you can block time with me on my calendar.
Regards,
Malvika
Hi Malvika Chaudhary,
Hope you are doing good.
Malvika, I download the POSTMAN Canary build but the problem I am getting is under the performance section RUN button is not enabled for me. Moreover, I am getting an error that says ‘Something went wrong while setting up your system for performance testing’.
Can you please assist why I am getting this error?
Hey Simranjeet,
The postman performance testing feature is now generally available and can be accessed via our Production app (downnload from here)
I am dropping you an email to understand further if you still face this problem in the prod app.
Thanks,
Malvika