Measure your Postman Collection’s quality with the Postman API and Spectral rules

Postman Collections are an excellent tool for collaborating with users in the API design, development, testing, and documentation processes. You can share your collection with others, who can then help in the development process by providing their feedback in comments.

To help your API users, you should always define the following items in a collection:

• Documentation: Adding descriptions to the collection and its requests helps users understand the collection’s purpose and how your API works. At the collection’s top level, provide details for how users can get started. Describing requests in your collection helps users to understand how it works and what to expect from it.
• Examples: Providing examples of different responses (such as success, unauthenticated errors, and bad requests) for each request helps users understand the expected responses and adapt their code.
• Variables: Defining generic requests with global, environment, or collection variables lets users easily test APIs against different environments or conditions by simply changing the variables’ values.
• Authentication: Defining authentication at the global or request level makes it much easier for users to test the APIs.
• Tests: Providing tests in your collection can help your users get started testing your API’s behavior.

When a collection lacks these items, it becomes difficult for users to get any value from it or use it. If you intend to share your collection with users, whether internal or public, it’s good practice to properly define these items to improve the quality of your collections.

In this blog post, we’ll show you how to create a tool for your automation that evaluates the quality of a collection using the Postman API, Spectral CLI, and jq. You can find the code in our postman-collection-spectral-linter GitHub repository.

Best practices for defining the quality rules for a Postman Collection

The postman-collection-spectral-linter tool uses Spectral rules to detect any quality issues in a Postman Collection. Spectral rules are a method for defining the conditions that a JSON or YAML file must meet. The Spectral CLI tool processes a given JSON or YAML file and checks if that file meets all the rules in this file.

For example, we define a collection-description-is-mandatory rule that checks the collection.info property in the collection’s JSON for a description property:

rules:
  collection-description-is-mandatory:
    description: Collection must have a description.
    given: $.collection.info
    severity: error
    then:
      field: description
      function: truthy

Measuring quality

Based on our experience as Postman users and our experience working with Postman users around the world, we have seen several key elements that a collection should include to be considered a high-quality collection. We’ve defined these features in the tool’s rules.yaml file:

Errors

• All requests must have a description.
• Requests must have a URL.
• All requests have at least a 2XX or 3XX example.
• All responses must have a status code.
• DELETE responses must have a 200 or 204 status code.
• Requests with path variables require a 404 response.
• All request query parameters must have a description.
• All request path parameters must have a description.
• All PATCH requests require a request body.
• All responses must have a status code.
• All authenticated requests require a 401 response.

Warnings

• HTTP should not be used
• Response bodies should be in JSON
• All folders should have a description
• Request names should not end in “Copy”

This ruleset is not exhaustive and may not necessarily meet your use case. You can add your own customized rules to fit your needs, extend any of the default rules, or remove rules you don’t need in the rules.yaml file.

How it works

When you run the tool’s shell script (Bash) command, it:

• Retrieves the collection in JSON format with the Postman API’s GET /collections/{collectionId} endpoint
• Uses Spectral CLI to validate a set of defined rules against the collection
• Checks whether the Spectral results contain errors and returns them the script’s exit status. If any of the rules fail, then the command fails

The script is very straightforward:

# Perform a curl request against the Postman API. Save the request body in a file called collection.json
curl -s -H "x-api-key: $API_KEY" https://api.postman.com/collections/$COLLECTION_UID | jq . > _collection.json

# Lint the collection using spectral. Save the result in a JSON file
spectral lint _collection.json --ruleset $RULES_PATH -f json --quiet > _result.json

# search for errors (severity=0) in the result file. If there are errors, exit with a non-zero status code
jq '.[] | select(.severity==0)' -e _result.json >/dev/null

if [ $? -eq 0 ]
  then
    exit 1
fi

• In line 2, the tool uses the Postman API to retrieve the collection in JSON format. It then uses jq to beautify the output and saves it as a temporary _collection.json file.
• In line 5, Spectral CLI checks the rules in the rules.yaml file, then dumps the results to a temporary _results.json file in JSON format.
• In line 8, jq performs a search for errors with a 0 (error) severity.
• In line 10, the tool checks if the previous command succeeded with an execution status of 0. In that case, it returns an error execution status of 1. You can use this status code in scripts to evaluate if the collection contains any errors. Additionally, you can use the _result.json file contents with the error details to perform any kind of notification.

Using the tool

Once you clone the GitHub repository, you can run the shell script with the following command:

./linter.sh -c <collection_uid> [-k <postman_api_key>] [-r path_to_ruleset_file]

In this command, the postman_api_key parameter defaults to a POSTMAN_API_KEY environment variable and path_to_ruleset_file defaults to the rules.yaml ruleset. You can find the ruleset in our postman-collection-spectral-linter GitHub repository, and you can extend it by adding new rules or turning off any existing rules you don’t need.

Summary

The Postman API provides full CRUD capabilities that let you manage your collections. You can easily retrieve the collection JSON using the Postman API and a valid API key. To continuously validate whether your collection follows your quality standards, you can define your Spectral rules and check those rules automatically with the Spectral CLI. It’s important to integrate quality checks in your CI/CD workflows to improve the overall quality of your collections.

Next steps

• Define your own rules, adapted to the quality level you want in your collections. You can extend the best practices rules we provided in this blog post.
• Use the tool to automatically notify users when they don’t meet the quality standards to help your team maintain great collections.
• Use the detailed results of the Spectral CLI to send notifications or add comments to a collection with the Postman API.

Do you have any use cases related to collection quality? We would love to hear about it—please let us know in a comment below!