Introducing the Next-Generation Postman URL Processor

Kin Lane

April 6, 2020

The essential ingredient of any Postman API request is the Uniform Resource Locator (URL). The URL provides all the details that your browser, servers, and developers need to find the API resource they’re requesting. A URL contains eight individual parts that collectively provide a uniform way to locate any desired resources on the web; that might sound pretty straightforward, but in reality it presents some very big challenges. At Postman, we’re constantly working on tackling these challenges, and this work has culminated in our recent release of the next-generation URL processor.

But let’s back up a bit and take a closer look at those challenges, because therein lies the value of the new URL processor.

What Are The Moving Parts of a URL?

To help understand the challenges involved with working with URLs, it helps to understand each of the moving parts of a URL, what service providers are putting into them, and what many of the assumptions are which are being made on the staggering number of URLs that exist across the web.

Here are the eight core elements of any URL, which provide the building blocks of each request that is made on the web:

• Scheme: The HTTP or HTTPS beginnings
• UserInfo: Any username or password
• Subdomain: The beginning of an address
• Domain: The Internet domain of the URL
• Port: The machine port to route through
• Path: The folder path to the resource
• Query: The parameters defining resources
• Fragment: The pieces of each document

All of these moving parts result in a complete URL that typically looks something like this:

http://sub.domain.com:80/path/to/the/resourceparameter1=value&parameter2=value#Fragment

Browsers, Postman, and other web services and tools use these URLs to navigate the web, and it’s how web and mobile application providers make their data, content, media, and algorithms accessible. As the name suggests, URLs provide us all with a uniform way to locate valuable resources that are made available in a public or private way, using a common, agreed-upon approach. While URLs have done amazing things to standardize how we make digital resources available online, there are two main challenges that Postman (and all other service providers) faces when it comes to using URLs in the wild, which exponentially increase the number of problems we might encounter:

• Parsing: Ensuring that Postman understands how to break down each part of any single URL
• Encoding: Understanding the different types of encoding that are possible for each of the parts

Each part of a URL has to be properly parsed and then encoded using a different set of rules that may be interpreted in different ways across different service providers, which introduces a great deal of complexity. Historically, Postman has had just a single approach to processing an entire URL, but with our next-generation URL processor, the Postman platform is more flexible while parsing and encoding URLs. This allows us to establish a clearer boundary for each part of a URL, which then results in a cleaner encoding.

How the Next-Generation URL Processor Works

Postman’s next-generation URL processor leverages knowledge gleaned from the way that a URL is constructed and then provides the least-ambiguous and most-forgiving URL parsing, based on learnings from the countless different URL combinations we’ve encountered over the years. Here are some of the items that make up this next-generation experience:

• Non-regex parser: This generates meta information about the URL while parsing, keeping it broken down as a data structure (object) through to the lowest depths of our code layers.
• Path-specific encoding: Postman can enforce the following characters as part of the path: “, <, >, #, ?, {, }, and SPACE. It can then apply precise encoding rules to just the path.
• Query-specific encoding: Postman can enforce the following characters as part of the path: “, #, &, ‘, <, =, >, and SPACE. It can then apply precise encoding rules to just the query.
• User Info-specific encoding: Postman can enforce the following characters as part of the path: ” “, <, >, #, ?, {, }, /, :, ;, =, @, [, \, ], ^, |, and SPACE. It can then apply precise encoding rules to just the user info.
• Encoding unsafe character: When unsafe charters are included as part of the URL, Postman will now identify and encode them depending on which part they are found.
• Trailing white spaces: Postman will not remove trailing white spaces, allowing more flexibility when it comes to using white spaces as a true troubleshooting and debugging solution.
• Postman variables: This enables awareness of Postman variables {{this}}), and addressing of potential parsing conflicts, while also allowing the usage of nested variable names in URLs.
• IPv6 support: We have begun laying the foundation for wider support of IPv6 as part of the overall URL troubleshooting and debugging.
• IP shorthand: This gives you more flexibility when it comes to allowing IP addresses to be used as part of the URL structure, expanding how the addressing for APIs can work.
• Double slashes: You now have support for \\ in the path for Windows compatibility, ensuring seamless functionality across different platforms, and tools and services that operate on them.
• Newlines and whitespace: You’ll get better handling of newlines and whitespace characters, sticking with Postman’s philosophy of making sure we help developers troubleshoot APIs. See the full blog post about this topic.
• Configurability: With the new approach to parsing URLs, we can now make specific parsing and encoding patterns and algorithms an on/off switch within Postman settings.

Postman’s new approach to parsing and encoding URLs has taken into consideration the widest possible number of scenarios we’ve encountered over the years, while also making sure we are still aligned with the latest web standards. With this significant update, we’re uncertain of the impact the recent changes will have on legacy applications and integrations, which is why we’ve made the URL parser as an optional switch you can turn on or off within your Postman settings. More usage, feedback, and work will be required before we fully understand what additional configuration settings will be needed to further fine-tune how and what the new processor does when working with URLs out in the wild. That said, the foundation has definitely been laid for a more flexible and powerful approach to working with URLs, which are the core element of every API request being made within Postman.

Conclusion

This next-generation URL processor is designed to address a whole spectrum of challenges API developers face. It will provide more flexibility regarding the types of URLs used, while staying true to Postman’s commitment to providing developers with superior visibility into what happens behind the scenes with each API request for debugging and troubleshooting purposes.

The best way to see the new URL processor in action is to turn on the feature in your Postman app settings. You can also browse the multiple GitHub issues that were resolved with this release of the URL processor, so you can see how Postman community feedback drove this (and every) improvement in Postman.

And finally, are you making your voice heard? If you have a bug or an idea, we’d always like to hear your feedback on our GitHub issue tracker.

GitHub issues resolved with this release of the Postman URL processor:

• Variables with reserved characters are not resolved – #5779, #7316
• Nested variables are not resolved – #7965
• Line breaks are not handled properly – #4267
• Trailing whitespaces gets trimmed – #6097
• Encoding issues in path – #3469 #7432 #6126 #7155 #7300 #7123
• Encoding issues in query params – #4555 #4171 #5752