How (and Why) Postman Created a Data-Driven Hiring Process

An effective hiring process is one of the most crucial parts of scaling an organization. Especially when a company scales at least 10 times its size, it needs to figure out how to evolve various processes and practices while still staying true to its values. Postman has always been committed to hiring the highest-quality talent, but we didn’t always do it in the most efficient way. This blog post will show how we have addressed the hiring scaling problem with analytics and systems thinking.

At the end of the day, it all comes down to solving a problem with constraints—process, people, and tools. By implementing a hiring maturity model and using actionable metrics, we’ve been able to have a predictable timeline from interview to hire. This has helped Postman’s entire talent acquisition team plan and manage its pipelines proactively—ultimately fast-tracking the hiring process to secure desired candidates as our organization rapidly grows. Here’s how we’ve optimized our hiring process.

The starting ingredients

In order to create an efficient scaling policy, we first needed observability into what was happening in the now. Leveraging our in-house data team’s commitment to data democratization, we decided to bring in two different perspectives:

• A view from external observers about how they would perceive the problem statement
• Tools and expertise to mine data once given the problem statement

This was the beginning of an experimental journey to make our hiring funnel better with a joint partnership within the company.

Current hiring funnel

In the fall of 2020, we created a mind-map of our hiring funnel showing the stages in our internal application tracking system (ATS). Based on that, our current funnel looks something like this:

Although the granular aspect of the above funnel gives a strong sense of what is working and what is not, we wanted to also add an abstraction on top to give meaning to each stage. We ended up modeling our hiring funnel after this approach:

This abstraction helps us better understand the overall journey of a candidate when they interview with us. At the granular level of the funnel, we do have a well-defined set of criteria for each stage in order to understand what we are trying to evaluate clearly. This practice also helps make the hiring journey consistent for all candidates.

Defining success

At Postman, we begin with a thorough discussion around a problem statement (not just the solution) in order to determine whether we are solving the right problem and addressing it for the right reasons. Understanding the “why” of the problem statement from the start helps us define the prioritization and the expected value (once the problem gets solved)—which means greater efficiency.

For the problem statement around scaling hiring, we came up with the following questions before going ahead:

• What if we don’t solve this problem now and focus on something else?
  • The answer to this was pretty obvious: We were solving this for the entire organization because hiring has an exponential effect. For example, the process used by 20 recruitment team members conducting more than 100 interviews can potentially lead to 200 new hires whose work can impact more than 14 million Postman users.
• What does success look like if we solve this problem?
  • This question deserves a whole section for itself. Let’s take a deep dive into this a bit more.

What does success look like?

Devising a process focused on goals, signals, and metrics (as derived from this Success Metrics & Signals methodology), we came up with the following:

Goals

• Hire fast while still upholding the standards that retain the highest-quality candidates
• Provide an amazing interview experience to each candidate

Signals

The following observable results are signs of an effective process:

• Postman teams are able to scale up faster by hiring faster
• Recruitment teams actively focus on candidate experience; candidates are provided with sufficient details about the hiring process and ask less questions because they are well-informed
• Positive Glassdoor reviews

Metrics

The following metrics are treated as the proxies for the signals mentioned above:

• Time to hire (i.e., the time it takes to secure a candidate)
• Time per interview stage per candidate
• Responsiveness to questions by candidates
• Number of questions by candidates to recruiters
• Number of positive/negative experiences recorded on Glassdoor

How we created an effective, efficient hiring process

Based on our experience, we have realized that there are two main types of metrics for this process:

• Impact-driven metrics: These metrics translate the goals into something measurable but are generally slower to obtain. (The metrics listed above are mostly impact-driven metrics.)
• Process-driven metrics: These metrics denote the efficiency of a given process. With a pre-defined process, these metrics provide immediate feedback on what needs to be done. In essence, these metrics are immediately actionable and help move the impact-driven metrics in the long-term.

This realization led us to devise certain process-driven metrics (in addition to the above impact-driven metrics) to help us determine when we are moving in the right direction or not. We now distill these metrics down to the following two scores:

• Process performance score (PPS)
• Candidate experience score (CES)

Next, we’ll explain how we defined these scores and how we made them actionable with near-real-time feedback.

Process performance score

A process performance score lets us determine how fast we are moving candidates through our hiring pipeline. We measure this score in a range of 0-100%, where 100% is the ideal score. Let’s discuss how we measure this.

The two key characteristics of this score are that it is immediately actionable and gives near–real-time feedback (i.e., the score changes as soon as someone acts on it).

In order to make this metric actionable, we added a few constraints to the hiring pipeline to ensure that candidates move actively through the pipeline. An interesting example would be a simple hiring pipeline: A candidate applies, interviews, and then is offered a job. In this pipeline, we added one constraint as a service-level objective (SLO) per stage as:

Note: The above is a simplified example of our SLO table. We complement this table with an additional value ranking the priority of hiring a specific candidate. Priority calculation is specific to each organization. For Postman, it is mainly based on the relevance of a candidate to a given job description.

As soon as the above threshold is breached—for example, a candidate stays in the Applied stage for more than one day—we consider it to be a “violation.” Every violation penalizes the score with a certain value (more on this later).

Calculating a process performance score

As an example, the overall formula for a Process Performance Score that has two violations looks like this:

Process performance score (in %) = 100 – (violation 1 impact) – (violation 2 impact)

The effect of the SLO violation on the score can be amplified by the following:

• Priority of the candidate
• Time: The older the violation is, the more effect it has on a score

Actionability

Every violation is listed in our tracking dashboards with a dedicated recruiter for each hiring pipeline. The action item becomes fairly obvious (i.e., moving the candidate to the next stage could require scheduling an interview, coordinating the internal team, and resolving any existing blockers). As you can see, the action itself can vary but the intent is clear: We want to move the candidate to the next stage.

Feedback loop

Our engineering systems sync PPS data from our application tracking system (ATS) to our data warehouse (further accessible as reports) every hour. This ensures that once we’ve taken an action, the score reflects the same within two hours at the most—giving us these small-wins to achieve a bigger goal.

Candidate experience score

Candidate experience score refers to our team’s service and responsiveness to the candidates within the hiring funnel. From a candidate’s standpoint, we wanted to ensure the following:

• The interview process should be consistent for all the candidates
  • A candidate should have enough information about the interviews in advance so that they can prepare accordingly
  • A candidate should receive the latest updates on their application within the hiring funnel
• When a candidate asks a question, we should communicate and respond within a given amount of time

How we made Postman job interviews more consistent

We consider an interview to be a collaborative discussion between the candidate and the interviewer. This philosophy builds on the fact that interviewers are not supposed to judge the candidate based solely on the candidate’s skills; instead, the interviewers should collaborate and observe the candidate’s thought process and ability to ask questions.

This philosophy drove us to create a deterministic framework for our interview process in order to provide a consistent experience to all the candidates. In a nutshell, this framework focuses on the traits and values that we look for in candidates (rather than the skills), and assigns evaluations of those traits and values across various interview rounds.

How we inform candidates at every stage of the hiring funnel

Building further on the philosophy mentioned above, the success of a candidate within an interview round depends not only on them, but also on the information we provide them prior to each round, the environment in which the discussion is happening, and many other variables. To decrease the variability across candidates, we started sending out automated emails whenever a candidate moves from one stage to another stage of the hiring funnel.

These emails are written to build a user story for the candidate. We provide details of what to expect during the next interview round, but we also keep the emails concise to avoid information overload. This communication helps set the candidate up for success rather than failure. It also decreases bias on our side previously caused if a candidate was unprepared due to a lack of awareness as to what the interview would entail.

Ensuring continuous communication

Communication is the key to success, not just when you work for an organization but also when you interact with anyone outside of your organization. So, Postman’s hiring team is focused on providing continuous information throughout the interview process and then clearly closing out each interview process once a decision is made.

In order to ensure that this communication is working from a measurable standpoint, we added a few SLOs in our communication channel (i.e., emails) as well. For instance, for each stage of the hiring process we should now respond to a candidate’s queries faster—within 1 business day:

Given the above SLO definition, we follow the same model used for the process performance score: The score is penalized if the SLO is violated, and the recruiter is held accountable for the violation. However, as soon as we respond to the candidate (via email), the violation is removed. While this scoring system has really helped track more consistent communication when hiring, it’s not a full-proof implementation, and so we continue working on refining it as we encounter gaps.

Success

Until now, we’ve talked about how we made our hiring process better. But it all doesn’t matter if we don’t see the impact. As discussed earlier, we focused on a combination of objective and subjective signals to evaluate the impact.

Here are the metrics and the resulting changes we measured for each:

• Time to hire: Decreased by approximately 10%. Since this metric depends on the overall hires, we saw minimal difference because the overall volume of candidates increased.

• Time spent by candidate per stage: Decreased by approximately 15% average across all stages. (We have only looked at the list of candidates who are more relevant to the jobs that they applied for.) This specific metric saw the maximum improvement because of SLOs targeted at the same. We saw a huge decrease in the last few months. Note: We consider this to be a major impact because our hiring funnel scaled up by 4x while we were implementing this mechanism. So while we scaled up, having a crucial metric decrease by 15% is not bad.

• Responsiveness to candidate queries: We now respond to candidates within 24-48 hours for most of the queries. There are still some gaps that we discover via various interactions between recruiters and candidates, and those are where our future improvements lie.

• Number of queries to recruiters by candidates: This was one of the subjective metrics that we need to unearth. Since the queries can come via email or phone, we relied on the subjective evaluation of this metric. After discussing with the recruitment team, we concluded that this metric decreased by approximately 40-50% after we implemented the automated emails.

• Glassdoor reviews: We subjectively kept track of Glassdoor reviews, celebrated small wins, and took immediate actions on the misses (any negative feedback). This thinking led us to treat the system mentioned above as a product and continue improving it.

What’s next?

We continue to learn and modify how we do things as we grow as a company. When it comes to this effective data-driven hiring process, we plan to make improvements by extending the hiring funnel and also by implementing an interviewer experience score for better interviewer onboarding and consistency.

Do you want to experience the Postman hiring process firsthand? Learn about joining the Postman team here.