Founded in 2011, Postmates has become well-known for its mobile and web apps that enable consumers to order on-demand from local businesses. With the Postmates API, any merchant can integrate last-mile delivery into their business. Postmates is also the strategic partner for large national retailers like Walmart, Walgreens and Chipotle that run their deliveries on Postmates’ Urban Logistics platform. With “Serve”, an autonomous robot, Postmates is expanding into delivery with autonomous vehicles.
The driver behind Postmates’ growth is data, with an intelligent layer embedded into all Postmates products and operations. With hundreds of thousands of deliveries every day, the only scalable way to achieve growth for a market place is automation. And data drives much of that automation. It’s how Postmates has become the largest, most reliable on-demand delivery and pickup platform in the US and Mexico.
Intermix has become integral for the Postmates data team to keep things running and be responsive to issues that prevent the business and partners from getting the data they need.
Intermix helped us grow. When you’re fast moving, supporting exponential growth, you’re pushing the limits of your data infrastructure. We’ve used Intermix from day one to track new people joining Postmates, what BI dashboards they run, and find and fix the most problematic queries.Aaron Biller
ENGINEERING LEAD FOR DATA, POSTMATES
How a Cloud Warehouse Transformed Postmates’ Digital Operations
A successful, on-time delivery is the single most important event for Postmates’ business.
“In the very, very early days, we would query our production database to understand how many deliveries we had for the past day. Reporting on our metrics would happen with spreadsheets. Clearly that wouldn’t scale, and so we shifted our analytics to a data warehouse and used Amazon Redshift” says Aaron Biller, Engineering Lead for Data at Postmates.
Using a cloud warehouse has been transformative for Postmates’ business. Next to the main Postmates app, the data team has built integrations with 3rd party services. Examples include Facebook for ad campaigns, Stripe for payments or Zendesk for support. “You can write a query that combines 13 data sources in one single query and just run it and get data. That’s extraordinarily useful and powerful from an analytics and reporting perspective.”
Today, the data team has built an infrastructure that drives four major use cases for data:
- Reporting: Daily reporting on successful deliveries and other KPIs along different dimensions like market, partners, service, etc.
- Operations: Minimize the cost per delivery by optimizing route efficiencies, batching / chaining orders, improving delivery times and reducing wait times for couriers
- Payments: Calculate payouts to Postmates’ fleet of couriers, based on their completed pick-ups, deliveries, wait times, distance traveled and also bonuses and tips.
- Risk / Fraud: Calculate and predict risk factors for credit card fraud and other transactions that involve payments such as growth promotions and funding of Fleet debit cards.
“Amazon Redshift is the place where everything goes. It’s really useful for everybody to have all data in one place.”
Business-critical Data Products and Data Microservices
“Data is ubiquitous at Postmates. It’s much more than reporting – we’re delivering data as a product and support data microservices”, says Biller.
Consider fraud prevention. On-demand platforms like Postmates have a unique exposure to payments and fraud because they have to assess risk in real-time. While the warehouse does not operate in real-time, it ingests and transforms event data from all transactions for downstream consumption by predictive models and real-time services.
The Postmates risk team has engineered an internal risk detection microservice called “Pegasus”. Event data passes through a series of transformations in Redshift and feeds into “business rules”, which take the transformed data as input and produce decisions as output, with live decisions for every individual transaction on the Postmates platform.
Dealing with Large Data with Intermix
As things have progressed at Postmates, they added more developers, more microservices and more data sources. “The amount of data we have, period, and the amount of new data we generate every day has expanded exponentially,” describes Biller the growth at Postmates.
Consider the amount of data collected during “peak delivery time” on Sunday nights, when people order their dinner to eat at home.
“Three years ago, we captured data from a certain number of ongoing deliveries on a Sunday night at peak. We’re now at about 30x the number of deliveries in flight. And we’re also monitoring and tracking so many more events per delivery. In short, we’re doing 30x the deliveries, and a single delivery includes 10x the data, and it just keeps growing,” explains Biller.
With self-service access to the data warehouse by a growing number of people, tables became larger, ETL workflows more complicated and analyst queries longer.
The amount of information in Intermix is fantastic. We find the most heavily used tables and the least used tables. Intermix allows us to fast-track that.Aaron Biller
ENGINEERING LEAD FOR DATA, POSTMATES
“Our biggest challenge has been to stay on top of data changes. People adding columns and shifting around data. Queries require a lot more attention to stay accurate and correct. Intermix helps us with introspection of our infrastructure. We’re able to see what we’re running, who is running which queries and how often. We use intermix for that a lot.”
Biller lays out four use cases for Intermix:
- Track query SLAs and successful completion of ETL jobs. Postmates has a narrow 3-hour window every night to finish all ETL jobs behind reports and data services for downstream usage.
- Identify huge queries, break them down into more modular steps and migrate them into Airflow DAGs so the same huge query doesn’t run a hundred times when dashboards get refreshed across the analyst base.
- Identify the largest and fastest growing tables, and work with their owners to deprecate and move historical data into S3 and use Amazon Redshift Spectrum.
- Baselining tables and queries, and continuous work to improve cluster and query efficiency.
“We have these a-ha moments with Intermix”, says Biller. “We have over 1,000 users who run dashboards, with self-service access to data. As a company, Postmates is too big to keep tabs on every individual user and query. But with intermix, we can do that. We can prioritize work that affects our daily reporting and the cost of our cluster. For example, we’ve found tables that had grown to over 3 TB in size, costing us $10,000 per month. We can work with the users directly to fix that.”
Continuous Improvement with Historical Performance Data.
The Postmates data team has been an early Intermix customer, when the number of dashboard users at Postmates was well below 100 people, far from the more than 1,000 today.
We user Intermix on a daily basis. Any time we have a query running that’s causing issues, space that’s spiking, or any pageable alert – we find ourselves using IntermixCharles Bergmeier
DATA ENGINEER, POSTMATES
Charles Bergmeier is a data engineer at Postmates. “There are a lot of realities we have to deal with as a data team,” says Bergmeier. “Our usage is growing exponentially, but our performance needs to improve as well over time, and yet we don’t want to spend a lot of money on more resources.”
Especially with one large event table that keeps growing over time, not all data in the table and the queries that touch the table are useful anymore. A lot of maintenance work needs to happen to keep queries efficient and cluster size and cost under control. With new tables and queries showing up every day, the team uses intermix to investigate table-level metrics, understand what SORT and DIST keys to apply, or which tables would benefit from a new SORT key.
“For us, the killer Intermix feature is history. In the various consoles, you only get a temporary snapshot. With Intermix, I can drill all the way back to earlier in the year, and I get the full history of our metrics. I can see how our disk utilization goes up, how query speeds have improved, and I can see the clean ups we’ve done in the past week.”
With collaboration features, along with the ability to bring geographically dispersed teams into a common workspace, Intermix resulted in increased productivity for the data team. “Our downstream consumers tell us they understand the impact of their queries much better,” says Biller. “Although they are spread across the US, Intermix makes it like you’re in the same room.”