DoorDash: Managing On-Time Meal Delivery with StarTree ThirdEye

Summary

With real-time anomaly detection from StarTree ThirdEye, DoorDash can:

• Provide 500 partners with customized alerts about order status, delivery times and other critical logistical information
• Quickly identify the root cause of anomalies
• Deliver notifications to partners in <5 minutes
• Conduct <1 second searches across massive fulfillment and delivery performance data sets
• Analyze performance across multiple business dimensions 
• Create new dimensions on the fly
• Generate alerts across more than 500 dimensions at sub-100 millisecond latency.

To DoorDash customers, the face of the company is the person who drops off a meal or groceries at their door. But there’s much more to the U.S.’s leading delivery service than a fleet of drivers.

The $9 billion delivery giant serves over 550,000 restaurants and 100,000 non-restaurant stores in more than 6,000 cities across North America. Its 37 million active customers place over 2 billion incoming orders annually. In addition to its consumer service, DoorDash operates a white-label fulfillment offering called Drive. The service allows merchants to access the company’s extensive fleet of over one million delivery professionals without the hassle of building a logistics network. Merchants can request a driver at any time, track orders, streamline delivery costs, and drive incremental sales.  They can also seamlessly integrate DoorDash’s digital ordering and payment solutions into their physical and online channels.

Application programming interfaces (APIs) make all this possible. These APIs, accessible through a developer portal, allow partners to seamlessly integrate DoorDash services into their systems and monitor performance in real-time. About 500 partners use DoorDash APIs to track order fulfillment and delivery performance and even monitor marketing campaigns.

These businesses are keen to know when anomalies occur such as variations in website traffic, fulfillment performance, customer order activity, missed deliveries and other business-critical issues. Those can indicate problems with their website and e-commerce operations that rob them of revenue.

Moving from manual to automated anomaly detection

Previously, DoorDash notified partners manually when anomalies occurred, a costly and inefficient approach that couldn’t scale and couldn’t be effectively acted upon. Timely alerts are important because businesses want to know when website issues may affect order processing or when spikes in traffic indicate that a recent marketing campaign touched a nerve with customers.

DoorDash wanted to identify issues immediately and reduce the time needed to notify partners to less than five minutes when they occurred, said Will Gan, a DoorDash software engineer, in a recent presentation at the Real-Time Analytics Summit 2024. However, achieving that goal involved some major challenges.

One is that data streams must be monitored in real-time with anomaly detection on the fly. Conventional relational databases are designed for batch processing, where data is written, stored, and queried in batches rather than continuously. The SQL queries used to access relational data scan large datasets or complex table joins in deriving insights, which is a lengthy process.

Relational databases also aren’t built to handle streaming data natively. That makes them ill-suited for anomaly detection, which involves identifying patterns or outliers in continuous incoming data streams. In DoorDash’s case, data cardinality — the number of unique values in a data set — was a problem because hundreds of partners each had different issues to track.

Apache Pinot + StarTree ThirdEye: Detecting anomalies in real-time

For a solution DoorDash engineers turned to StarTree ThirdEye, a real-time anomaly detection application built on StarTree Cloud, a real-time analytics platform powered by Apache Pinot. ThirdEye automatically spots outliers in time-series data using multiple detection techniques, including statistical methods, machine learning models, and customizable rules.

When an anomaly is detected, ThirdEye allows users to investigate the root cause by providing insights into which dimensions or factors contributed to the anomaly based on real-time statistical trend analysis. That avoids the problem caused by simple threshold-based alerts, which generate alarms whenever certain metrics fall above or below manually-set static bounds. False alarms can cause engineers to respond slowly to events or even ignore them, a problem called “alert fatigue.”

ThirdEye continuously ingests data streams in real-time, allowing businesses to identify issues as they occur quickly. It uses Pinot’s inverted index data structure to optimize query performance on columns of data with high cardinality. The inverted index enables fast full-text search by mapping content, such as words or terms, to locations within a dataset for efficient retrieval based on those terms.

The inverted index addressed an important pain point for DoorDash. “Essentially, we wanted to alert each partner independently,” Gan said. “The table contains all the data, but when a query is run, it only filters on a single partner.”

Another useful ThirdEye feature is multidimensional anomaly detection. That analyzes data across dimensions such as time, geography, user segments and product categories. For example, it can pinpoint a sudden drop in sales specific to a particular region or customer rather than present it as a general trend. That helps with root cause analysis, identifying the core issues causing abnormal behavior. It also cut down on false alerts.

DoorDash leverages customized anomaly notifications for partners

“Instead of creating 500 alerts for all our partners, we simply have one alert with each developer ID or partner as the dimension,” Gan said. “It essentially creates 500 smaller alerts.”

ThirdEye Dimension Recommendations use historical data to predict trends and recommend key dimension contributors. Engineers can monitor business metrics in real-time across multiple dimensions using a variety of prepackaged algorithms and can also write their own.

DoorDash also took advantage of ThirdEye’s derived column capability. That creates a new column by applying transformations or calculations of existing columns in a dataset, enabling users to generate new metrics or attributes not present in the raw data.

DoorDash engineers created derived columns based on its five-minute response time goal and used staggered cron expressions to spread the processing load over a longer timeframe. The result is that each partner gets a different view and a notification system customized to its needs with minimal latency. At the same time, DoorDash can view all anomalies across its partner landscape to identify broader issues.

The results were quick and dramatic.  API partner satisfaction, as measured by the open rate of notifications, improved by 60%.  The company met its five-minute granularity target and succeeded in generating alerts on a dozen key metrics across more than 500 dimensions at less than 100 millisecond latency.

“This enables us to scale to over 500 partners and have anomaly detection pipelines running for each of them,” Gan said. And thanks to Pinot, ThirdEye can scale almost infinitely. That’s important for DoorDash, whose 20% annual growth rate shows no signs of slowing.

Get started today

You can get a free developer account on StarTree Cloud. There’s no trial limit to it. Use it for free, forever. When you’re ready to move into production, you can move into one of our cost-effective packages just right for your business.