Walmart and Razorpay Power Real-Time Analytics with Apache Pinot

From the world’s largest retailer to a fast-growing payments startup, data-intensive enterprises are leveraging Apache Pinot for their most demanding analytic needs.

Extracting Insights from Massive Datasets

Walmart is a global business with a large and well-developed data analytics program. As its digital business has grown, Walmart has focused on ensuring the integrity of the customer ordering experience: the company wants to leverage data to track every order on walmart.com for fulfillment accuracy and follow-through, but until switching to Apache Pinot, that was a major challenge.

With its previous analytics system, Walmart’s order and inventory management visibility was limited, and their anomaly detection and issue resolution capabilities were reactive rather than proactive. Terabytes of data every week challenged their ingest capabilities, and the data itself contained so many dimensions that aggregate views were difficult to obtain. All of which impacted the customer experience.

Pinot Changes the Game for Walmart

Faced with these roadblocks, Walmart sought to transform their analytic capabilities by adopting Apache Pinot — and the results have been dramatic. Pinot’s low query latency (10s of ms) across Walmart’s massive datasets gives teams real-time analytics and deep visibility into order flow, which translates into fast remediation of issues in the order lifecycle. Huge reductions in MTTD and MTTR, often by hours, have helped Walmart deliver an exceptional customer experience on their website.

The Pinot system ingests 14M events/min from Kafka, with less than 900ms of lag before data is ready to query, ensuring unprecedented data freshness. Smart indexing from a broad array of indexes, predicate pushdown, full and partial upsert features, and multi-tenancy support round out the Pinot capabilities that Walmart cites as game-changers for their operations.

Today, every order on walmart.com goes through Pinot. With near real-time ingestion, Walmart gets accurate information on orders and inventories across a huge diversity of dimensions, and they can quickly pinpoint and correct anomalies. Pinot functions as a single, go-to dashboard for everyone, from leadership to engineering to support teams.

With Pinot, Walmart enjoys granular, real-time analytics across its entire inventory of products

Razorpay Seeks Transaction Tracing at Scale

As one of the largest payment platforms in India, Razorpay handles 60B transactions per year, or 7M per day. To track all of those payment requests, Razorpay relies on modern distributed tracing tools that combine 100+ microservices and use a waterfall model to illustrate the flow of each and every request.

To operate at scale, such tracing systems require an analytics engine that can deliver near real-time data ingestion and sub-second query latency on large data volumes. The database must be able to slice and dice data with high cardinality, and it must provide high availability so that the enterprise can maintain access even when things go wrong.

Apache Pinot Proves to be the Optimal Solution

In seeking an analytics engine that met their needs, Razorpay evaluated Elasticsearch, but found that, for their use cases, query latency rose by up to 10x as data volumes grew. They looked at Apache Druid, but the lack of a real-time edition and native upsert capabilities eliminated that option. Razorpay also considered Presto — which powers their data lake — but its response times were so high as to be unworkable in a real-time context.

Apache Pinot, on the other hand, with high throughput on data ingest and fast queries even on large data volumes, met Razorpay’s needs. Today Razorpay ingests around 300K events/sec, with spikes up to 1M events/sec, for a total data ingest of multiple TBs per day. Even so, trace freshness remains less than 1 second. Pinot works well with high cardinality fields, delivers high availability, and — importantly for Razorpay — is Kubernetes-native. Pinot’s broad variety of indexes, and its ability to automatically prune queries to eliminate unnecessary steps, has further accelerated Razorpay’s query performance.

Ultimately, Razorpay found that Apache Pinot not only delivered unprecedented power for their trace program, but could also handle most of their other OLAP use cases as well. This enabled Razorpay to streamline operations by consolidating their analytics stack and avoiding the need for multiple management platforms.

Razorpay easily powers their real-time analytics stack with Apache Pinot