At Meesho, real-time analytics provides the operational backbone for incident management and product analysis. As their platform scaled, their architectural requirements shifted from monitoring high-level metric aggregates to requiring granular, raw-data exploration.
This post details Meesho’s journey from a legacy Flink/Druid stack, the evaluation of modern OLAP alternatives, and the architectural implementation of Apache Pinot combined with a custom federation layer to achieve sub-second latencies at scale.
The Legacy Bottleneck: When Aggregates Aren’t Enough
Meesho’s initial Near Real-Time (NRT) platform relied on Apache Druid powered by Apache Flink ingestion. This architecture was optimized for pre-aggregated data (minute-level dimensional aggregates).
This approach worked well initially and offerred:
- SQL Join Support: Druid offered early support for flexible querying over aggregated data.
- Caching Efficiency: Per-segment and whole-query caching were highly effective for low-cardinality, pre-aggregated datasets.
The Challenge
As Meesho moved toward user journey analysis and funnel Analysis, they hit a hard ceiling. Pre-aggregates obscure the “why” behind metric shifts. They needed to slice raw data across app versions, regions, and devices to pinpoint root causes.
Attempting to force raw-data workloads onto an aggregate-optimized setup resulted in:
- Join Bottlenecks: Performance degraded significantly when joining high-cardinality datasets.
- Scan Costs: Without specialized indexes for raw data, scan costs skyrocketed.
- Limited Upsert Capabilities: Managing mutable state (like user profiles) was inefficient.
The Selection Process: Optimization for P90 Latency
Meesho established P90 latency under 2 seconds as the new SLAs for a next-generation datastore. It needed to be able to reach this across massive event volumes, with query windows ranging from minutes to months.
Meesho benchmarked the following contenders against our workloads:
- StarRocks
- ClickHouse
- Apache Pinot
Apache Pinot emerged as the winner based on performance, resilience, and its ability to handle high-concurrency low-latency queries.
Why Apache Pinot?
Pinot solved specific architectural pain points that general-purpose OLAP stores often struggle with at our scale:
1. Advanced Indexing Strategies
Unlike systems that rely heavily on full table scans or simple partition pruning, Pinot offers a rich set of indexes (such as Inverted, Sorted, and Range indexes). This allows the query engine to jump directly to relevant documents, minimizing data scans and drastically improving CPU efficiency.
2. Intelligent Data Co-location
Distributed joins are notoriously expensive due to network shuffle. Pinot supports Data Co-location, allowing us to strategically shard and place related tables (e.g., an OBT table and auxiliary dimension tables) on the same nodes. This localizes the join computation, eliminating network overhead for specific query patterns like cohort analysis or identity merging.
3. Native Upsert Support
For real-time user profiling, data immutability is a constraint. Pinot provides robust upsert support, enabling them to handle partial updates to records. This is essential for keeping dimension tables accurate in real-time without rewriting entire segments.
The Data Model: One Big Table (OBT) vs. Star Schema
Meesho evaluated a normalized schema (separate tables per event) versus a denormalized One Big Table (OBT) approach.
They chose OBT for the storage layer.
- Columnar Compression: In a columnar store like Pinot, repetitive fields in a denormalized table compress extremely well, mitigating storage bloat.
- Query Velocity: OBT eliminates the need for complex joins at query time, offering predictable low-latency performance.
- Operational Simplicity: Managing a single unified schema proved easier than maintaining consistency across fragmented event tables.
The Architecture: The “Prism” Federation Layer
To prevent vendor lock-in and manage the “Hot” vs. Cold” data lifecycle, Meesho built an abstraction layer called the Prism Federator. This query engine sits between the client and the storage layers (Pinot for Real-Time, Data Lake/Trino for historical).
Key Engineering Features of Prism:
- Smart Routing: The Federator parses queries and routes them based on time windows or attributes. Recent data hits Pinot; historical queries are offloaded to the Data Lake/Trino.
- Request Coalescing (Single-Flight): To handle high concurrency, Prism detects identical concurrent requests and collapses them into a single downstream query to prevent thundering herds on the backend.
- Rule-Based Caching: Caches high-frequency queries to bypass the storage layer entirely.
Results and Wins
The re-architecture has operationalized real-time data for Meesho’s engineering and product teams:
- Incident Management: They can now trigger alerts on raw data anomalies (e.g., order failures per specific app version) in near real-time.
- Granular RCA: Engineers can drill down into user journeys without waiting for batch jobs, validating feature impact instantly.
- Scalability: The system handles massive event throughput cost-efficiently,
Learn More
Meesho’s was Pinot implementation is supported by StarTree. To learn more about how you can get the most from Apache Pinot, Book a Demo or Request a Trial today.
