“We switched to Apache Pinot from Elastic and reduced costs by 70%, and improved performance 3x”
StarTree helps them succeed.
At StarTree, we work with world class engineering teams who are building some of the most sophisticated data-intensive systems. Many of them have been using Elasticsearch to handle query intensive use cases. But as data under management grows, and requirements expand, they run into difficulties when they try to use Elastic for fast, or high concurrency aggregations and analytics.
That’s when they start looking at purpose-built real-time OLAP systems like Apache Pinot.
Companies like Uber, Cisco WebEx, and Uniqode have all transitioned from Elasticsearch to Apache Pinot. The results are always impressive: faster queries, simpler pipelines, improved reliability, and dramatically lower infrastructure costs.
ElasticSearch works… until it doesn’t
It’s not uncommon for engineering leaders to find themselves hitting a wall while trying to run analytics on data in ElasticSearch at scale. Their issues typically include lagging dashboards, query timeouts, and sticker shock from scaling out clusters to keep up with high-volume data streams.
Elasticsearch was designed for full-text search and log analytics. It’s been a common goto for offloading expensive queries from operational databases. But Elastic’s document-based architecture isn’t well suited for handling the kind of real-time aggregations that many modern workloads demand.
Common problems include:
- Sluggish query at scale A leading problem is performance degrades with large datasets or high cardinality.
- Storage bloat Elasticsearch storage needs can easily balloon to 3-5x the original data size. Data needs to be replicated to enable aggregation. Lucene segments accumulate unless you’re constantly managing them.
- Update and retention headaches Elastic was designed to search text that wasn’t expected to change much. Users of Elasticsearch report expensive reindexing for updates/deletes; limited TTL handling.
- Growing infrastructure costs and operational complexity Users of ElasticSearch often complain they need to significantly over-provision to maintain reliability.
- Poor data freshness with ingestion-to-query lags Elasticsearch users typically experience delays between ingest and index availability.
Elastic still works well for simple aggregations at moderate scale, especially if you need flexible text search plus some analytics. But if you require freshness, fast query performance, high concurrency, you should consider a purpose built OLAP system
With Apache Pinot, Uber can do real-time upserts and get query results from 1.5PB of data with <100ms P99 latencies.
When to use Apache Pinot over ElasticSearch?
Fast query response
Elasticsearch’s inverted index and document based architecture makes aggregating on fields like user ID, session ID, or product SKUs slow at scale.
Apache Pinot’s columnar architecture was designed for this. When log data is stored in columns rather than documents, queries only need to access the specific columns instead of scanning entire documents. This enables aggregate queries that are 5-150x faster than in Elasticsearch.
On top of that, Pinot’s flexible indexing approach provides higher performance and flexibility for working with a wide range of data types.
Uniqode needed to aggregate QR scan activity across regions, time zones, and device types — all high-cardinality dimensions.
Pinot handled these queries 10x faster than Elasticsearch could.
Cisco WebEx used Pinot + Grafana to build real-time observability dashboards for their video conferencing platform. Pinot gave them streaming ingest and near-instant insights into live user experience metrics, something their previous system couldn’t deliver
Data freshness at scale
Elasticsearch microbatches data on ingest which adds latency during indexing and refresh cycles. Data ingested can take seconds or minutes before becoming queryable.
Apache Pinot provides native ingestion of data directly from streaming sources like Kafka, Kinesis and other stream processors – enabling sub-second freshness with exactly-once guarantees. This is especially important where fast anomaly detection is essential: Mobile crash monitoring, security incident response, system observability.
Power high concurrency, interactive dashboards
Elasticsearch can buckle under the pressure of hundreds or thousands of simultaneous analytical queries — especially when powering dashboards or public-facing analytics.
Apache Pinot was built for high QPS and concurrency. It executes queries in a distributed, vectorized fashion and can scale to hundreds of thousands of requests per second with millisecond latency.
BurdaForward moved to Pinot to deliver real-time content performance dashboards to journalists, editors, and product managers. Pinot scales with their usage without latency spikes or service degradation
“Even at massive scale—handling billions of updates per server and processing over 1 billion transactions annually with sub-second latencies—Pinot delivers consistent real-time analytics performance without the complex workarounds required by Elasticsearch.”
Reduce storage costs
Elasticsearch native document storage model suffers from poor compression and leads to storage bloat. Accessing older data in cold storage often means rehydrating entire segments just to answer a single question.
Apache Pinot’s columnar storage model offers dramatic compression and reduced storage requirements. And with Precise fetching on cloud object storage you can store historical data in Pinot, Parquet or Iceberg formats and still query it efficiently. Precise fetching significantly reduces infrastructure costs by avoiding unnecessary data scans and movement.
Simplify operations and lower overhead
With Elasticsearch, teams often deal with reindexing, index template management, and replica configuration — all of which add operational overhead and infrastructure cost.
Apache Pinot eliminates reindexing with it’s schema-on-write model. It supports automatic tiered storage, built-in upserts, and easy schema evolution.
Ken Krugler describes the decisions behind AdBeat‘s transition from Elastic to Pinot.
Rethink Observability on Open Source
Modern observability workloads are increasingly driven by analytics and require more than just log analysis. This guide explains the options for building a future-ready observability platform, including:
- From Closed Platforms to Custom Architectures: See how teams at Netflix and Pinterest are swapping rigid tools for modular stacks.
- Lower Costs without Sacrificing Insight: Learn how Cisco Webex optimized performance while reducing storage costs.
- Own Your Stack, Own Your Future: With OpenTelemetry and Kafka as your building blocks, create a tailored observability platform.
Download a copy free
Additional Resources
A database ideal for real-time analytics – at speed and scale.
Apache Pinot is a real-time OLAP database built for ultra-fast analytics on high-volume data. It supports sub-second queries on billions of rows, even with high cardinality and concurrency — ideal for working with telemetry, logs, streaming business data and more.
StarTree Cloud delivers a fully managed Pinot experience with enterprise-grade security, automated scaling, and seamless integrations (Kafka, Flink, BI tools, etc.). It gives your team the power to build real-time dashboards, alerts, and APIs on live data — without compromising performance or reliability.
If you’re tracking fast-moving data and need millisecond-level insights at scale, Pinot + StarTree can help you move faster and operate smarter.
