Uniqode: 10x Faster Queries with Apache Pinot and StarTree over Elasticsearch

Summary

How Uniqode serves real-time QR code analytics to users

With more than 204 million QR code scans worldwide, Uniqode serves diverse use cases across retail, financial services, consumer goods, healthcare, and other industries. Uniqode’s dashboards allow users to see their top-performing QR codes and analyze code scans by filters such as location, time of day, and device used.

The company’s real-time data ingestion pipeline includes Uniqode’s analytics server, Amazon Kinesis Data Streams, and Amazon Lambda. That data is then sent to StarTree Cloud, powered by Apache Pinot, and delivered to users in the form of analytics dashboards.

Challenges of using Elasticsearch

Initially Uniqode had used Elasticsearch, the popular search engine, to generate their analytics. However, Elasticsearch was never designed to support user-facing analytics at scale. As Uniqode’s user base and cluster footprint grew rapidly, they experienced a multitude of issues with Elasticsearch. User-facing dashboards had loading challenges due to high latencies, high error rates, and slow Queries Per Second (QPS). All of these issues directly impacted end users and affected the likelihood of users converting to Uniqode’s paid plans.

The key issues Uniqode identified with Elasticsearch were:

• High p95 latencies >20 seconds
• High error rate
• 2.5× – 3×​​ lower QPS than expected
• Large cluster footprint (grew by 2× every few months)

The Uniqode team saw p95 latency for queries run using Elasticsearch often exceeding 20 seconds, as shown in the graph below.

These high latencies were coupled with frequent query timeouts, causing Uniqode’s users to wait a considerable amount of time to view their QR code analytics. At peak times, the number of errors for queries run in Elasticsearch could be in the thousands — with one peak surpassing 15,000 errors in a 2 hour period, or two errors every second.

Although Uniqode had built-in retry mechanisms for errors, these callbacks led to a very low QPS on their Elasticsearch clusters. The team should have been able to serve 100+ QPS, but were only serving 30 to 40 QPS due to high latencies and error retry rates.

Comparing performance of Elasticsearch vs. Apache Pinot

After realizing they needed a long-term replacement solution, Uniqode determined their best option would be a real-time analytics database. This was the case for Apache Pinot; it was designed-for-purpose for fast online analytical processing (OLAP) workloads. Uniqode did a small POC of Apache Pinot in-house and were impressed with its capabilities.

The team performed extensive testing to ensure Pinot would eliminate the problems they were experiencing with Elasticsearch. Stress testing on the Pinot cluster found they were able to support 100 concurrent users and 200 QPS with p95 latencies as low as 500 ms.

Why Uniqode chose fully-managed Pinot with StarTree Cloud

One of the issues Uniqode had faced with Elasticsearch was its steep learning curve. Their small data team lacked specialized knowledge of Elasticsearch, and they had difficulty finding resources with that experience. Apache Pinot had a vibrant community and excellent documentation. Uniqode also saw the benefits of working with StarTree to access fully managed Pinot, help from Pinot experts, and additional features not available in open source Pinot (such a Data Manager for easy data ingestion).

Of course, performance was a key reason that Uniqode selected Pinot and StarTree Cloud. Even without using explicit indexing, the Beconstac team saw improved run times compared to Elasticsearch.

10x faster queries with Apache Pinot and StarTree

With StarTree Cloud, powered by Apache Pinot, the Uniqode team saw latencies between 0.5 and 1 second and p95 latency improved by more than 20x. They also experienced zero errors — a drastic improvement compared to the error rates of Elasticsearch.

The performance gains stem from the architectural differences and advantages of Apache Pinot vs. Elasticsearch. Apache Pinot was designed-for-purpose for real-time analytics. It uses an efficient columnar store format, supports fast aggregations and flexible index types like the star-tree index, and for ingestion of streaming data and real-time upserts.

When Uniqode compared query performance among Elasticsearch, Apache Pinot, and StarTree, they saw a 10x improvement overall across latencies (including p50, p75, p90, p95, and p99). The chart below shows query runtimes for Elasticsearch, open source Apache Pinot (without indexing), and StarTree Cloud (with indexing).

Reduced cluster footprint

In addition to query performance improvements, Uniqode was able to significantly reduce their cluster footprint. The results from switching to Apache Pinot included:

• 50% reduction in number of nodes needed
• 10x reduction in number of cores being run
• 7x reduction in memory

Apache Pinot = Lower latencies, reduced infrastructure, and zero errors

On the whole, Uniqode saw:

• 90% reduction in infrastructure size
• 90% reduction in p99 latency
• 100% reduction in errors

They were also able to significantly improve their QPS, with Pinot giving them the ability to run many more concurrent queries than Elasticsearch.

See why companies are migrating from Elasticsearch to Pinot

Uniqode isn’t the only company who saw drastic improvements in their real-time analytics performance by switching to Apache Pinot. Uber replaced Elasticsearch with Pinot and saved $2M annually, reduced database cores by 50%, and saw page load times become nearly 3x faster.

Cisco Webex also migrated their real-time analytics and observability to Pinot after facing high latencies. The Webex team found that Apache Pinot provided 5x to 150x lower latencies than Elasticsearch.

Watch the full meetup to hear the stories from Uniqode, Uber, and Webex.

Learn more about Apache Pinot

Check out our comprehensive list of stories from Apache Pinot users to learn about how companies are using Pinot for user-facing analytics, anomaly detection, observability, security, and more.

You can also get access to fully managed Apache Pinot by signing up for a free trial of StarTree Cloud. If you’re interested in seeing how StarTree compares to Elasticsearch, visit our StarTree Cloud vs. Elasticsearch comparison page.