• Uniqode serves QR code analytics to 30K customers with Apache Pinot

  • Uniqode found Apache Pinot p95 latencies were 20x faster than Elasticsearch

  • Uniqode saw 10x improvement in overall query performance across latencies

  • Uniqode saw 100% reduction in errors with Pinot and StarTree

  • Uniqode shrank their cluster footprint (50% reduction in nodes) by moving from Elasticsearch to Apache Pinot

    Uniqode relies on Apache Pinot and StarTree for user-facing analytics

    Quick Response (QR) codes have been around since the 1990s and had explosive growth across industries thanks to the broad adoption of mobile browsers and camera-equipped smartphones. In the U.S. alone, the number of users that will scan a QR code to seek information on products and services continues to rise, and is now nearing 100 million consumers

    Uniqode (formerly Beaconstac) is a provider of a dynamic QR code solution to run campaigns and analytics, engaging audiences and prospects immediately while building customer trust and loyalty. Real-time data is essential for the 30,000+ customers of Uniqode who want instantaneous insights, discovering patterns and trends in when and how users are scanning their QR codes.

    Watch now

    This article is based on an in-person meetup. Watch the meetup here:

    How Uniqode serves real-time QR code analytics to users

    With more than 204 million QR code scans worldwide, Uniqode serves diverse use cases across retail, financial services, consumer goods, healthcare, and other industries. Uniqode's dashboards allow users to see their top-performing QR codes and analyze code scans by filters such as location, time of day, and device used. 

    The company’s real-time data ingestion pipeline includes Uniqode's analytics server, Amazon Kinesis Data Streams, and Amazon Lambda. That data is then sent to StarTree Cloud, powered by Apache Pinot, and delivered to users in the form of analytics dashboards.

    Beaconstac real-time data streaming architecture

    Challenges of using Elasticsearch

    Initially Uniqode had used Elasticsearch, the popular search engine, to generate their analytics. However, Elasticsearch was never designed to support user-facing analytics at scale. As Uniqode's user base and cluster footprint grew rapidly, they experienced a multitude of issues with Elasticsearch. User-facing dashboards had loading challenges due to high latencies, high error rates, and slow Queries Per Second (QPS). All of these issues directly impacted end users and affected the likelihood of users converting to Uniqode's paid plans.

    The key issues Uniqode identified with Elasticsearch were:

    • High p95 latencies >20 seconds

    • High error rate

    • 2.5× - 3×​​ lower QPS than expected

    • Large cluster footprint (grew by 2× every few months)

    The Uniqode team saw p95 latency for queries run using Elasticsearch often exceeding 20 seconds, as shown in the graph below.

    Beaconstac latency performance with Elasticsearch

    These high latencies were coupled with frequent query timeouts, causing Uniqode's users to wait a considerable amount of time to view their QR code analytics. At peak times, the number of errors for queries run in Elasticsearch could be in the thousands — with one peak surpassing 15,000 errors in a 2 hour period, or two errors every second.

    Beaconstac error rate with Elasticsearch

    Although Uniqode had built-in retry mechanisms for errors, these callbacks led to a very low QPS on their Elasticsearch clusters. The team should have been able to serve 100+ QPS, but were only serving 30 to 40 QPS due to high latencies and error retry rates.

    Comparing performance of Elasticsearch vs. Apache Pinot

    After realizing they needed a long-term replacement solution, Uniqode determined their best option would be a real-time analytics database. This was the case for Apache Pinot; it was designed-for-purpose for fast online analytical processing (OLAP) workloads. Uniqode did a small POC of Apache Pinot in-house and were impressed with its capabilities.

    The team performed extensive testing to ensure Pinot would eliminate the problems they were experiencing with Elasticsearch. Stress testing on the Pinot cluster found they were able to support 100 concurrent users and 200 QPS with p95 latencies as low as 500 ms.

    Why Uniqode chose fully-managed Pinot with StarTree Cloud

    One of the issues Uniqode had faced with Elasticsearch was its steep learning curve. Their small data team lacked specialized knowledge of Elasticsearch, and they had difficulty finding resources with that experience. Apache Pinot had a vibrant community and excellent documentation. Uniqode also saw the benefits of working with StarTree to access fully managed Pinot, help from Pinot experts, and additional features not available in open source Pinot (such as Data Manager for easy data ingestion).

    Of course, performance was a key reason that Uniqode selected Pinot and StarTree Cloud. Even without using explicit indexing, the Beconstac team saw improved run times compared to Elasticsearch.

    10x faster queries with Apache Pinot and StarTree

    With StarTree Cloud, powered by Apache Pinot, the Uniqode team saw latencies between 0.5 and 1 second and p95 latency improved by more than 20x. They also experienced zero errors — a drastic improvement compared to the error rates of Elasticsearch.

    The performance gains stem from the architectural differences and advantages of Apache Pinot vs. Elasticsearch. Apache Pinot was designed-for-purpose for real-time analytics. It uses an efficient columnar store format, supports fast aggregations and flexible index types like the star-tree index, and for ingestion of streaming data and real-time upserts.

    Beaconstac latency with Apache Pinot

    When Uniqode compared query performance among Elasticsearch, Apache Pinot, and StarTree, they saw a 10x improvement overall across latencies (including p50, p75, p90, p95, and p99). The chart below shows query runtimes for Elasticsearch, open source Apache Pinot (without indexing), and StarTree Cloud (with indexing).

    Beaconstac query performance with Elasticsearch vs Apache Pinot vs StarTree Cloud

    Reduced cluster footprint

    In addition to query performance improvements, Uniqode was able to significantly reduce their cluster footprint. The results from switching to Apache Pinot included:

    • 50% reduction in number of nodes needed

    • 10x reduction in number of cores being run

    • 7x reduction in memory

    Beaconstac cluster footprint improvements

    Apache Pinot = Lower latencies, reduced infrastructure, and zero errors

    On the whole, Uniqode saw: 

    • 90% reduction in infrastructure size

    • 90% reduction in p99 latency

    • 100% reduction in errors

    They were also able to significantly improve their QPS, with Pinot giving them the ability to run many more concurrent queries than Elasticsearch.

    Beaconstac performance improvements with Apache Pinot and StarTree Cloud

    See why companies are migrating from Elasticsearch to Pinot

    Uniqode isn’t the only company who saw drastic improvements in their real-time analytics performance by switching to Apache Pinot. Uber replaced Elasticsearch with Pinot and saved $2M annually, reduced database cores by 50%, and saw page load times become nearly 3x faster. 

    Cisco Webex also migrated their real-time analytics and observability to Pinot after facing high latencies. The Webex team found that Apache Pinot provided 5x to 150x lower latencies than Elasticsearch.

    Watch the full meetup to hear the stories from Uniqode, Uber, and Webex.

    Learn more about Apache Pinot

    Check out our comprehensive list of stories from Apache Pinot users to learn about how companies are using Pinot for user-facing analytics, anomaly detection, observability, security, and more. 

    You can also get access to fully managed Apache Pinot by signing up for a free trial of StarTree Cloud. If you’re interested in seeing how StarTree compares to Elasticsearch, visit our StarTree Cloud vs. Elasticsearch comparison page.

    User Focus

    Insights from existing users running real-time analytics at scale

    Sovrn Moves from 24-Hour Delay to Real-Time Data with StarTree Cloud

    Read More

    Webex Meets Unprecedented Analytics Demand with Pinot

    Read more

    Walmart and Razorpay Power Real-Time Analytics with Apache Pinot

    Read more