Webex Meets Unprecedented Analytics Demand with Apache Pinot

Cisco Webex has been a pivotal digital resource during the Covid pandemic — empowering governments, healthcare organizations, educational institutions, and whole industries to work from home — and is now an essential piece in the emergence of a hybrid model of work. With a network of 47 data centers around the world, today Webex hosts 30B+ meetings and 8B+ calls per month for hundreds of millions of global users.

No surprise that in the course of its operations Webex must take in and analyze huge amounts of data. 100,000 meetings per hour generate 1TB per minute of telemetry and metrics data. Add in calls and events, and the total comes to 100TB of data per day. Webex seeks to ingest and analyze all that data in real-time, not just to push it back in real-time to external users, but to give internal users and partners the data to build new and improved experiences for customers.

Adapting to the Pandemic’s New Normal

From the beginning, Webex engineers have had a mandate to enable real-time analytics with the fastest possible performance for ingestion and retrieval of data. Moreover, as they’ve grown, Webex has made a point of always trying to both simplify and consolidate their technology stack and ensure a cost-effective, self-service platform.

When the pandemic hit, Webex teams saw that demand for analytics around meetings and calls quickly jumped. Where they had previously provided customer IT admins with reports once per day or once per week, now those admins were requesting more analytics, faster, and in greater detail.

The data needed to fulfill these customer requests covered 1B+ events per day, analyzed across 150+ dimensions. It was data already provided to internal teams, for which Webex had been using Kafka to upload the data to Elasticsearch and then leverage a combination of pre-aggregates, raw queries, and weekly data roll-ups to serve the data while keeping costs down. But this technique could not handle rapidly increasing scale and real-time customer demand.

Red-Pilling the Challenge of Data at Scale

To meet that demand, Webex considered undertaking even more pre-aggregation, but the cost in compute and storage was simply too high. Instead, they looked to deploy a runtime aggregation solution and evaluated POCs from Clickhouse, ElasticSearch, and Apache Pinot.

For the evaluation, Webex put together comparable vendor systems, gathered one month of media data (roughly 30 billion rows), and began running test queries. While Elasticsearch soon proved to be unsuitable for Webex’s use cases, Pinot was able to establish clear advantages over Clickhouse in the three areas critical to Webex:

• Performance. Unlike Clickhouse, Pinot was able to maintain exceptionally low p99 latencies, even at higher concurrencies. Pinot ran faster in 5 of 6 query types, comparably in 1, and for 3 of 6 query types it actually ran more than 4x faster than Clickhouse. In addition, Pinot’s throughput was 40% higher than that of 
 Clickhouse.

• Ease of Operations. The Webex team found Pinot somewhat more involved to set up, but once deployed, Pinot was natively much easier to use, with simple cluster management, quality of service, and plenty of automation — whereas Clickhouse required far more manual configuration and did not support multi-tenancy or tiered storage.

• Extensibility. Because both Pinot and Clickhouse are relatively new solutions, Webex wanted to ensure extensibility to potential new use cases. Here, too, they preferred Pinot, which is written in Java, rather than Clickhouse, written in the more complex C++.

Throw Everything on Pinot

After going with Pinot, operations personnel saw striking increases in responsiveness, with their dashboards populating in seconds instead of 10s of minutes. That gave them confidence to put more and more of their workloads on Pinot. Today Webex runs Pinot on 100+ production nodes across most of their global data centers. Pinot’s speed and ease of use have enabled Webex to meet the unprecedented demands of the pandemic — and to accelerate their rollout of new customer experiences powered by real-time analytics.