TEST — Real-Time Date Insights at WIX

Making Kafka Queryable with Apache Pinot

Tim Berglund, VP of DevRel @ StarTree

Apache Kafka has become the standard infrastructure for event-driven and streaming data systems. The stunningly simple abstraction of the distributed log provides exactly what modern microservices and real-time systems need, but no choice is without its tradeoffs. Logs are an excellent way to keep track of events, but they are notoriously difficult to query. Given a constellation of services exchanging events with each other and reacting to inputs in real time, how can you find out—and gain insight into—what has just happened? How, in other words, do you query a log? This is where Apache Pinot comes in.Developed at LinkedIn alongside Kafka, Pinot is a distributed, real-time analytics database designed to ingest data from Kafka (and other sources) and make it instantly queryable at low latency in the face of a huge number of concurrent requests. All that data tucked neatly away into topics, maintaining an immutable record of how the state of the system has evolved, can now be ingested into Pinot and made accessible through simple SQL queries.This talk explores Pinot’s internal architecture, how its integration with Kafka is specially optimized, and how Pinot fits architecturally in the modern streaming stack. You’ll leave understanding how Pinot works, how it fits together with Kafka, where it has been used successfully in the real world, and what steps to take next in your own Pinot learning journey.

Boosting Wix analytics performance with Pinot

Daniil Dubin, Data Engineering @ WIX

Oleksandr Kylinskyi, Engineering Manager @ WIX

In order to provide Wix’s 250M+ users with insights into their business we developed Wix Analytics – the product with dashboards and reports on multiple subjects.

Originally built on top of Snowflake warehouse and Looker it became an immediate success. Unfortunately as the popularity grew, higher RPM resulted in degrading performance and high costs. We analyzed the workload and realized that for the relatively simple high RPM low latency queries Snowflake + Looker combination was not the best fit.

We went on a journey to discover a more performant and cost effective solution ending up with a complex pipeline based on kafka SQL for data enrichment and pre-aggregation and Pinot as the query engine. The result over-exceeded our expectations. Today we are serving 100K+ RPM with sub second latency.

During the talk we’ll go over the problem, decision making process, evaluation of alternatives and highlight some pitfalls that may save you some time should you decide to build a similar solution.

Proactive Anomaly Detection for BI Events using Apache Pinot

Vladyslav Shamaida Senior Software Engineer @ WIX

In Wix, we collect a vast amount of business intelligence (BI) events that represent information about the usage of our products. With dozens of teams, hundreds of artifacts, and billions of BI events daily, it’s crucial to have a real-time anomaly detection system to ensure prompt issue resolution and maintain the health of our BI data.

As our Data Infrastructure team had already achieved great results by integrating Apache Pinot into our user-facing analytics system, the solution lay on the surface. So we decided to build an anomaly detection system on top of it. It’s a real-time datastore and a performant query engine – exactly what we need to run frequent checks over a big number of time-series data.
Furthermore, we explored ThirdEye, a managed solution that seamlessly integrates with Apache Pinot to provide advanced anomaly detection capabilities.

In this talk, we will share our experience, the technical details, and the benefits new tools have brought to our organization in ensuring the reliability and integrity of our mission-critical BI data.