Real-Time Analytics Summit: Developers Assembled!

Peter Corless
ByWritten byPeter Corless
May 9, 20238 minutes read

Real-Time Analytics Summit 2023 brought together an impressive collection of industry-leading developers, architects of the world’s fastest streaming analytics databases, and representatives of all the essential real-time data technologies. It was an assembly of industry leaders and innovators, all gathered in one place to advance the state of the art. Here is your on-the-ground reporting straight from the halls of the Hotel Nikko, San Francisco, where the event was held recently, April 25th and 26th, 2023.

Kishore’s Keynote: The Rise of Real-Time Analytics

StarTree Cloud's Kishore Gopalakrishna At RTA Summit 2023

The event kicked off with a keynote from StarTree founder and CEO Kishore Gopalakrishna. He noted three critical dimensions that are required to move from historical Online Analytical Processing (OLAP) systems, such as batch-oriented data warehouses, to real-time analytics. Data freshness, low latency, and high concurrency. It requires new, purpose-built databases to capture data in real time from event streaming systems, index it quickly, and make it immediately available for querying. Plus, they need flexibility and scalability to support magnitudes greater queries per second (QPS). These were the factors that gave rise to such databases as Apache Pinot™.

You can watch Kishore’s full keynote now:

Gwen Shapira: Modern SaaS is Real-Time SaaS

Gwen Shapira At RTA Summit 2023

Gwen Shapira, co-founder and Chief Product Officer at Nile, was the headline speaker on the second day of Real-Time Analytics Summit. The central premise of Gwen's talk is that “The real value of real-time data lies outside the C-suite, in the hands of users who need real-time data for real-time decisions and action.” As Gwen pointed out, "Business strategy doesn’t change every 20 milliseconds," but apps that empower line workers, front-line managers, and customers greatly benefit from real-time data. Gwen emphasized the need for data freshness so that users can make critical decisions throughout the day. Note: data freshness is not the same thing as a low-latency, responsive UI. What if your app renders quickly, but all the data presented is hours or days out-of-date? 

You can watch Gwen’s full keynote presentation here:

Panel Discussion: Microsoft, LinkedIn, Cisco, and DoorDash Weigh In

The second day of the Summit also saw StarTree’s VP of Developer Relations, Tim Berglund, lead a lively and insightful panel discussion between four powerhouse leaders in real-time analytics:

  • Dipti Borkar, VP and GM, Microsoft

  • Kapil Surlaker, VP of Engineering, LinkedIn

  • Sachin Joshi, Sr Director of Engineering, Cisco Webex

  • Sudhir Tonse, Director of Engineering, DoorDash

They each spoke about the historic shifts real-time analytics allowed in their organizations. Kapil (LinkedIn) noted that a new type of database was required to run these workloads: "There's no way you're going to run that level of QPS from a data warehouse. Search kind of worked but wasn't built for this.” This was what led them to invent Apache Pinot internally. The value was only obvious after the new database was available. But it allowed LinkedIn to do things that were never possible before.

RTA Summit 2023 Real-Time Analytics Panel

From left to right: Tim Berglund, VP of Developer Relations, StarTree; Dipti Borkar, VP and GM, Microsoft; Kapil Surlaker, VP of Engineering, LinkedIn; Sachin Joshi, Sr Director of Engineering, Cisco Webex; Sudhir Tonse, Director of Engineering, DoorDash

Sachin (Cisco) talked about how the pandemic drove the growth for Cisco Webex. Suddenly it wasn’t just the service admins; it was HR department heads and CxOs that wanted to know what features were used and how much time was spent in-app. And it all needed live troubleshooting.

Sudhir (DoorDash) dove into the convergence of multiple waves of advances: how the adoption of microservices, public cloud services, and on-demand consumer services all drove the need for real-time data analytics.

Dipti (Microsoft) observed the move from batch, nightly updates to real-time was the drive to derive value out of data, plus the trend towards “Saas-ification.” While much began in open source, it’s become mainstream today, and noted the massive scale of Azure Analytics. She emphasized how Microsoft “dogfoods” and uses real-time analytics for telemetry and logging.

You can watch the full video here:

Real-Time Analytics Databases Compared

Real-Time Analytics is not a single-vendor driven industry. In fact, it is an industry primarily driven by a rich and diverse array of open source software options, including Apache Druid, Apache Pinot, and Clickhouse. To that end, A Tale of Three Real-Time OLAP Databases was a session comparing these three leading databases. Presented by StarTree’s Chinmay Soman and Neha Pawar, the goal was to objectively assess the current state of the art, clarify and objectively compare each respective database’s capabilities, and help guide users to the right real-time analytics database for their use case.

Apache Pinot, Apache Druid, and ClickHouse Comparison

This talk was accompanied by a related blog, which you can read in full: A Tale of Three Real-Time OLAP Databases: Apache Pinot, Apache Druid, and ClickHouse

Real-Time Analytics Summit was a great showcase for Apache Pinot. It included technical deep dives, discussions of the latest features and capabilities, real-world implementations, use cases, and lessons learned. Here’s a list of the Apache Pinot related talks. In due time we’ll make recordings of these available on-demand:

  • Query Processing Resiliency in Pinot — LinkedIn’s Vivek Iyer and Jia Guo shared how they worked to make Apache Pinot more robust by adding two new capabilities: first, Adaptive Server Selection at the Pinot broker to route queries intelligently to the best available servers, and second, RunTime Query Killing to protect the Pinot servers from crashing. I captured this deep technical dive live in this thread on Twitter.

  • Unlock the value of your data with Apache Pinot and AWS — AWS Solutions Architects JP Santana and Wahab Syed discussed how users can integrate Apache Pinot better into their existing AWS ecosystems.

  • Leveraging Pinot’s Plugin Architecture to Build a Unified Stream Decoder — DoorDash’s Mike Davis showed how the food delivery giant created Iguazu, their in-house stream processing framework with Apache Pinot by developing a custom StreamMessageDecoder implementation incorporating the Iguazu consumer library.

  • Gently Down the Stream with Apache Pinot — StarTree’s own Navina Ramesh explored how powerful real-time ingestion features in Apache Pinot can almost (but not quite) eliminate the need for stream processing pipelines. She also took a peek into the real-time engine at the heart of Apache Pinot, including its stream consumer, decoder, indexer, pipeline transformer, and upsert manager. Last, she looked at transform functions and ingestion transforms available in Apache Pinot.

  • Minions to The Rescue—Tackling Complex Operations in Apache Pinot — StarTree’s Haitao Zhang and Xiaobing Li discussed the purpose and implementation of the Minion component in Apache Pinot. In brief, they are standby helpers, built on top of Apache Helix, that can improve certain mundane, generic, or even complex tasks such as file ingestion, segment refresh, time or value partitioning, or data merges or splits. My Twitter thread gives a brief glimpse into their talk here.

  • Hydrating Real-Time Data in Pinot with Fast Batch Queries on Trino — Starburst’s Elon Azoulay and Martin Traverso showed how to combine Trino, the distributed SQL query engine, with Apache Pinot. It also went into the recent updates in the Trino-Pinot connector.

  • Building a Real-Time IoT Application with Apache Pulsar and Apache Pinot — Cloudera’s Tim Spann (pictured) is always a great presenter for users looking to get hands-on, how-to practical guidance. His live demo took attendees step-by-step through getting IoT thermal sensor data first into Apache Pulsar event streams, and then into Apache Pinot for analysis via the pinot-pulsar stream ingestion connector. You can read his detailed blog here and check out the GitHub repo here.

Timothy Spann At Real Time Analytics Summit 2023

  • Backfill Upsert Table Via Flink/Pinot Connector — Uber’s Yupeng Fu shared how the transportation leader decided to deal once and for all with the real-world pain of backfilling upsert data. The challenge is that upserts tend to be on a live table, and backfilling is traditionally a batch task. So what’s the best way to tackle this?

StarTree Cloud Talks

StarTree Cloud, powered by Apache Pinot, also includes additional capabilities. Two sessions delved into specific features of our fully managed cloud Database-as-a-Service (DBaaS):

More Industry Talks

Thank You to Everyone!

An event like Real-Time Analytics Summit takes a lot of work in preparation, content review, behind the scenes, and on stage. To all of our sponsors, speakers, reviewers, StarTree staff, and attendees, I wanted to thank you all for making the 2023 event so successful. Look forward to more on-demand videos in the coming weeks ahead and more blog posts that will provide deeper dives into the topics presented in the individual sessions.

Apache Pinot