How Real-Time Analytics with Apache Pinot and StarTree Keep Magic Eden Atop the NFT World

Summary

• NFT market leader Magic Eden used Apache Pinot and StarTree to cut response times for analytical queries from minutes to seconds.
• Apache Pinot improved query performance by 96%, reduced hardware costs through integration with RocksDB, and boosted developer productivity.
• Magic Eden’s trading platform now delivers real-time floor prices, trading volumes, top traders, and historical trends with exceptional accuracy, allowing users to make informed decisions in dynamic market conditions.
• Pinot’s upsert capability and dynamic filtering empower Magic Eden to outpace competitors in speed and reliability while enhancing developer satisfaction and reducing operational expenses.

Relying on real-time analytics for blockchain transactions

The growth prospects for non-fungible tokens (NFTs), the unique digital signatures representing ownership of an asset or content, are robust. Grand View Research estimates the global NFT market totaled nearly $36 billion in 2024 and will grow at a robust 34% annual rate through 2030.

As the leading NFT marketplace, Magic Eden must sustain its advantage by providing its NFT-trading customers with unparalleled real-time analytics. “We need to help users make real-time buying and selling decisions in a dynamic market,” said Sidney Zhang, Co-Founder and CTO of Magic Eden, in a presentation at the 2024 Real-Time Analytics Summit presented by StarTree.

Achieving this goal involves significant challenges. Magic Eden operates across five blockchains, generating vast volumes of activity. Because blockchain transactions are inherently public, it must contend with competitors who vie to offer faster real-time analytics based on Magic Eden’s own data. “There was a time when our competitors’ real-time analytics of our data were faster than ours. That was kind of embarrassing,” Zhang admitted.

NFT marketplace Magic Eden looks to Apache Pinot and StarTree

Magic Eden found solutions in StarTree Cloud, a managed real-time analytics service powered by Apache Pinot.

Real-time insights are critical in NFT trading, where prices can change from second to second. The most time-critical data elements include floor prices (the lowest price for items in a collection), trading volumes, and time-series data that helps to identify trends. Analytics needs to be accurate and low-latency, especially during high-activity periods like the launch of new collections or spikes in market activity.

Magic Eden previously relied on a combination of Spark ETL jobs, Kafka streams, and Postgres caches tied together by cron scripts. While functional, the setup’s slow ETL process created a five-minute delay in updating analytics, making it unacceptable for active traders. Efforts to implement Spark Streaming for real-time analytics proved prohibitively expensive.

Furthermore, implementing new metrics required learning complex tools and writing custom jobs. “To create new stats, someone had to write a new Python job,” Zhang said. “You had to learn a whole different toolchain.”

StarTree and Pinot reduced cache intervals from minutes to 5 seconds

The combination of Pinot and StarTree provided a scalable, efficient, and developer-friendly platform for handling mutable and immutable data. Switching to Pinot has transformed Magic Eden’s real-time analytics capabilities.

Pinot reduced cache intervals from minutes to as low as five seconds, ensuring near-instantaneous updates without any loss of functionality or compromises on data completeness. It improved query performance on high-traffic NFT collections by leveraging Pinot’s bitmap-based inverted indexes. “Pinot was one of the technological choices that we made that had no compromise on any dimension,” Zhang noted.

Pinot’s ability to handle SQL queries has made it easier for developers to implement new features with fewer bugs. That has paid off in the form of lower frustration levels among the team.  “Beyond developer productivity, it also helped with developer happiness,” Zhang noted.

Leveraging real-time upserts to ensure data accuracy and detect wash trading

Pinot’s upsert feature allowed Magic Eden to manage updates and deletes efficiently. Upserts combine update and insert functions to determine whether a record needs to be updated or appended to a data table. Upsert metadata tracks primary keys and timestamps, ensuring only the most recent data is used for queries, even when events arrive out of order. “We know that if older events come in, they won’t overwrite the data,” said StarTree Co-Founder Xiang Fu. “That makes it more versatile for real-world scenarios where data can change over time.”

Upserts helped solve challenges like ensuring data accuracy and detecting wash trading. In this market manipulation strategy, a trader simultaneously buys and sells the same financial instrument to create misleading activity. Using Apache Pinot’s star-tree index feature enabled Magic Eden to index popular NFT collections, drastically reducing query latency. Queries that used to take 2.5 seconds were decreased by 96% to 108 milliseconds.

StarTree’s scalable upserts are implemented using RocksDB, a high-performance key-value database. Using StarTree enabled Magic Eden to handle up to 20 times more data per machine, significantly lowering infrastructure costs without sacrificing performance.

Leading real-time NFT analytics while reducing costs

Adopting Apache Pinot and StarTree delivered transformative results for Magic Eden. Real-time analytics update seamlessly, empowering traders with actionable insights during market fluctuations. Developers can deploy features faster with fewer bugs, enhancing productivity and satisfaction. Magic Eden reduced operational expenses by optimizing hardware utilization with RocksDB and Pinot’s architecture. Most importantly, the company reclaimed its position as the leader in real-time NFT analytics, outpacing competitors in speed and reliability.

Magic Eden’s collection trading page now provides dynamic metrics such as real-time floor prices, top traders, and historical trends with unparalleled accuracy. Traders can use these insights to discover and quickly evaluate NFT collections. Distributed joins allowed Magic Eden to integrate dimensional data in real time, boosting query accuracy without compromising speed. Dynamic filtering cuts query times on distributed joins by substituting an inverted index lookup for a full table scan.

Adopting Apache Pinot and StarTree was a game-changer for Magic Eden in its pursuit of real-time analytics excellence. The platform now delivers fast, accurate, and scalable insights, empowering users to navigate the NFT marketplace confident that they are working with the latest and most accurate data.

“Pinot is very scalable and fast,” Zhang said. “It’s a lot easier for developers to work with. There’s less complaining, and the cost savings are significant.”

Watch the full video

View the full talk from Magic Eden from Real-Time Analytics Summit 2024 about how they use Apache Pinot and StarTree to power real-time NFT analytics.

Discover StarTree Cloud

If this case study sparks ideas for your own data architecture and use cases, you’re encouraged to learn more about StarTree Cloud, its advantages over open source Apache Pinot. If you already have heard enough, feel free to schedule a demo or sign up for a 30-day free trial.