By leveraging off-heap upserts with StarTree Cloud, Razorpay was able to reduce infrastructure costs by 50%.
Razorpay’s Real-Time Analytics Use Cases
A number of projects at Razorpay utilize StarTree Cloud, powered by Apache Pinot:
- Harvester — a series of microservices use cases revolving around real-time low-latency aggregations. For example, overviews of user accounts and activity (incoming money, payouts, reversed payouts, contacts created on RazorpayX, commissions, volume of transactions by sub-merchants, and success rate monitoring.
- Warehouse — merchant-facing queries that require sub-second latencies. Datasets are populated into Apache Pinot via an Apache Spark pipeline which reads data from a data lake on Amazon S3.
- Success Rate Analytics (SRA) Dashboard — Enables customers and internal teams to monitor payment success rate (SR) trends in real-time against various dimensions such as payment methods, banks, and so on. Also enables them to understand the failure reasons. Related to this are a transaction errors insights dashboard, as well as an ecosystem page to show the health of all banks and networks. Learn more here and here.
- Anomaly Detection — an internal-facing use case utilizing StarTree ThirdEye to detect and provide root cause analysis (RCA) for abnormalities from expected day-to-day behavior.
Razorpay’s Real-Time Analytics Architecture
Razorpay’s real-time data analytics architecture begins with various upstream transactional databases, including PostgreSQL, MySQL and TiDB. From those sources Razorpay uses Maxwell’s Daemon for Change Data Capture (CDC) to feed Apache Kafka streams. Apache Spark is utilized for real-time lookups from TiDB and to also provide stream processing (enrichment, denormalization, etc.), which is then fed back into Kafka topics.
Eventually data arrives in StarTree Cloud (powered by Apache Pinot), where it can be queried, along with other systems, by the distributed query engine Trino, which presents the data to Google Looker and Apache Superset for visualization.
Razorpay Leverages Scalable Upserts
StarTree Cloud includes capabilities for scalable real-time upserts. Upserts are two operations in one. If data does not already exist in the database, the operation proceeds just like an insert. Yet if a data record does exist, it acts as an update instead. With open source Apache Pinot, upserts are kept on-heap, which can cause memory pressure at high rates of ingestion, and would require bigger nodes, increasing costs as Razorpay scaled. With StarTree, Razorpay was able to manage the high volume of incoming events without impacting performance. In fact, Razorpay found that utilizing StarTree saved them 50% of the cost of AWS infrastructure for hosting upsert tables, allowing them to scale efficiently as their usage grows.
“We never had true real-time analytics per se before Apache Pinot. We had makeshift solutions that were based on Elasticsearch and Postgres. But in those systems, the average data freshness was around 15 to 20 minutes, because those [datasets] were still running as batches. With Pinot, we have a real-time stack based on Spark Streaming and Pinot.”
Tech Lead
Learn More About Apache Pinot
Apache Pinot was designed for real-time, user-facing analytics. Discover how Apache Pinot provides significant benefits over other real-time analytics databases like Apache Druid or Clickhouse.
Build with StarTree on AWS
Razorpay uses AWS as its cloud provider, and StarTree Cloud is an AWS ISV Accelerate partner. Discover how to build your business on AWS, provisioning StarTree Cloud directly from the AWS Marketplace.
Try Apache Pinot on StarTree Cloud
The best way to get experience real-time analytics and Apache Pinot is with StarTree Cloud. Request a trial to get started with a managed serverless account. Or Book a demo if you have questions and would like a quick tour of how StarTree Cloud can work for you.