Razorpay: Analyzing Financial Transactions Using StarTree Cloud
Razorpay is India’s fastest-growing payment processing company that handles millions of financial transactions per year. Razorpay chose StarTree, powered by Apache Pinot, for real-time analytics in its user-facing applications, including its Success Rate and internal monitoring dashboards.
- p99 latencies
- 1-3 seconds
- events ingested daily
- 200 million
- for upsert tables costs
- 50% savings
Summary
- Query P99 latencies in low single-digit (1-3) seconds
- Nearly 200 million events ingested daily
- Razorpay saved 50% of their infrastructure costs for upsert tables moving to StarTree Cloud
Tanmay, Tech Lead at Razorpay, and Anand Anantaraman, Associate Director at Razorpay, share how the company uses StarTree’s real-time analytics to monitor payment success rates across the more than 100 payments methods that are used across India. They also use StarTree ThirdEye for anomaly detection to proactively alert partners when issues arise, and to conduct root case analysis to resolve them quickly. By leveraging off-heap upserts with StarTree Cloud, Razorpay was able to reduce infrastructure costs by 50%.
Razorpay’s Real-Time Analytics Use Cases
A number of projects at Razorpay utilize StarTree Cloud, powered by Apache Pinot:
- Harvester — a series of microservices use cases revolving around real-time low-latency aggregations. For example, overviews of user accounts and activity (incoming money, payouts, reversed payouts, contacts created on RazorpayX, commissions, volume of transactions by sub-merchants, and success rate monitoring.
- Warehouse — merchant-facing queries that require sub-second latencies. Datasets are populated into Apache Pinot via an Apache Spark pipeline which reads data from a data lake on Amazon S3.
- Success Rate Analytics (SRA) Dashboard — Enables customers and internal teams to monitor payment success rate (SR) trends in real-time against various dimensions such as payment methods, banks, and so on. Also enables them to understand the failure reasons. Related to this are a transaction errors insights dashboard, as well as an ecosystem page to show the health of all banks and networks. Learn more here and here.
- Anomaly Detection — an internal-facing use case utilizing StarTree ThirdEye to detect and provide root cause analysis (RCA) for abnormalities from expected day-to-day behavior.
Razorpay’s Real-Time Analytics Architecture
Razorpay’s real-time data analytics architecture begins with various upstream transactional databases, including PostgreSQL, MySQL and TiDB. From those sources Razorpay uses Maxwell’s Daemon for Change Data Capture (CDC) to feed Apache Kafka streams. Apache Spark is utilized for real-time lookups from TiDB and to also provide stream processing (enrichment, denormalization, etc.), which is then fed back into Kafka topics.
Eventually data arrives in StarTree Cloud (powered by Apache Pinot), where it can be queried, along with other systems, by the distributed query engine Trino, which presents the data to Google Looker and Apache Superset for visualization.
Razorpay Leverages Scalable Upserts
StarTree Cloud includes capabilities for scalable real-time upserts. Upserts are two operations in one. If data does not already exist in the database, the operation proceeds just like an insert. Yet if a data record does exist, it acts as an update instead. With open source Apache Pinot, upserts are kept on-heap, which can cause memory pressure at high rates of ingestion, and would require bigger nodes, increasing costs as Razorpay scaled. With StarTree, Razorpay was able to manage the high volume of incoming events without impacting performance. In fact, Razorpay found that utilizing StarTree saved them 50% of the cost of AWS infrastructure for hosting upsert tables, allowing them to scale efficiently as their usage grows.
"We never had true real-time analytics per se before Apache Pinot. We had makeshift solutions that were based on Elasticsearch and Postgres. But in those systems, the average data freshness was around 15 to 20 minutes, because those [datasets] were still running as batches. With Pinot, we have a real-time stack based on Spark Streaming and Pinot."
Learn More About Apache Pinot
Apache Pinot was designed for real-time, user-facing analytics. Discover how Apache Pinot provides significant benefits over other real-time analytics databases like Apache Druid or Clickhouse.
Build with StarTree on AWS
Razorpay uses AWS as its cloud provider, and StarTree Cloud is an AWS ISV Accelerate partner. Discover how to build your business on AWS, provisioning StarTree Cloud directly from the AWS Marketplace.
Discover StarTree Cloud
If this case study sparks ideas for your own data architecture and use cases, you’re encouraged to learn more about StarTree Cloud and its advantages over open source Apache Pinot. If you want to take next steps feel free to schedule a demo and sign up for a free account.
