Real-Time Observability of Financial Transactions Using StarTree Cloud

Summary

• A financial services company required a distributed tracing architecture that could scale to meet its needs
• The company standardized on Open Telemetry and open source distributed tracing tools including Jaeger and Hypertrace
• The company used Apache Kafka to stream open telemetry data in real-time
• StarTree, powered by Apache Pinot, was selected for real-time analytics

Financial services companies need to track financial transactions across a wide range of payment methods. They might accept over 100 payment methods from credit and debit cards, to net banking, to various national and international payment standards and currencies depending on the region wherein they operate. To ensure payments are processed successfully, or disputes, if they arise, are resolved swiftly, one financial services company developed an extensive distributed tracing architecture for transactions, making it a first-class citizen in their observability architecture.

As their team delved deeper into the needs of real-time transactions tracing, the more it recognized the limitations of traditional Application Performance Monitoring (APM) tools and services. Their metrics infrastructure could not scale to handle trillions of data points, nor could it easily answer specific questions with high cardinality such as which merchants were currently seeing issues via a specific payment route?

Moving to a disaggregated observability stack

Many current single-vendor observability solutions lock customers into suboptimal choices. They can’t mix-and-match best-of-breed solutions at each layer of their observability stack. Replacing or substituting any component at any layer requires migrating away from the entire stack. Lastly but importantly, customers are locked into pricing in a disadvantageous manner. 

By moving to a disaggregated observability stack, the company can take advantage of the following benefits:

The financial services company wanted to scale their metrics and distributed tracing framework to handle over 100,000 spans in a single request while not overspending on their infrastructure. They looked at moving towards a disaggregated observability stack, and selected a number of different elements, mixing open source software (OSS) as well as commercial vendor solutions.

Observability produces an immense amount of real-time data that the customer knew would require data streaming, as well as a fast analytics platform to interpret these signals and make sense of the data points in real-time.

They required their stack to conform to the emerging Open Telemetry (OTel) standard, and also wanted to incorporate open source tools like Jaeger for distributed tracing agents, and Hypertrace for tracing visualization.

The financial services company had already embraced open source Apache Kafka as their real-time data streaming solution, and employed Confluent Schema Registry on top of it to provide normalization, grouping, trace enrichment and view generation.

For real-time analytics the company employed StarTree Cloud, powered by Apache Pinot. Pinot immediately ingests and indexes fresh open telemetry data coming directly from Kafka topics, and provides milliseconds-to-seconds scale insights that power observability dashboards built on Hypertrace.

Underneath it all, the company ensured each of these services were orchestrated using Kubernetes.

Advantages of StarTree Cloud

A few specific advantages StarTree Cloud provided were Apache Pinot’s star-tree index as well as tiered storage.

Star-Tree Index

By employing the star-tree index, unique to Apache Pinot, the company saw an immediate query performance improvement of 4× on key queries over traditional range or inverted indexes.

Tiered storage

Moving from an all Solid State Drive (SSD) deployment to a mixed SSD – S3 object store configuration using StarTree Cloud tiered storage meant the company could more than double its overall data storage allowing longer data retention while also saving roughly 50% of their costs. This alone addressed numerous business needs while remaining transparent to users.

Learn more about Apache Pinot

Apache Pinot was designed for real-time, user-facing analytics. Discover how Apache Pinot provides significant benefits over other real-time analytics databases like Apache Druid or Clickhouse.

Build with StarTree on AWS

The company was deployed on AWS as its cloud provider. StarTree is an AWS ISV Accelerate partner. Discover how to build your business on AWS, provisioning StarTree Cloud directly from the AWS Marketplace.

Discover StarTree Cloud

If this case study sparks ideas for your own data architecture and use cases, you’re encouraged to learn more about StarTree Cloud, its advantages over open source Apache Pinot. If you already have heard enough, feel free to schedule a demo or sign up for a 30-day free trial.