Resources
Blog

Why StarTree Is the Best Rockset Alternative for Real-Time Analytics


1695678491-1661531391-screenshot_1
Chinmay Soman
released on
June 28, 2024

Let me begin by congratulating the Rockset team on the recent collaboration with OpenAI. It is a testament to the great product you folks have built. Unfortunately, this does mean that current Rockset customers need to find a suitable replacement.

If we look at the overall real-time analytics landscape, there are 3 technologies (and their related fully-managed versions) that stand out as a possible alternative:

  • Apache Pinot (StarTree)
  • Apache Druid (Imply)
  • ClickHouse (ClickHouse Cloud or Tinybird)

There are other similar technologies, but we’ll focus on these 3 since they’re the most popular and widely used in the market today. In this blog, I’ll go over why we think StarTree and Apache Pinot are the best replacement for Rockset in terms of features and use cases.

StarTree Cloud, powered by Apache Pinot

Apache Pinot is an open source distributed columnar database that forms the backbone of mission-critical analytical systems at companies like Uber, Doordash, Stripe, LinkedIn, Cisco, and others. Pinot is able to ingest data from various sources of data such as streaming (Kafka/Kinesis/Pubsub) and batch (S3/GCS/ADLS), as well as SQL sources such as Snowflake and BigQuery. It was designed for powering high QPS (tens of thousands of queries per second) on terabytes of data, all while keeping query latencies in milliseconds.

StarTree Cloud provides a fully-managed, cheaper, faster offering of Apache Pinot across all 3 cloud providers. StarTree Cloud’s rich ingestion experience, ability to handle massive QPS and data scale, and being battle-tested in production makes it a natural fit as a Rockset alternative. We’ve also added Vector Index support to Apache Pinot for doing similarity search queries on embeddings stored in the database. Let’s take a deeper look at some key features that make it the top alternative.

Top 3 reasons for choosing Apache Pinot and StarTree as a Rockset replacement

1. Support for real-time upserts

From the above list of alternatives, Apache Pinot is the only system that can support true real-time upserts — a key requirement when identifying a Rockset alternative. Rockset provides a way to upsert records in real-time for a given primary key. This is required to ensure query results are accurate in the presence of data upserts and deletes. This is also needed for analyzing Change Data Capture (CDC) streams coming from DynamoDB/MongoDB/MySQL/Postgres.

In our blog Real-Time Upserts in Apache Pinot and StarTree Cloud, we’ve described in detail how this works in Apache Pinot and how StarTree Cloud increases the scalability and robustness of this feature. Apache Druid/Imply does not support real-time upserts as of today, and it’s severely limited in ClickHouse (only supports asynchronous compaction, which is not real-time).

A comparison of real-time upsert support in Apache Pinot, Apache Druid, and ClickHouse

Figure 1: Real-time Upsert support in Apache Pinot, Apache Druid and ClickHouse

2. Support for handling unstructured or semi-structured data

Another key feature of Rockset that customers love is the ability to analyze free structured JSON or text documents. Apache Pinot has great support for this as well. Users can ingest nested JSON columns as-is — without needing to preprocess or transform it in any way. We can then configure a JSON index on this column and query it using the JSON_MATCH predicate, which allows us to filter relevant records within milliseconds. Here’s a sample query that leverages JSON index for doing a regex based filter:

SELECT ... FROM mytable 
WHERE JSON_MATCH(person, 'REGEXP_LIKE("$.addresses[*].street", ''.*st.*'')')

Copy

JSON index support in Apache Pinot enables extremely fast query processing on highly nested JSON columns, as shown in this benchmark comparing latency observed with and without the JSON index:

Benchmark comparing latency observed with and without the JSON index in Apache Pinot

Figure 2: Benchmark comparing query latency with and without the JSON index

To learn more, please refer to this great video tutorial about the JSON index.

In addition, Pinot also has a way to perform powerful text search queries on arbitrary text data. This can be achieved by configuring a Text Index on the corresponding column to accelerate query processing. For instance, here’s a sample query that leverages Pinot’s text index to filter out logs matching the given regex:

SELECT SKILLS_COL FROM MyTable 
WHERE text_match(SKILLS_COL, '/.*Exception/')

Similar to JSON index, Text Index also provides an order of magnitude improvement in query performance as shown in the following benchmark.

Benchmark showing query latency with and without Text Index

Figure 3: Benchmark showing query latency with UDF and with Text Index

3. Intelligent materialized views using the Star-Tree Index

The star-tree index, unique to Apache Pinot, provides users the ability to build an intelligent materialized view for pre-computing certain aggregations for a wide range of dimension filters. This provides many orders of magnitude improvement in query latency as well as overall query throughput as shown below:

Benchmark of QPS and query latency with and without the star-tree index

Figure 4: Benchmark showing QPS and query latency with and without the star-tree index

However, unlike traditional materialized views, the star-tree index provides an innovative way to control the degree of materialization done, thus allowing the user to trade off query latency versus storage overhead. It’s also dynamic and can be easily modified without having to re-ingest the data. Finally, users can configure multiple different star-tree indexes for the same table and the right index will be picked up for a given user query. This is a unique feature that does not exist in other databases (including Rockset).

The star-tree index is being used in most of the latency-sensitive, mission-critical applications such as the metrics store built by Stripe and the Restaurant Manager application from Uber. To learn more about star-tree index, check out this blog.

Feature comparison of StarTree Cloud vs. Rockset

Compared to the alternatives, StarTree Cloud comes closest in terms of feature parity with Rockset, especially when it comes to supporting real time upserts and analyzing unstructured data. The rich ingestion capabilities of Pinot ensure a seamless migration path for existing Rockset users. In addition, Pinot and Startree provide innovations like the star-tree index and cloud tiered storage that stand out as differentiators.

Here’s a comprehensive comparison of features across StarTree Cloud and Rockset:

Feature Rockset StarTree Cloud
Deployment
SaaS – Serverless
SaaS – Dedicated
BYOC
On Premises Open source only
Scaling T-shirt sizes Increase resources of individual service that needs more
Ingestion
Real-Time Ingestion
Bulk File Ingestion
(via stream)

(independent of stream, async)
Upserts
Partial Upserts Yes, but no access to current values
Deduplication
Insert-Time Aggregations
Time Partitioning
Column Value Partitioning Fixed count, PK only Configurable, any column, multiple
Background Re-Partitioning
Write API
(beta)
Materialized Views
(as rollups)

(as StarTree indices)
Backups
Queries
Postgres Compatible Query Language Due 2024
Wire-Compatible Driver Due 2024
Joins
Collocated Joins
Nested JSON Support
(different due to fixed schema)
Pagination (Beyond Limit & Offset)
(beta)
UDFs
(JavaScript)

(Groovy)
Query Lambdas
RBAC
Table Aliases
Query Scaling
(via multiple VIs)

(via k8s & replica groups)
Indices
Configurable Per Column
Columnar
Inverted
Time Column Partitioned
Range, Sorted
GeoSpatial
(S2)

(H3)
BloomFilters N/A
Text
JSON
(effectively)
Vector
Extras
Anomaly Detection
(ThirdEye)

Conclusion

Migrating a database is never easy, especially with many choices to consider. We strongly feel StarTree Cloud comes closest in terms of feature parity with Rockset, especially with features like real-time upserts, JSON and Text Index support, and the star-tree index.

If you are a current Rockset user — or even if you found this article compelling regardless of your current database of choice — the good news is that there’s now a StarTree Cloud free tier for you to get started prototyping in preparation for your migration. Also please make sure you contact us for a demo. We’d love to listen to your needs, understand your use case, and answer any questions you may have on whether StarTree Cloud is the best database for your real-time analytics needs.

CONTACT US TODAY

Ready to deploy real-time analytics?

Start for free or book a demo with our team.