Apache Pinot is built for real-time analytics at very large scale. Production clusters can hold hundreds of terabytes to petabytes of data, ingest millions of events per second, and return queries in milliseconds.
For most deployments, scaling Pinot is straightforward: add servers as data and query volume grow.
At extreme scale, however, the challenge grows beyond how much data Pinot can store or query. It is how many individual pieces of data the cluster has to coordinate.
In our scale testing, we found that as Pinot moves above a million or more segments, control-plane work can begin to dominate. Operations such as adding segments, recovering servers, maintaining cluster state, and coordinating ingestion can become increasingly expensive even when the underlying servers remain capable of storing and querying more data.
In this scenario, the data plane can still have room to scale while the control plane becomes the bottleneck.
To address that problem, StarTree has introduced Segment Groups, a capability for Apache Pinot that reduces the number of objects the control plane needs to manage without physically merging the underlying segments.
The result is an architecture that can scale to 10 million segments and beyond while retaining the fine-grained data layout that makes Pinot efficient for real-time analytics.
When segment count becomes a scaling dimension
A Pinot table is made up of segments: immutable, self-contained columnar units containing data and indexes.
Segments are fundamental to how Pinot operates. They are used for:
- placement and replication across servers;
- query routing and pruning;
- parallel query execution;
- retention and lifecycle management;
- ingestion and background processing.
That model works extremely well because each segment can be managed independently.
But independence also has a cost; every segment is an object the Pinot control plane needs to know about.
The controller needs to track where it should live. Apache Helix needs to reconcile its desired and actual state. ZooKeeper stores metadata describing it. Brokers use segment information when deciding where queries should be routed.
As the number of segments grows, so does the amount of metadata these components must continually process.
At ordinary scale, this overhead is small relative to the work the cluster is doing. At very high segment counts, that relationship can reverse.
How to recognize the metadata ceiling
The symptoms don’t necessarily look like a storage problem.
You may still have sufficient server capacity. Query execution itself may continue to scale. But operational tasks around the data become progressively slower.
In our scale tests, at approximately one million segments we saw an IdealState update take around 25 seconds. A full Helix reconciliation pass at approximately 1.5 million segments could take more than two minutes.
The same underlying pressure can show up in several ways:
- Ingestion falls behind. The controller cannot coordinate newly created segments quickly enough to maintain the required ingestion rate.
- Controller CPU and memory rise. In our testing, ZooKeeper deserialization consumed about 40% of CPU and 65% of memory allocations, while garbage collection accounted for roughly another 30% of CPU.
- Server recovery becomes slow. Bringing a server back online can require Helix to coordinate state transitions for very large numbers of segments.
- ZooKeeper processes substantially more data. Large cluster-state structures have to be repeatedly read, serialized, and written, increasing snapshot size and write volume.
The important point is that all of these symptoms have a common cause.
Too many individual control-plane objects are being processed on each operation.
Adding more data servers doesn’t address that problem, because the bottleneck isn’t the capacity of the servers storing the data.
Why this problem is becoming more common
Two workload patterns make very large segment counts particularly important.
Extreme-scale real-time ingestion
Consider an application ingesting on the order of 100 million events per second while maintaining sub-minute freshness and a 30-day retention window.
At peak ingestion rates, that workload can create approximately 300,000 new segments per day and accumulate around 10 million live segments.
Pinot’s servers can scale to handle the ingestion and query workload. The harder problem becomes coordinating the metadata associated with all of those segments.
Large open-table-format datasets
The same issue appears from a completely different direction with open table formats such as Apache Iceberg.
Large Iceberg tables can contain millions of Parquet files. When those files are exposed through StarTree for low-latency querying, file metadata maps naturally to Pinot segments.
A five-million-file Iceberg table can therefore begin life as a five-million-segment Pinot table.
Unlike the real-time use case, you aren’t gradually approaching a high segment count. You can start there on day one.
Both patterns point to the same architectural question:
How do you preserve fine-grained physical data while avoiding fine-grained control-plane coordination?
What to do before changing the architecture
If segment count is becoming a concern, there are already several useful tools available to Pinot operators.
They remain good practices. But it is important to understand what each one does—and where its limits are.
Use fewer, larger segments
For many Pinot deployments, this should be the first lever to consider.
Minion merge tasks can compact multiple small segments into fewer larger segments, directly reducing the number of objects the control plane has to manage.
There are trade-offs, however.
Physical merging requires data to be rewritten, so its cost grows with data volume. Larger segments can also reduce pruning precision: a segment covering a wider time range is less likely to be eliminated entirely by a time filter.
Compaction therefore works well when the physical segment layout itself can be simplified.
It is less attractive when the existing fine-grained layout is useful and only the metadata cardinality is causing problems.
Split very large tables
Another option is to divide a large physical table into several tables and expose them behind one logical table.
This can reduce some controller pressure. In our testing, for example, controller garbage-collection time fell from approximately 30% to around 15% at the same overall segment count.
But splitting tables does not reduce the total amount of metadata the cluster must ultimately process.
Helix still sees the cluster’s overall ZooKeeper state. In addition, splitting one large IdealState into many smaller ones can reduce the effectiveness of commit batching. In our testing, splitting a workload across 10 physical tables resulted in roughly five times as many IdealState updates per second at the same segment-creation rate.
Table sharding can therefore buy additional headroom, but it does not fundamentally change the relationship between segment count and control-plane work.
And if tables are being created primarily to stay below a metadata threshold rather than because they represent meaningful architectural boundaries, they introduce operational complexity of their own.
Segment Groups: change the unit of coordination, not the unit of storage
Segment Groups take a different approach.
Instead of making the physical segments larger, StarTree makes the control-plane representation larger.
A Segment Group combines the metadata for a configurable number of physical segments—around 100 is a typical example—into a single object from the perspective of the control plane.
The underlying segments are not merged.
The distinction is crucial.
The control plane operates on groups. The data plane continues operating on individual segments.
ZooKeeper, Helix, the controller, and brokers can therefore reason about dramatically fewer objects, while servers and minions retain access to the original segments.
With a grouping factor of approximately 100, a table containing 10 million physical segments can be represented by roughly 100,000 control-plane groups rather than 10 million individually coordinated objects.
That changes the scaling equation without requiring the data itself to be rewritten.
Why metadata grouping is different from segment merging
Physically merging segments and grouping their metadata may initially sound similar, but they solve different problems.
With Segment Groups:
- the underlying segment files stay where they are;
- no large-scale physical data rewrite is required;
- hundreds of segments can be represented by one group regardless of their combined physical size;
- servers can still perform precise operations against individual segments;
- control-plane components enumerate far fewer objects.
For real-time tables, grouping can happen as a background operation after individual segments are created.
For Iceberg tables, groups can instead be formed when file metadata is ingested, allowing large batches of files to enter Pinot already represented as a smaller number of control-plane objects.
This separation between physical granularity and coordination granularity is the core architectural idea behind Segment Groups.
What changes—and what doesn’t
Introducing another level of abstraction into a distributed system raises reasonable questions.
Does the data move? Does query behavior change? Is recovery more expensive? Does grouping sacrifice pruning precision?
The design of Segment Groups is intended to keep those effects isolated to the control plane.
The physical data does not change
Segment Groups merge metadata, not segment files.
The server eventually expands a group back into its constituent physical segments and continues loading and querying those segments individually.
That means grouping does not require rewriting the underlying dataset.
Group creation is atomic
When individual segments are replaced by a group, queries must never see both representations simultaneously—and must never see neither.
Pinot already has machinery for performing this kind of transition through segment lineage.
During group formation, the existing segment representation remains active until the replacement group is ready. Once every replica of the group is online, routing transitions to the group. The standalone segment metadata can then be retired.
From the query’s perspective, the change is atomic.
Servers normally don’t need to download the data again
Grouping metadata does not imply re-fetching every segment from deep storage.
When a server brings a group online, it can use the cheapest valid source for each member:
- if the segment is already loaded with the expected CRC, it can simply be re-registered;
- if the files are already present locally, they can be loaded from disk;
- only genuinely missing files need to be fetched from deep storage.
The same physical segment can temporarily be referenced by both its original standalone representation and its new group representation during the transition.
So in the common case, creating a Segment Group is primarily metadata work rather than data movement.
Query pruning remains precise
Grouping does make one stage of pruning coarser.
A broker sees the metadata associated with a group rather than every individual segment, so a group can cover a wider time range than one segment.
But broker pruning is only the first stage.
After a server receives the request and expands the group, it can still perform precise pruning against the group’s individual segments using their time ranges and partition information.
The broker reduces fan-out at group granularity; the server retains fine-grained pruning at segment granularity.
The physical query execution model therefore remains segment-oriented.
Segment Groups and Iceberg
Iceberg adds another requirement: snapshot consistency.
A query pinned to an Iceberg snapshot must see exactly the set of files belonging to that snapshot, even while files are being added or removed.
For Iceberg-backed tables, Segment Groups are therefore immutable and scoped to snapshots.
A change in the underlying file set can create a new group alongside the old one, while the snapshot manifest determines which group a particular query should use.
Group names are content-addressed based on their member set. Recreating the same set is therefore idempotent, and unchanged partitions can reuse an existing group across snapshots.
Physical files are also shared by reference and are reclaimed only when no live group still references them.
This allows millions of Iceberg files to be represented by tens of thousands of routable control-plane groups without giving up snapshot-specific query semantics.
When to consider Segment Groups
Not every Pinot deployment needs this capability.
For most clusters, the normal segment model provides an excellent balance of simplicity, isolation, and scale.
Segment Groups become relevant when the number of segments itself starts becoming an architectural constraint.
That may be the case if:
- you expect segment counts to reach into the millions;
- high ingestion rates and short freshness intervals continually create large numbers of segments;
- ingestion remains computationally manageable but controller coordination begins falling behind;
- server restart or recovery time increases significantly as the table grows;
- controller CPU, GC, or memory increasingly goes toward processing cluster metadata;
- you’re onboarding Iceberg or similar datasets containing millions of files;
- you’re increasing physical segment size primarily to control metadata rather than because larger segments are otherwise desirable;
- you’re creating artificial table boundaries solely to stay below a control-plane scaling threshold.
In those situations, it is worth distinguishing between two very different problems:
Do you have too many physical pieces of data, or does the control plane simply have to keep track of too many things?
If the physical granularity remains useful, changing the unit of coordination can be a better answer than changing the data layout itself.
Scaling Pinot beyond 10 million segments
Pinot’s architecture already distributes data storage, ingestion, and query execution across servers.
At extreme segment counts, the remaining challenge is making sure the cluster’s coordination model scales at the same rate.
Segment Groups do that by allowing different parts of the system to operate at different levels of granularity.
The control plane works with aggregated metadata. The data plane continues working with individual segments.
That makes it possible to reduce control-plane cardinality by roughly two orders of magnitude while preserving the physical segment model underneath it.
For very large real-time workloads and open-table-format datasets, that allows a single Pinot cluster to move from managing around a million individually coordinated segments toward 10 million segments and beyond.
The bottleneck was never how much data Pinot could hold. It was how many things it had to remember. Segment groups change what counts as a thing.
Segment Groups are now available as an opt-in StarTree capability for Apache Pinot.
