In May 2026, AWS reported a thermal event affecting infrastructure in a single Availability Zone. EC2 instances and EBS volumes on impacted hardware were impaired, and AWS shifted traffic away from the affected zone for many services.
That is exactly the kind of failure high-availability architectures are supposed to survive.
And yet, incidents like this are a reminder that “multi-AZ” is not a magic phrase. An application only benefits from availability zones if its compute, storage, routing, scheduling, and data replication are actually designed around those failure boundaries.
And for real-time analytics, this matters even more. Apache Pinot often sits behind user-facing dashboards, observability workflows, personalization systems, anomaly detection, fraud detection, and operational decision-making. When the analytics layer becomes unavailable, organizations do not simply lose a report. They lose live visibility into what is happening right now.
Apache Pinot gives organizations a strong distributed foundation for this kind of workload. But high availability does not come from Pinot alone, or Kubernetes alone, or a cloud provider alone. It comes from aligning all of those layers around the same failure boundaries.
Put simply:
High availability is not just about having replicas. It is about making sure replicas do not fail together.
“Multi-AZ” is the starting point
Many organizations know they should run critical systems across multiple Availability Zones. That is a good starting point, but it is not the same thing as knowing how every component behaves when one of those zones becomes unhealthy.
A Pinot deployment may span multiple zones, but there are still important questions to answer:
- Are brokers spread across zones?
- Are controllers and ZooKeeper nodes deployed with the right redundancy?
- Are server replicas for the same table segments placed in independent failure domains?
- Does the load balancer stop sending traffic to unhealthy brokers?
- Can brokers route queries to healthy server replicas when part of the cluster is unavailable?
- Is deep storage durable and accessible outside the affected zone?
- Do monitoring and alerting detect partial degradation before users do?
These questions are where high availability becomes an architectural exercise rather than a checkbox.
Pinot has the right primitives for resilient real-time analytics: replication, segment assignment, broker routing, replica groups, and Helix-managed cluster state. But those primitives need to line up with the infrastructure underneath them.
The cloud defines the failure domains. Kubernetes decides where workloads run. Pinot decides where data lives and how queries are routed.
A highly available deployment needs all three layers to agree.
The three layers of Pinot high availability
A resilient Pinot deployment depends on three layers working together: cloud infrastructure, Kubernetes placement, and Pinot-aware replication and routing.
1. Cloud infrastructure: define the blast radius
The cloud layer defines what can fail together.
A production Pinot deployment should be designed around the assumption that a zone, node pool, storage path, or network path may become impaired. The goal is not to prevent every infrastructure issue. The goal is to keep an infrastructure issue from becoming an application outage.
Start with one practical question:
If one Availability Zone becomes unavailable, what disappears with it?
Answer that question for each part of the system:
| Area | What to check |
|---|---|
| Compute | Are brokers, servers, controllers, and supporting services spread across zones? |
| Storage | Are local volumes, persistent volumes, and deep storage aligned with the recovery model? |
| Networking | Can clients still reach healthy brokers if one zone is impaired? |
| Load balancing | Are unhealthy endpoints removed quickly from routing? |
| Capacity | Is there enough headroom in the remaining zones to continue serving critical workloads? |
| Operations | Are monitoring, deployment, and recovery tools still usable during a zone event? |
For Pinot specifically, deep storage deserves special attention. Local disks and zonal volumes may be appropriate for runtime performance, but long-term segment durability should rely on storage that supports the required recovery model, such as S3, GCS, or ADLS.
2. Kubernetes placement: turn topology into runtime behavior
Kubernetes is the layer that turns cloud topology into actual workload placement.
A cluster may span multiple zones, but that alone does not guarantee that critical pods are distributed in a way that protects the application. Without intentional scheduling rules, replicas can still concentrate in the same failure domain.
For Pinot, placement affects several parts of the serving path:
- Brokers need to remain reachable.
- Controllers need to survive routine maintenance and infrastructure churn.
- Servers need to avoid concentrating replicated data in the same zone.
- Stateful pods need to stay aligned with their storage.
- Maintenance events should not drain too much capacity at once.
This is where Kubernetes controls such as topology spread constraints, pod anti-affinity, node pools, readiness probes, liveness probes, startup probes, resource requests and limits, and PodDisruptionBudgets become part of the HA design.
A useful test is to look at each Pinot component and ask:
Would Kubernetes be allowed to place too many of these pods in the same failure domain?
If the answer is yes, then the system may look redundant without actually being resilient.
3. Pinot-aware replication and routing: keep queries serving
Pinot is where replicas become query resilience.
At the Pinot layer, high availability depends on more than the number of replicas. It depends on where those replicas are placed, how segments are assigned, how brokers route queries, and how the cluster reacts when instances become unavailable.
A healthy Pinot cluster should not need to rebuild itself during an outage in order to keep serving. It should already have the right replicas in the right places.
For HA planning, focus on these questions:
| Pinot concern | Why it matters |
|---|---|
| Segment replication | Replicas should be placed across independent failure domains, not concentrated together. |
| Replica groups | Broker routing should be predictable and able to avoid unhealthy paths. |
| Broker routing | Queries should continue to reach healthy servers that hold the required segments. |
| Helix state | Cluster state should reflect which instances and segments are available. |
| Table configuration | Replication, routing, and assignment choices should match the workload’s availability goals. |
| Rebalancing and upgrades | Operational changes should avoid taking too much capacity offline at once. |
The most important work happens before failure. By the time a zone is unhealthy, the deployment should already be arranged so that traffic can continue flowing through healthy brokers to healthy server replicas.
What should happen during a zone failure?
Imagine a Pinot cluster running across three Availability Zones.
One zone becomes unhealthy.
In a well-designed deployment, clients stop sending traffic to unhealthy brokers in that zone. Brokers in the remaining zones continue receiving queries. Those brokers route queries to healthy server replicas that still hold the required segments. Helix observes the failed instances and updates cluster state. Kubernetes may reschedule workloads where possible, but query availability does not depend on instantly rebuilding the failed zone.
The system keeps serving because the blast radius was already contained.
That is the difference between recovery and availability.
Recovery is what happens after failure.
Availability is what keeps working during failure.
Both matter. But for user-facing analytics, the difference is important. If customers, operators, or businesses depend on real-time answers, then the serving path needs to survive common infrastructure failures without waiting for a full recovery cycle.
A practical HA checklist for Pinot operators
If you are evaluating your Pinot HA posture, a good first step is to map every critical component to the failure domain it depends on.
Here is a practical checklist to start with.
Broker layer
- Are there multiple brokers?
- Are brokers spread across Availability Zones?
- Does the load balancer route only to healthy brokers?
- Are readiness probes configured so unhealthy brokers are removed from traffic?
- Can remaining brokers handle the expected query load if one zone is unavailable?
Server layer
- Are table replicas distributed across independent failure domains?
- Are segment assignment and replica placement aligned with zone topology?
- Is there enough serving capacity if one zone is lost?
- Are server pods protected from excessive voluntary disruption during maintenance?
- Are startup probes configured for servers that need time to load segments?
Controller and coordination layer
- Are controllers deployed redundantly?
- Is the coordination layer resilient enough for the cluster’s operational requirements?
- Can the cluster continue serving existing queries during control-plane disruption?
- Are operational workflows designed to avoid unnecessary metadata or assignment churn during incidents?
Storage layer
- Is deep storage configured for durability beyond a single node or zone?
- Are segment recovery paths tested?
- Do persistent volume choices match the intended failure model?
- Are local disks treated as serving infrastructure rather than the only durable copy of data?
Routing and query behavior
- Are broker routing strategies aligned with replica placement?
- Can brokers avoid unhealthy or overloaded servers?
- Are partial responses monitored?
- Are latency percentiles monitored by workload and table?
- Are critical workloads isolated where appropriate?
Monitoring and operations
- Are alerts based on user-visible symptoms such as query latency, errors, partial responses, ingestion delay, and segment availability?
- Are alerts able to detect partial degradation, not just total outage?
- Are upgrades, rebalances, and maintenance workflows tested under load?
- Does the team regularly test zone-level failure scenarios?
The goal of this checklist is not to make HA feel more complicated than it needs to be. The goal is to make the design visible. Once each dependency is mapped, you can make informed tradeoffs about cost, complexity, and availability.
Where StarTree helps
If your organization wants to operate every layer of this model themselves. Apache Pinot gives you the flexibility to do that.
But if you want the performance and flexibility of Pinot without having to become experts in every operational failure mode of distributed OLAP systems.
That is where StarTree can help.
StarTree operates Pinot as a cloud-native real-time analytics platform. That means bringing together Pinot’s distributed systems primitives with the infrastructure, placement, routing, monitoring, upgrade, and recovery patterns required to run Pinot reliably in production.
For customers, this means Pinot deployments can be designed around real cloud failure domains rather than generic infrastructure assumptions. Brokers, controllers, servers, table replicas, monitoring, and operational workflows are managed with availability in mind.
In practice, that helps teams with:
- Pinot deployments aligned with cloud Availability Zones.
- Broker, controller, and server placement patterns designed around failure domains.
- Table and segment placement strategies that support resilient query serving.
- Safer upgrade, maintenance, and recovery workflows.
- Monitoring designed around real-world failure modes, including partial degradation.
- Operational support from a team deeply involved in Apache Pinot.
The value is not that failures disappear. The value is that common infrastructure failures do not have to become customer-facing analytics outages.
Pinot power, cloud-grade resilience
Cloud outages are inevitable. Infrastructure maintenance is inevitable. Nodes fail, zones degrade, networks partition, and workloads change.
A resilient Pinot architecture starts by accepting that reality and designing around it.
Apache Pinot provides the distributed analytics foundation: fast query serving, scalable ingestion, replication, segment assignment, broker routing, and cluster state management.
High availability comes from making sure those capabilities are aligned with the cloud and Kubernetes layers underneath them.
For teams building user-facing analytics, observability, personalization, fraud detection, or operational intelligence, that alignment matters. Real-time analytics is often most valuable during fast-moving moments, which are exactly the moments when the system needs to keep serving.
The practical lesson is simple:
Do not stop at “we have replicas.”
Ask whether those replicas can fail together.
Then design the system so that when one part of the cloud has a bad day, your real-time analytics keep answering.
