When a Zone Fails, Don't Let Your Cluster Rebalance Itself to Death

An availability zone goes dark and takes a third of the cluster with it. My instinct, the first time I thought this scenario through, was that the danger is the lost capacity. It is not. A well-placed cluster keeps serving reads and writes through a zone loss. The danger is what the datastore does next: it notices thousands of missing shard copies and starts rebuilding every one of them onto the nodes you have left — the same nodes now carrying the surviving traffic.

InfoQ's July 2026 write-up of how Uber keeps OpenSearch clusters serving through zone outages crystallized this for me, because the fix is the opposite of what monitoring culture trains you to want. Uber configures its clusters to refuse to heal. A zone loss leaves shards unassigned, the cluster goes yellow, and it stays yellow — deliberately — until the zone returns. What I found when I dug through the allocation-awareness docs behind that decision is a pattern that generalizes far beyond OpenSearch: declare every failure domain up front, so the cluster can tell the difference between a node that died and a domain that is missing.

The heal that kills you

To see why the default behavior is dangerous, follow what Elasticsearch and OpenSearch do when a node leaves the cluster. The sequence has three steps: the manager promotes replicas to replace lost primaries, it allocates new replicas on the remaining nodes to restore the configured copy count, and it rebalances shards evenly across whatever is left. Step one is cheap — a metadata change. Steps two and three are bulk data movement.

For a single dead node, that movement is the right call. The default index.unassigned.node_left.delayed_timeout even waits one minute before starting, precisely because the docs recognize that copying whole shards over the network to replace a node that reboots and comes back two minutes later is wasted work — the docs call this out as shard shuffling the cluster never needed.

Now scale the failure from one node to one zone. In a three-zone cluster, a zone outage removes a third of the nodes at once. Every shard that had a copy there is now under-replicated, and the allocation logic sees no reason not to fix that. So the cluster attempts to re-create a third of its total data set — reading from the surviving copies, streaming over the network, writing to disks on the survivors. The nodes doing this extra work are the same ones absorbing the failed zone's query and indexing load. Uber's engineering write-up names the cost directly: large-scale data movement across surviving nodes consumes disk I/O, CPU, and network bandwidth, and can push the remaining zones into instability, latency spikes, or cascading failure.

There is a second-order trap hiding in disk usage. Re-replication concentrates data onto fewer machines, and the disk watermarks that govern allocation sit at 85% (stop allocating new shards), 90% (start relocating shards away), and 95% (flood stage — index writes get blocked). A cluster that was comfortably at 60% disk per node before the outage can walk itself into flood stage during the heal. At that point the failure has cascaded from "one zone offline" to "cluster refuses writes," and no zone failure was required for the second step — the recovery caused it. AWS documented the same failure shape in the managed OpenSearch Service: redistributed shards skew onto surviving nodes, skewed shards mean skewed traffic, and a domain without storage headroom ends up write-blocked mid-recovery.

The uncomfortable conclusion I keep coming back to: after a zone loss, the cluster's healing reflex is a denial-of-service attack it launches against itself.

What "stable yellow" actually preserves

The alternative is to hold the degraded state. If every index keeps at least three copies — one per zone — then losing a zone leaves every shard with two live copies. Reads work. Writes work. Quorum-style replication still has a majority of copies. Nothing about the outage, by itself, threatens availability.

Cluster health goes yellow, and this is where the framing matters. Yellow is a statement about replica assignment, not a verdict on whether the cluster works. In a zone outage, an unassigned replica is not a fault to be fixed — it is an accurate description of reality. The zone is gone; the replicas that lived there have nowhere valid to be. A cluster that reports yellow and serves traffic at normal latency is in a better state than one that reports green because it crammed three copies of everything into two zones while running its disks toward flood stage.

The insight I took from Uber's design is that "yellow" has to become a stable state, not a transitional one. The default configuration treats yellow as a queue of pending repairs. The zone-resilient configuration treats yellow as the designed answer to a specific question: what should the cluster do when a whole failure domain is missing? Answer: nothing. Wait.

Uneven zones quietly break awareness

Getting to that answer takes two mechanisms, and the first one is the mundane-looking prerequisite that physical topology quietly breaks.

Allocation awareness is the standard feature: tag each node with an attribute (zone, rack_id), list that attribute in cluster.routing.allocation.awareness.attributes, and the allocator spreads copies of each shard across attribute values so a single domain failure cannot take out all copies of anything.

The catch is arithmetic. Awareness can only spread copies evenly if the domains have room for them evenly. Physical zones rarely cooperate — hardware generations differ, capacity arrives in one zone before another, a batch of nodes gets replaced. Uber found that with uneven node counts per zone, the awareness logic fails to find valid placements for every shard, and the cluster sits in a bad yellow: shards unassigned not because a zone died but because the topology never had a legal home for them, plus disk skew and hot nodes on the overloaded side.

Their fix is a layer of indirection they call isolation groups: logical failure domains guaranteed an equal node count, each mapped onto a physical zone, with membership that survives hardware replacement. Uber runs three groups for most services, so one physical zone maps to exactly one group and a zone failure removes at most a third of capacity — and the pattern is the default for every Tier 3 and higher OpenSearch and Elasticsearch cluster they operate. The abstraction matters less than the invariant it enforces — equal capacity per declared failure domain. Any platform can maintain the same invariant with node labels and provisioning discipline. Without it, everything downstream of awareness degrades from a guarantee into a best effort. Uber runs the same pattern for Apache Pinot, mapping isolation-group IDs onto replica-group pools so segment replicas survive a full zone loss — which tells me the invariant is portable, not an OpenSearch quirk.

Forced awareness is the cluster refusing to heal

Plain awareness has a default that surprises people: it governs where copies go, not whether they go. The Elasticsearch reference is explicit that if one location fails, the allocator spreads the lost copies across the remaining locations. Awareness alone does not prevent the rebalance storm — it happily rebuilds a third of the data set inside the surviving two-thirds, keeping copies of each shard apart where it can.

Forced awareness closes exactly that gap. You declare the complete set of domain values in advance:

cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone-a,zone-b,zone-c

The declaration changes what a missing zone means. Without it, the allocator only knows about the zones it can currently see — when zone-c vanishes, the world simply has two zones, and two zones can hold three copies. With the full list declared, the allocator knows a domain is missing rather than nonexistent. Copies that belong in zone-c stay unassigned until zone-c nodes return. The docs state the trade plainly: the cluster will "prefer to leave some replicas unassigned" over overloading the remaining locations.

That unassigned state has exactly two exits, and both are worth knowing before you rely on it. The zone's nodes rejoin and the copies rebuild into their declared home. Or an operator rewrites the force list, dropping the dead value, and the cluster resumes the rebuild it was holding back. There is no timer, no gradual relaxation, no threshold at which the cluster changes its mind on its own. The config is the whole policy.

This is worth a picture: the same three-zone cluster, same zone loss, with and without the declaration — one side melting into a storm of copy arrows, the other side holding two copies per shard and a row of deliberately empty slots.

​

The two mechanisms compose into a single behavior. Equal-capacity domains make normal operation green — every shard finds a legal home, no disk skew. Forced awareness makes zone failure a stable yellow — no storm, no watermark spiral, and a recovery that begins only when the zone actually returns, at which point the missing copies rebuild into their declared home rather than shuffling twice.

One consequence worth naming: forced awareness also constrains scale-down. Shrink one zone's node count below the others and copies pinned to that domain have nowhere to go. The declaration is a contract, and the provisioning side has to keep honoring it. That is the operational price of making "missing domain" a first-class state.

The managers have to survive the same failure

Data shards are only half the story. The cluster manager quorum lives in the same zones, and a design that keeps data serving but loses the ability to elect a manager has still failed.

Uber runs five manager-eligible nodes across the three domains — a 2-2-1 split — instead of the textbook three. With cluster.auto_shrink_voting_configuration enabled (it defaults to true), the voting configuration tracks the live node set. Losing the two-manager zone leaves three voters, which re-form a quorum at two of three; the configuration then shrinks, so the cluster tolerates one further manager loss. Zone-plus-one, sequentially. Three managers in three zones survives the zone loss too, but the next single failure after that ends elections.

I will not re-derive static quorum sizing here — the 2f + 1 arithmetic is its own topic. The point specific to zone failure is dynamic: auto-shrink buys tolerance for sequential failures, not simultaneous ones, and the reference docs are careful to say the voting configuration only sheds departed nodes while it still holds at least three. Five managers is not "more is better"; it is the smallest count where a whole domain plus a straggler can die in order and the cluster still holds elections.

When self-healing is the right answer

The honest counter-argument to all of this: yellow is a window of reduced redundancy, and the window stays open for the length of the outage. With three copies and a zone down, every shard is one node failure away from a single surviving copy, and forced awareness will not rebuild that lost copy in the surviving zones. The design trades a certain, self-inflicted overload against an uncertain, low-probability second failure. Uber's write-up frames it as trading immediate full replication for stability, and I think that trade is right for large clusters — but it is a trade, and it obligates you to monitor the yellow state and to have a decision point for a long outage (re-declare the force list to two zones and accept the rebuild, on your schedule, at a throttle you choose).

The choice also depends on what failed and what headroom exists, which is why I would not apply forced awareness reflexively:

• Transient single-node blips are what delayed allocation already handles. Tuning index.unassigned.node_left.delayed_timeout up from its one-minute default is the cheap knob, and it needs no topology discipline.
• Small clusters with real headroom — the kind where survivors can absorb a full re-replication below the 85% low watermark and the recovery traffic fits the network — lose little by letting the default heal run. The storm is a scale phenomenon; at three nodes it is a drizzle.
• Anything without equal-capacity domains cannot use forced awareness safely. Fix the arithmetic first or the config produces the bad yellow — permanently unassigned shards in a healthy cluster.
• Managed platforms are converging on this anyway. AWS OpenSearch Service applies zone awareness and load-aware shard allocation during zonal failures on its own: the allocator weighs provisioned capacity, actual capacity, and total shard copies against the expected average shards per node, and refuses to hand a surviving node more than its share of the dead zone's shards. I read that as the strongest available signal that the vendors handling the most zone outages consider refuse-to-rebalance the correct default at scale.

What stays with me from this study is that the pattern inverts a habit. I spent years treating cluster self-healing as the feature and manual intervention as the failure mode. At zone scale it flips: the mature configuration is the one that encodes, ahead of time, which failures the cluster should not try to fix.

Takeaways

• The default reaction to a lost zone is to rebuild a third of the data set onto the surviving two-thirds — recovery traffic and disk watermarks can cascade one zone outage into a write-blocked cluster.
• Keep at least three copies of every shard, one per failure domain; a zone loss then costs redundancy, not availability.
• Declare all failure domains up front (forced awareness) so the cluster distinguishes "domain missing" from "domain never existed" and holds a stable yellow instead of healing.
• Awareness math only works over equal-capacity domains; enforce equal node counts per zone (Uber's isolation groups) before anything else.
• Run five manager nodes in a 2-2-1 split with voting auto-shrink to survive a zone plus one sequential failure.
• Treat stable yellow as an obligation: alert on it, bound it in time, and pre-decide when and how you force the rebuild during an extended outage.

Reach for this when the cluster is large enough that re-replicating a zone's data would saturate survivor disks or network, when domains have (or can be given) equal capacity, and when you can staff the runbook that a deliberate yellow requires. Avoid it for small clusters with ample headroom, for topologies with uneven zones, and as a substitute for delayed-allocation tuning on ordinary node churn — the storm you are defending against has to be real before it is worth teaching your cluster to stand still.