At a glance
The single most consequential alert for a Galera (MariaDB Galera Cluster) deployment. It fires when the cluster loses its Primary Component, meaning quorum is gone, or when a node disappears from the cluster membership. This is MariaDB-distinctive: Galera quorum loss makes the cluster go read-only across all nodes to protect consistency. The cluster does not fail one node and carry on; it deliberately stops accepting writes everywhere until quorum is restored. For a DBA this is an all-hands event: writes are blocked, the storefront cannot place orders, and the only safe recoveries require careful, ordered intervention. There is no “wait and see” with split-brain.
| What it tracks | An alert list of quorum and membership events: wsrep_cluster_status leaving Primary, or wsrep_cluster_size dropping below its expected value (a node lost). Each entry records the timestamp, the observed status, and the new cluster size. |
| Data source | Galera wsrep status variables: SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status', wsrep_cluster_size, wsrep_local_state_comment, and wsrep_ready, sampled in real time across the connected nodes. |
| Time window | RT (real-time). Evaluated on every sample; an alert is raised the moment the condition is observed. |
| Alert trigger | wsrep_cluster_status != Primary OR cluster_size dropped. Either a non-Primary status (Non-Primary / Disconnected) or a reduction in node count fires the alert. |
| Severity | Critical. This is a Hero card. Non-Primary status means the cluster is read-only and writes are failing right now. |
| Roles | DBA, platform, SRE, on-call |
Calculation
Galera is a synchronous multi-master replication layer (wsrep). To avoid split-brain, it uses a quorum rule: a partition can only remain Primary (accept writes) if it holds more than half the votes of the last known membership. The card evaluates two conditions on every sample:wsrep_cluster_status takes one of three values: Primary (healthy, has quorum, accepts writes), Non-Primary (lost quorum, read-only, refuses writes with WSREP has not yet prepared node for application use), or Disconnected (the node is not part of any component). wsrep_cluster_size is the number of nodes currently in the Primary Component. A drop from 3 to 2 in a three-node cluster is a lost node but quorum survives (2 of 3 is still a majority); a drop from 2 to 1 in that same cluster usually means the surviving node also goes Non-Primary because 1 of 3 is not a majority. The MariaDB-distinctive consequence: when quorum is lost, every node refuses writes, not just the failed one.
Worked example
A platform team runs a three-node MariaDB 10.6 Galera cluster (node-a, node-b, node-c) across two availability zones, with node-a and node-b in AZ-1 and node-c in AZ-2. Snapshot taken on 18 Apr 26 at 14:20 BST when the inter-AZ network link degraded.
| Node | wsrep_cluster_status | wsrep_cluster_size | wsrep_local_state_comment |
|---|---|---|---|
| node-a (AZ-1) | Primary | 2 | Synced |
| node-b (AZ-1) | Primary | 2 | Synced |
| node-c (AZ-2) | Non-Primary | 1 | Initialized |
node-c’s status change and for the cluster-size drop from 3 to 2.
The on-call DBA confirmed the diagnosis on each side:
Synced, and cluster size went back to 3. No manual recovery was needed.
The dangerous variant: had the partition been even (a two-node cluster splitting 1/1, or a three-node losing two nodes at once), no side would have a majority and the entire cluster would go Non-Primary, blocking all writes. Recovery then requires SET GLOBAL wsrep_provider_options='pc.bootstrap=YES'; on the most up-to-date node, which is a deliberate, manual override only safe when you are certain the other partition is truly gone.
Three takeaways:
- Galera fails closed, not open. Unlike async replication where a lost replica is invisible to writers, Galera blocks writes cluster-wide on quorum loss. That is a feature (no split-brain, no divergent data) but it means a quorum event is always write-impacting.
- Run an odd number of nodes, or use a Galera Arbitrator (garbd). Even-sized clusters have no tiebreaker and are prone to total write-block on a clean 1/1 split. Three nodes (or two nodes plus garbd) survive a single failure with quorum intact.
- Never bootstrap blindly.
pc.bootstrap=YESforces a partition to become Primary. If you run it on the wrong (stale) node, or on both partitions, you create real split-brain with divergent data that is painful to merge. Bootstrap only the verified-newest node, and only when the other side is confirmed down.
Sibling cards
| Card | Why pair it with this alert | What the combination tells you |
|---|---|---|
| Galera Cluster Status | The continuous status gauge behind this alert. | The gauge shows current Primary/Non-Primary state; this alert captures the transition event and its timing. |
| Galera Cluster Size | The live node count. | Confirms whether a node was lost and whether the remaining nodes still hold a majority. |
| Galera Flow Control Paused % | Detects a slow node throttling the cluster. | A node falling behind on flow control often precedes it being evicted and triggering this alert. |
| Failover Readiness | Whether the cluster can survive the next failure. | After one node is lost, this tells you if you can afford to lose another before total write-block. |
| Active Async Replicas | Async replicas attached to the cluster. | During quorum loss, an async replica may be the only readable copy of recent writes. |
| Last Successful Backup (hours ago) | Your recovery floor. | If recovery from split-brain goes wrong, the backup age is your worst-case data loss. |
| InnoDB / XtraDB Buffer Pool Hit Rate % | Per-node engine health. | After a node rejoins via SST, a cold buffer pool explains temporary slow reads. |
| MariaDB Health Score | The composite roll-up. | A quorum event collapses the composite immediately; it is the heaviest single driver. |
Reconciling against the source
Where to look in MariaDB / Galera’s own tooling:Why our view may legitimately differ from a single node’sSHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';for Primary / Non-Primary / Disconnected.SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';for the current node count in the Primary Component.SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';for this node’s join state (Synced, Donor/Desynced, Joiner, Initialized).SHOW GLOBAL STATUS LIKE 'wsrep_ready';(ONmeans this node will accept queries).SHOW GLOBAL STATUS LIKE 'wsrep_incoming_addresses';for the member list, and the MariaDB error log forWSREPstate-transition messages.
SHOW STATUS:
| Reason | Direction | Why |
|---|---|---|
| Per-node viewpoint | Different status per node | Each node reports its own partition’s view. During a split, the majority side reads Primary and the minority side reads Non-Primary at the same instant. Our card aggregates across the connected nodes. |
| Sampling vs transition | Brief lag | A node passing through Joiner/Donor states during a state transfer is a transient that a single manual SHOW STATUS might catch mid-flight. |
| Which node the connector polls | Reachability | If the connector can only reach the partition that went Non-Primary, the cluster as a whole may actually be fine on the other side. Cross-check from another node. |
| garbd (arbitrator) presence | Size off by one | A Galera Arbitrator counts toward quorum votes but does not store data; wsrep_cluster_size includes it, which can surprise you when reconciling node inventory. |
SHOW STATUS. SkySQL surfaces Galera topology and node health in its console; on self-managed clusters the MariaDB error log and wsrep_notify_cmd hooks are the authoritative record of membership changes. There is no equivalent in single-node RDS for MariaDB, which uses async replicas rather than Galera; quorum concepts do not apply there.
Known limitations / FAQs
Q: One node shows Non-Primary but the others show Primary. Is the cluster down? No. That is the normal, healthy outcome of a network partition: the majority side stays Primary and keeps serving, and the minority side correctly goes Non-Primary to avoid split-brain. The cluster is degraded (less redundancy) but writes continue on the majority. The minority node will rejoin automatically when connectivity returns, usually via IST. The alert still fires because you have lost a node and should restore it. Q: All nodes show Non-Primary. What now? That means no partition holds a majority (an even split, or too many nodes lost at once), so the whole cluster has gone read-only to protect consistency. First, confirm which node has the most recent committed writeset (wsrep_last_committed). Then bootstrap only that node with SET GLOBAL wsrep_provider_options='pc.bootstrap=YES';, verify it goes Primary, and let the others rejoin. Never bootstrap two partitions, and never bootstrap a stale node, or you create real split-brain.
Q: Why does Galera block writes everywhere instead of just on the failed node?
Because synchronous multi-master replication guarantees all nodes agree on every committed transaction. If a minority partition kept accepting writes, it would diverge from the majority, and merging divergent histories is effectively impossible without data loss. Blocking writes on quorum loss is the price of that strong consistency guarantee. It is a deliberate, correct design choice.
Q: We run only two Galera nodes. Why is that risky?
A two-node cluster has no tiebreaker. A clean 1/1 split leaves neither side with a majority, so both go Non-Primary and all writes stop, even though both nodes are alive. Add a third node, or deploy a lightweight Galera Arbitrator (garbd) as a voting-only member, so a single failure leaves a clear majority. Odd vote counts are the rule.
Q: A node disappeared but cluster_size only dropped by one and status stayed Primary. Why did the alert still fire?
Losing a node is itself an alert condition (cluster_size dropped), independent of quorum. Even when the survivors keep quorum and stay Primary, you have lost redundancy: you are now one failure away from trouble. The alert prompts you to investigate and restore the missing node. Check Failover Readiness to see how much margin you have left.
Q: After a node rejoins, queries on it are slow. Is something wrong?
Usually not. A node that rejoins via SST (full State Snapshot Transfer) starts with a cold InnoDB buffer pool, so its first reads hit disk until the cache warms. Watch InnoDB / XtraDB Buffer Pool Hit Rate % recover over a few minutes. If you can, keep traffic off a freshly rejoined node until it reports Synced and the hit rate climbs back to normal.