About the Author
Richard Lingsch
Enterprise Infrastructure Strategy, Nubius Solutions
Richard has spent 30+ years in infrastructure, hosting, cloud, and application delivery, from IT consulting at Deloitte to co-founding eApps Hosting, where he led the shift from Domino and Java/Tomcat hosting to Xen, KVM, and enterprise OpenNebula. He works with midmarket companies reassessing VMware and virtualization economics, helping them segment workloads and migrate only where the business case holds.
A Percona XtraDB Cluster can appear healthy at the server level while refusing writes because it has lost its PRIMARY component. MySQL may still be listening and accepting connections, but Galera will protect consistency by preventing writes until quorum or authoritative cluster state is restored.
This is not a normal database outage. A standalone MySQL server that fails, fails visibly. A Galera-based cluster does something more unsettling: it protects your data by refusing to serve it. That is correct behaviour. It is also the reason recovery requires understanding what the cluster is protecting against, because the wrong recovery command executed under pressure will happily discard committed transactions.
This is a practitioner’s guide to Percona XtraDB Cluster recovery, written for the person who has to type the commands.
Understand What Quorum Means Before You Touch Anything
Percona XtraDB Cluster uses Galera replication, which is synchronous and certification-based. A write is not committed until it has been replicated to and certified by the cluster. This gives you consistency guarantees that asynchronous replication cannot, and it comes with a hard requirement: the cluster must know it holds a majority.
If a three-node cluster splits so that one node is isolated and two remain together, the pair has quorum and continues serving. The isolated node knows it is in a minority and refuses writes. This is the system working exactly as designed. The isolated node cannot know whether the other two are still serving traffic, and if it accepted writes it would create a divergence that could not be reconciled.
If the split is even, for example a two-node cluster or a four-node cluster splitting two and two, neither side has a majority and both refuse writes. This is why Percona recommends avoiding two-node and other even-numbered cluster designs unless an additional voting member such as an arbitrator is used to preserve quorum.
The first diagnostic question in any recovery is therefore not what broke. It is which nodes can see each other, and does any group among them constitute a majority of the last known cluster size.
The Status Variables That Tell You Where You Are
Before running any recovery procedure, gather state from every node that is reachable. The relevant variables are few and each answers a specific question.
The cluster size variable tells you how many nodes this node believes are in its component. The cluster status variable tells you whether that component is primary. The local state comment tells you what this node is doing: synced, donor, joining, or something less encouraging. The connected and ready variables tell you whether the node is participating at all.
The critical one is the last committed sequence number. Every node tracks the position of the last transaction it applied. In a recovery scenario, this number is what determines which node holds the most complete data, and therefore which node should become the basis for rebuilding the cluster.
Gather these from every node before deciding anything. A recovery decision made from the state of one node is a guess.
Scenario One: A Single Node Is Out
The most common situation and the least dramatic. Two of three nodes are healthy and serving. One is down or non-primary.
Do not touch the healthy pair. They have quorum, they are serving traffic, and any intervention risks the thing you are trying to prevent.
Start the failed node normally. It will attempt to rejoin the cluster and request a state transfer. If its local state is close enough to the cluster’s current position, and if the write-set cache on a donor node still contains the intervening transactions, it will receive an incremental state transfer and catch up in seconds.
If the gap is too large, the node will require a full State Snapshot Transfer, or SST, which copies a consistent dataset from a donor. The operational impact depends on the SST method. Percona’s default xtrabackup-v2 method is designed to keep the donor operational, although the transfer still consumes storage, network, and system resources. Donor selection should therefore take current workload and capacity into account.
The variable worth tuning in advance is the write-set cache size. It determines how far behind a node can fall and still recover incrementally. Sized too small, every brief node outage becomes a full snapshot transfer. Sized generously against your write volume and your typical maintenance window, most node restarts become non-events.
Scenario Two: The Cluster Has No Primary Component
Every node reports non-primary. Nothing accepts writes. This is a network partition that left no majority, or a sequence of failures that took the cluster below quorum.
The recovery here requires a human decision, because the cluster cannot make it safely on its own.
If you know with confidence that the other nodes are genuinely down and not merely unreachable, you can instruct one component to consider itself primary. The bootstrap operation does this. It tells a node to stop waiting for a majority and to declare its component authoritative.
The danger is exact and worth stating plainly. If you bootstrap a component while another component elsewhere is also serving writes, you have created two independent clusters that both believe they are authoritative. They will both accept writes. Those writes will diverge. There is no automated reconciliation for this. Someone will spend days comparing datasets and choosing which transactions to sacrifice.
So before bootstrapping, verify. Not assume. Verify that the other nodes are actually down, from a vantage point that is not dependent on the same network path that failed.
If manual bootstrap is required, first verify the state of every node that participated in the cluster. The bootstrap candidate should be the trusted node with the most advanced recovered position. A higher sequence number alone is not sufficient if another node is unreachable or the candidate’s state may be incomplete or corrupt.Once the bootstrapped node is primary, start the remaining nodes normally. They will join and receive state transfers from the primary.
Scenario Three: Every Node Is Down
Total cluster shutdown, whether from a power event, a datacentre issue, or a cascading failure.
After an unclean shutdown, determine the recovered position on every node using mysqld –wsrep-recover. Compare the recovered UUID and sequence number from each member and verify that the candidate node’s state is trustworthy. The most advanced verified node is then the appropriate bootstrap candidate. After it becomes PRIMARY and SYNCED, start the remaining nodes normally.
The state file on disk holds a safe-to-bootstrap flag. If a node was shut down cleanly as the last member of the cluster, this flag is set on that node, and it is the correct one to bootstrap. If no node has it set, which is the usual case after an unclean shutdown, you fall back to comparing sequence numbers.
Do not set safe_to_bootstrap: 1 on an arbitrary node simply because it is reachable. After an unclean shutdown, Percona’s documented recovery procedure is to recover the position on every former cluster member, identify and validate the most advanced node, and only then set safe_to_bootstrap: 1 on that chosen node before bootstrapping the cluster.
Scenario Four: A Node Refuses to Rejoin
The node starts, attempts a state transfer, and fails. Repeatedly.
Common causes include SST authentication or configuration problems, blocked cluster or transfer ports, insufficient disk space, incorrect node addresses, security controls such as SELinux or AppArmor, and incompatible software or SST configurations. Check both the joiner and donor logs before changing the node state or retrying the transfer.
Check the error log on both the joiner and the donor. State transfer failures log on both sides, and the donor’s log frequently carries the useful message while the joiner’s log carries a generic failure.
If the node’s local state is corrupt beyond repair, the deterministic fix is to remove its data directory entirely and let it join as a fresh node receiving a full snapshot. This is slow on a large dataset but it is reliable, and it eliminates an entire class of subtle corruption that intermittent rejoin failures often indicate.
The Failure Modes Worth Preventing Instead
Recovery procedures are necessary. Needing them frequently is a design problem.
Node count should be odd. Three is the minimum that survives a single failure with quorum intact. Five gives you two failures. Two gives you nothing but false confidence.
If three full data-bearing nodes are not practical, a Galera arbitrator can provide an additional vote without storing the database itself. That can allow a two-node deployment to retain quorum after the loss of one data node, although the arbitrator remains another component that must be placed and operated deliberately.Network stability matters more than in an asynchronous topology, because Galera is synchronous. A network with intermittent packet loss will produce cluster evictions that look like node failures. Tune the group communication timeouts against your actual network characteristics rather than leaving defaults that assume a datacentre LAN, particularly if nodes are separated by any distance.
Flow control is worth understanding before it surprises you. If one node cannot apply write-sets as fast as they arrive, it signals the cluster to slow down, and the whole cluster’s write throughput drops to the speed of the slowest node. A node on inferior storage does not just perform badly. It makes every node perform badly. Monitor flow control pause time as a first-class metric.
Large transactions are a recurring source of trouble. Galera certifies write-sets, and a very large transaction produces a very large write-set that must be transmitted, certified, and applied everywhere. A bulk delete of millions of rows in one statement can stall the cluster. Batch them.
Schema changes need a deliberate method. The default blocks the cluster for the duration of the change. Rolling schema upgrade takes each node out in turn, which is safer but requires that the change is backwards compatible. Online schema change tools handle the harder cases. Choosing the wrong method on a large table during business hours is a self-inflicted outage.
Backups Are Not the Cluster
A Galera cluster replicates. Replication is not backup. A dropped table replicates to every node in milliseconds, perfectly and consistently.
You need real backups, taken from a node that is not serving production traffic, with a tested restore path. The emphasis is on tested. A backup that has never been restored into a working database is an assumption.
Test the restore quarterly, into a real target, with a real application pointed at it. Measure how long it takes. That number is your actual recovery time objective regardless of what the document says.
Backups, disaster recovery configuration, and monitoring across database and proxy layers are core to what we deliver under Nubius Managed AppOps, where MySQL, MariaDB, Percona XtraDB, MongoDB, and ProxySQL are managed from a sysadmin perspective with replication, tuning, and performance monitoring included rather than optional.
The Runbook Nobody Writes Until After the Incident
Write it before. It needs to answer, without ambiguity: how to gather state from every node, how to determine which node has the highest sequence number, how to decide whether to bootstrap, who is authorised to make that decision, and how to verify the other nodes are genuinely down rather than merely unreachable.
That last one is the crux. Every catastrophic Galera recovery starts with someone bootstrapping a node because it was the one they could reach.
Rehearse it. A runbook that has been read is not a runbook that works. Take a node out in a staging cluster and follow the document. You will find the gaps, and finding them on a Wednesday afternoon costs nothing.
The gap between knowing the platform and having recovered it under pressure is real and it is expensive to close internally, which is the broader argument in our piece on the cloud skills gap.
Where Nubius Fits
Percona XtraDB Cluster is explicitly within the database stack managed by Nubius Managed AppOps, along with the ProxySQL and HAProxy layers that route traffic to it and the Linux operating systems underneath. We handle replication, tuning, monitoring, patching, backup schedules, and disaster recovery policy, with 24/7 monitoring and alerting rather than reactive response after the alert your team missed.
For complex one-off situations such as cluster upgrades or service recovery, Nubius OpsAssist AnyCloud provides on-demand engineering support across whichever platform the cluster runs on, including advanced troubleshooting for the environment issues that cause cluster instability in the first place.
If you have a Percona cluster that is either currently broken or quietly one node failure away from being broken, talk to our team.