Database administrators running MariaDB in a multi-master Galera setup face a problem that traditional MySQL replication monitoring doesn’t fully cover: a node can be up, accepting connections, and still be silently out of sync with the rest of the cluster. Monitoring MariaDB and Galera Cluster Replication Health means watching a different set of signals than single-master replication – flow control, certification conflicts, and quorum state matter as much as basic uptime checks.
A three-node Galera cluster handling an e-commerce checkout flow is a common deployment pattern. Each node can accept writes, and MySQL-style async replication lag doesn’t apply the same way – Galera uses synchronous (or more precisely, virtually synchronous) certification-based replication. That changes what “healthy” even means, and it’s where a lot of monitoring setups inherited from standalone MySQL or MariaDB fall short.
Why standard replication checks don’t work for Galera
Classic MySQL monitoring leans on Seconds_Behind_Master from SHOW SLAVE STATUS. Galera clusters don’t have a single master, so that metric is meaningless in a multi-master topology. Instead, the cluster exposes its state through wsrep_* status variables, and those are what actually need tracking.
The variable wsrep_local_state tells you whether a node is Synced (4), Donor/Desynced (2), or Joining (1). A node stuck in Joining for more than a few minutes during a State Snapshot Transfer (SST) is a red flag – on a 50GB dataset over gigabit, an SST via rsync typically takes 8-15 minutes; if it’s still running at 40 minutes, something’s wrong, often disk I/O contention on the donor.
wsrep_cluster_size and wsrep_cluster_status are the quorum indicators. A 3-node cluster that drops to size 2 has lost quorum tolerance – one more node failure and the whole cluster goes non-primary and stops accepting writes. This is the scenario that catches teams off guard: everything looks fine because two nodes are still serving traffic, but the cluster is one failure away from a full outage.
The metrics that actually predict trouble
Five wsrep variables carry most of the diagnostic weight in production:
wsrep_flow_control_paused shows the fraction of time a node has spent throttling writes to let slower nodes catch up. Anything consistently above 0.1 (10%) points to a node that can’t keep pace – usually the one with the slowest disks or an underpowered CPU in an otherwise identical hardware set.
wsrep_local_recv_queue_avg reflects how backed up the receive queue is. A sustained value above 1.0 during normal traffic (not just during a backup window) means the node is falling behind cluster-wide writes, which eventually triggers flow control across the whole cluster and slows every node down, not just the lagging one.
wsrep_cert_deps_distance and conflict/abort rates (wsrep_local_cert_failures, wsrep_local_bf_aborts) reveal certification conflicts – rows being written concurrently on different nodes that collide during certification. A spike here after a schema change or a new application deployment usually means someone introduced a hot row pattern, like a single counter row updated from every node simultaneously.
wsrep_cluster_conf_id increments every time cluster membership changes. If this number is climbing every few minutes in a cluster that isn’t intentionally scaling, nodes are flapping in and out – classic symptoms of a flaky network link or a misconfigured firewall timeout between data centers.
Setting up alerting without drowning in noise
An experienced DBA doesn’t alert on every wsrep_cluster_size change – that fires on every planned maintenance restart. Alerts should differentiate expected state transitions (a node going through SST during a scheduled rejoin) from unexpected ones (a node dropping to non-Primary state outside any maintenance window).
Practical thresholds that hold up in most production Galera deployments:
Cluster status not PRIMARY for more than 30 seconds – critical, page immediately, this means split-brain protection has kicked in and writes are being rejected.
wsrep_local_state stuck below 4 for more than 10 minutes outside a known SST window – warning, investigate the donor node’s disk and network load.
wsrep_flow_control_paused above 0.15 sustained for 5+ minutes – warning, check the slowest node’s hardware or query load.
wsrep_cluster_size below configured node count for more than 60 seconds – critical, quorum tolerance is degraded.
Tying these into a broader alerting strategy that avoids alert fatigue matters here – three nodes each independently alerting on the same underlying network blip generates noise that trains on-call engineers to ignore pages. Correlating wsrep alerts with the same event timestamp across nodes, rather than alerting per-node, cuts duplicate pages significantly.
A myth worth retiring
A persistent misconception is that Galera clusters don’t need replication monitoring because “it’s synchronous, so it can’t lag.” That’s not accurate. Galera is certification-based and applies writes asynchronously to the storage engine on non-originating nodes after certification succeeds – there’s still an apply queue, and under heavy write load or with a slow disk subsystem, that queue backs up measurably. wsrep_local_recv_queue existing as a metric at all disproves the “no lag possible” claim. Nodes can and do fall behind; the cluster just handles it through flow control instead of silently drifting, which is a different failure mode, not the absence of one.
Common mistakes worth avoiding
A frequent misstep is monitoring only mysqld process status and port 3306 availability. A node can pass both checks while sitting in a Donor/Desynced state, serving stale reads to any application pointed at it directly instead of through a proxy like ProxySQL or MariaDB MaxScale that’s aware of wsrep state.
Another common error is treating all three nodes as monitoring equals without accounting for the current donor. During an SST, the donor node’s CPU and I/O spike, and its query performance for regular traffic degrades – this is expected, not evidence of hardware failure. Alerting on the wrong signal here leads to a 3am page investigating a “slow node” that’s just doing its job as the SST source.
Underestimating gcache size is a third recurring issue. When gcache (the write-set cache used for Incremental State Transfer, or IST) is too small for the write volume between a node’s downtime and rejoin, Galera falls back to a full SST instead of the much faster IST – turning a 90-second rejoin into a 20-minute one. Monitoring gcache utilization alongside cluster state variables catches this before a routine restart turns into an extended degraded-quorum window.
FAQ
What’s the difference between monitoring MariaDB replication and Galera cluster replication?
Standard MariaDB (and MySQL) replication monitoring tracks a single master-to-slave lag value. Galera is multi-master and synchronous at the certification layer, so monitoring instead centers on wsrep_* status variables covering cluster membership, flow control, and certification conflicts rather than a simple lag counter.
How often should wsrep status variables be polled?
Every 15-30 seconds is typical for production clusters. Polling more frequently than every 10 seconds adds measurable overhead on busy nodes since each SHOW STATUS query has a small but real cost at scale.
Can a Galera cluster keep running with only one node left?
No. A single remaining node has no quorum and Galera will set it to non-Primary state, rejecting writes, unless pc.bootstrap is manually invoked – which should only be done deliberately, since it risks data divergence if other nodes come back online independently.
Galera’s wsrep variables give a clearer picture of cluster health than uptime checks ever will, but only if the specific state transitions – quorum loss, flow control pauses, certification conflicts – are surfaced as distinct alerts rather than folded into a generic “database down” check. Getting that granularity in place before the next node failure, not after, is what separates a 90-second failover from a multi-hour incident.