Elasticsearch clusters have a habit of looking healthy right up until the moment a node runs out of heap and starts dropping shards, which is exactly why monitoring Elasticsearch and log aggregation clusters needs to go well beyond checking if the cluster status is green. A green cluster status only means every shard has an active copy – it says nothing about query latency, indexing backpressure, or a node that’s three GC pauses away from falling out of the cluster.
Why cluster status alone isn’t enough
The three-state model (green, yellow, red) is useful as a headline number, but it’s a lagging indicator. By the time status flips to yellow, you’ve usually already lost a replica shard, and by red, you’ve lost a primary. A cluster running on 92% heap usage across every data node will report green right up until the first big aggregation query triggers a circuit breaker exception and takes down a coordinating node.
This is the myth worth busting directly: a green cluster does not mean a healthy cluster. Plenty of production ELK stacks run “green” for months while indexing latency creeps from 50ms to 800ms per bulk request, unnoticed because nobody wired up latency or JVM metrics alongside the basic health check.
The metrics that actually predict trouble
A handful of signals catch problems before they become outages:
JVM heap usage per node – anything sustained above 75% is a warning sign; Elasticsearch’s default heap sizing (50% of available RAM, capped around 30-32GB to keep compressed oops enabled) means heap pressure often precedes an OOM-triggered node crash by hours, not seconds.
Field data and query cache eviction rates – a rising eviction rate on the field data cache usually means someone added an aggregation on a high-cardinality field, like a raw IP or a UUID, without a keyword mapping optimized for it.
Indexing and search thread pool rejections – these show up in `_cat/thread_pool` and are the earliest hard signal that the cluster can’t keep up with incoming write or query volume. A rejection count that goes from 0 to climbing over ten minutes is a much better alert trigger than waiting for status to turn yellow.
Merge and refresh activity – on log-heavy indices with a 1-second refresh interval (Elasticsearch’s default), excessive segment merging can saturate disk I/O, especially on spinning disks or under-provisioned cloud volumes.
Unassigned shards – tracked separately from the color-coded status, this number tells you exactly how many shards are homeless and why, via `_cluster/allocation/explain`.
A realistic scenario: the log pipeline that outgrew its cluster
A common pattern in shops running the ELK stack for centralized logging: three data nodes handle application logs comfortably for a year, then a new microservice gets added that logs at DEBUG level in production. Daily index size jumps from 40GB to 180GB almost overnight. Indexing threads start queuing, bulk rejections appear in the logs, and Kibana dashboards get sluggish before anyone notices – because the only alert configured was on cluster status, and status stayed green the entire time since shards were still allocated, just slow.
The fix in that scenario is usually a mix of things: tightening index lifecycle management (ILM) policies to roll over indices by size rather than just by day, reducing shard count per index (a common mistake is defaulting to 5 primary shards per index regardless of data volume, which was the old Elasticsearch default before 7.0 changed it to 1), and adding heap and thread pool alerts that would have caught the trend two weeks earlier.
Common mistakes teams make with ES monitoring
Three patterns show up repeatedly in postmortems. First, alerting only on cluster color and ignoring node-level metrics – a single hot node with heap pressure can degrade the whole cluster’s query performance long before status changes. Second, treating shard count as a one-time setup decision instead of revisiting it as index size grows; over-sharded small indices waste heap on shard overhead, while under-sharded large indices create hot spots. Third, monitoring Elasticsearch itself but not the ingest pipeline feeding it – if Logstash or a Fluentd/Fluent Bit forwarder queue backs up, ES metrics look fine while log data is silently delayed by twenty minutes or more, which matters a lot during an incident when someone’s tailing logs expecting real-time data.
Setting up monitoring that catches this early
An experienced SRE typically layers monitoring in three tiers for an ES cluster. At the infrastructure level, track CPU, memory, disk I/O, and network on every node the same way you would any production server, since Elasticsearch is ultimately bound by the same hardware constraints as anything else. At the service level, poll the Elasticsearch APIs directly – `_cluster/health`, `_nodes/stats`, and `_cat/thread_pool?v` – on a 30-60 second interval, since these expose JVM, cache, and thread pool detail that generic host metrics never will. At the pipeline level, watch queue depth and processing lag on whatever sits in front of Elasticsearch, whether that’s Logstash, Fluentd, or a Kafka buffer, because a healthy cluster fed by a backed-up pipeline is still a broken observability stack from the end user’s perspective.
Disk space deserves its own callout here: Elasticsearch’s default behavior under the disk-based shard allocation watermark (85% used triggers no new shard allocation, 90% triggers shard relocation, 95% forces read-only index blocks as of recent 7.x/8.x defaults) means a cluster can go read-only during a traffic spike if nobody’s watching disk trend lines. Pairing disk space monitoring with these watermark thresholds as explicit alert points avoids getting blindsided by a 95% disk fill turning half your indices read-only at 2am.
For root-causing incidents after the fact, correlating ES node metrics against application-side error rates and log volume spikes is far faster with a proper root cause analysis workflow than digging through Kibana’s own diagnostic tools mid-incident, since Kibana’s query performance often degrades right when you need it most.
FAQ
What’s a healthy JVM heap usage level for Elasticsearch nodes?
Sustained usage under 75% is generally safe; regular spikes into the 85-90% range indicate the node needs either more heap (up to the ~30-32GB compressed oops ceiling) or a reduction in shard/query load. Old-generation GC pauses lasting more than a second or two are a stronger warning sign than the raw percentage alone.
How often should shard allocation be checked?
Unassigned shard count should be part of the same polling cycle as cluster health, typically every 30-60 seconds, since shard allocation issues often resolve or worsen quickly depending on whether it’s a transient network blip or a genuine disk watermark problem.
Does a yellow cluster status always mean data loss risk?
Not immediately – yellow means at least one replica shard isn’t allocated, which is common right after a single-node restart in a small cluster and usually self-heals within minutes. It becomes a real risk only if it persists, since a subsequent node failure at that point would mean an actual primary shard loss.
Elasticsearch monitoring done well means treating cluster status as one signal among many, not the whole picture – heap, thread pool rejections, shard allocation, and ingest pipeline lag together tell you what’s actually happening well before a status color changes.