Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Cloud and platforms

Kubernetes monitoring: the indicators to track, from cluster to application

In brief

Kubernetes monitoring is the continuous tracking of the state and consumption of a cluster at four levels: control plane, nodes, workloads and applications. Management Square details here the indicators to track at each level, how to read them and what none of them tells you about the other levels.

Page contents

Kubernetes monitoring is the continuous tracking of the state and consumption of a cluster at four levels: the control plane, the nodes, the workloads and the applications they run. It answers three questions. Can the cluster still accommodate and move workloads? Are those workloads running in the requested state? Are users being served?

The difficulty lies in the nature of the orchestrator. Kubernetes corrects some failures by itself: it restarts a container, reschedules a pod, adds a replica. A fault can therefore recur for a long time without being noticed. Nor does any level tell you about the others. A healthy cluster can host an application in error, and an application can respond while the cluster can no longer deploy anything.

Kubernetes monitoring, level by level

Level What is measured Where the measurement comes from What the figure does not tell you
Cluster (control plane) API server latency and errors, etcd health, pods awaiting scheduling Metrics exposed by the control plane components Whether the applications respond
Nodes Declared state, memory and disk pressure, reserved and consumed capacity Kubelet, system metrics exporter Whether the hosted pods are working
Workloads Available replicas, restarts, probes, consumption relative to requests and limits kube-state-metrics, cAdvisor built into the kubelet Whether the service delivered is correct
Application Throughput, error rate, response time, saturation Traffic entry point, code instrumentation Where the cause lies

Kubernetes components expose their metrics in the Prometheus format. This collection tool, combined with a dashboard tool such as Grafana, is therefore found in many set-ups. The choice of tooling matters less than the choice of indicators.

This breakdown does not reopen the debate between the two approaches, covered in our article observability vs monitoring. It sticks to what needs watching in a cluster, level by level.

The cluster: a control plane that responds and decides

The control plane receives all requests (API server), stores the state of the cluster (etcd) and decides where pods are placed (scheduler). Five measurements describe it: the latency and error rate of requests sent to the API server, the write latency and size of the etcd database, the frequency of leader changes, the number of pods the scheduler fails to place, and certificate expiry.

A slow control plane does not cut the service: applications already in place generally keep responding. On the other hand, nothing gets repaired, deployed or adapted to load at the expected pace any more. The effect appears elsewhere and later, which makes the cause hard to trace back without these measurements.

On a managed service, the provider operates the control plane and exposes only part of its metrics. The latency and errors observed from your own deployment tools then remain the most faithful measurement of what you experience.

The nodes: declared state, pressure and reserved capacity

Each node publishes conditions: ready or not, under memory pressure, under disk pressure, short of process identifiers. To these are added the usual server measurements (processor, memory, disk, network) and one measurement specific to Kubernetes: the share of allocatable capacity already reserved by the pods’ resource requests.

This last one is read before the others. The scheduler places a pod according to the resources it requests, without looking at the node’s actual consumption. A node whose processor is lightly used can therefore remain closed to any new pod that declares a request, because its capacity is fully reserved. Conversely, underestimated requests let pods pile up on a node that saturates.

An average calculated across the cluster hides the node in difficulty. The reading is done by node group, with the least favourable value. Nodes, moreover, are replaced: a useful alert concerns the proportion of unavailable nodes in a group, or the inability to add any, rarely a named machine.

The workloads: the gap between requested state and observed state

Kubernetes works by declaration: you describe a state, it strives to reach it. Monitoring workloads therefore amounts to measuring a gap. An add-on component, kube-state-metrics, translates the state of the cluster’s objects into metrics. cAdvisor, built into the kubelet, provides container consumption.

The indicators that count:

  • the number of available replicas relative to the number requested, for each deployment;
  • the time a pod spends pending, a sign of a lack of capacity or of a placement constraint that cannot be satisfied;
  • the rate of container restarts and their reason, in particular termination for exceeding memory (OOMKilled) and restart looping (CrashLoopBackOff);
  • readiness probe failures, which remove a pod from traffic distribution;
  • consumption relative to requests and limits, and the time during which a container’s processor is throttled by its limit;
  • autoscaling that has reached its maximum number of replicas;
  • scheduled jobs that have failed or were not triggered, and the filling of persistent volumes.

The restart counter is cumulative: it is its slope that raises the alarm. A pod in the Running state is not necessarily ready to receive traffic.

Consumption figures mislead in two ways. Modest average processor usage does not rule out throttling during peaks, which only the throttling measurement reveals. The memory displayed includes cache that the system can free: the value to track is the working set, the one the kubelet uses to decide whether to evict pods. Finally, cluster events, which explain a placement or probe failure, are kept for only one hour by default by the API server. They need to be exported if you want to review them after an incident.

The application: what the user of the service experiences

The first three levels describe the platform. The fourth is the only one that says whether the service is being delivered. It is measured at the traffic entry point and in the code: request throughput, error rate, response time, saturation of the resources the service depends on (queues, database connections). The book Site Reliability Engineering, written by Google engineers, calls them the four golden signals: latency, traffic, errors, saturation. An open standard such as OpenTelemetry is used to produce these measurements consistently from one service to another.

Response times are read in percentiles, for the reasons set out in our article on load testing. Alerts that wake up an on-call engineer are best based on this level: a pod that restarts without any effect on users deserves a ticket the next day. This hierarchy is that of service level objectives, presented in our page SRE and observability.

This level does not give the cause. A rising error rate designates a symptom. The explanation is sought by working back down towards the workloads, the nodes, then the control plane. The indicators at all four levels must therefore carry the same labels (namespace, application, version), otherwise the correlation is done by hand, in the middle of an incident.

Providing these labels, the collection and the alert rules along with the deployment model, rather than leaving each team to rebuild them, is a platform matter. It is the role of the golden paths described in our DevOps and platform engineering offering.

The questions to ask before adding a dashboard

  • Does each level have at least one tracked indicator, and an owner who looks at it?
  • Do urgent alerts concern the service delivered, and the others trends handled during working hours?
  • Does each container declare resource requests, without which reserved capacity means nothing?
  • Do readiness probes check that the application can respond, beyond the mere existence of the process?
  • Are events and metrics kept outside the cluster, so they remain readable when the cluster itself is down?
  • Is the monitoring set-up itself watched from outside the cluster?
  • Is a new service monitored from its first deployment, without any specific work by the team delivering it?

A negative answer designates the first workstream, before any new tool.

FAQ

Frequently asked questions

Another question? Ask us

Which indicators should be tracked first in a Kubernetes cluster?

Start with one indicator per level. For the control plane, the latency and error rate of the API server. For the nodes, the share of allocatable capacity already reserved by resource requests. For the workloads, the gap between requested replicas and available replicas, and the rate of restarts. For the application, the error rate and the response time seen by the user.

What is the difference between metrics-server and kube-state-metrics?

The metrics server (metrics-server) feeds the Kubernetes metrics API, which provides the minimum processor and memory measurements needed for autoscaling and for the kubectl top command. kube-state-metrics is an add-on component that queries the API server and exposes, as metrics, the state of the cluster's objects: replicas, pod phases, restarts. The first serves the orchestrator, the second serves monitoring.

Which tools are used for Kubernetes monitoring?

Kubernetes components expose their metrics in the Prometheus format, which makes this collection tool a frequent foundation, combined with a dashboard tool such as Grafana. OpenTelemetry is used to instrument applications consistently. Cloud providers and software vendors offer equivalent services. The tool weighs less on the outcome than the choice of indicators and alert thresholds.

Does a managed Kubernetes service remove the need to monitor the cluster?

No. The provider operates the control plane, but the nodes you configure, the workloads and the applications remain your responsibility, as do the resource requests and limits of each container. Control plane metrics are moreover only partly exposed: the latency and errors observed from your own deployment tools still need to be tracked.

Should you alert on pod processor and memory usage?

Rarely as an emergency. High consumption is not an outage as long as the service responds, and Kubernetes absorbs some of the variations through scaling. Immediate alerts are best focused on the service delivered: errors, response time, unavailability. Consumption is tracked as a trend, relative to requests and limits, to adjust sizing during working hours.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.