Kubernetes Monitoring Best Practices: A Comprehensive Guide

Let’s break down this query piece by piece, so we know what we are looking at:

• node_cpu_seconds_total{mode="idle"} measures the total time in seconds the CPU spends idle
• rate(...[5m]) calculates the per-second rate of change over a 5-minute window.
• avg by (instance) calculates the average per instance (node).
• … * 100 converts the number to a percentage.
• 100 - … subtracts the previous number (percentage idle) from 100%, giving the percentage used.

This example shows how to track CPU data over time. To get more information, navigate to the Graph or Explain tabs from this window.

Prometheus metrics for the four golden signals

The next step is to build on this demo by creating Prometheus queries that align with the four golden signals. Here are some relevant Prometheus metrics for the four golden signals:

• Latency:
• • http_request_duration_seconds_bucket
  • apiserver_request_duration_seconds_bucket
  • istio_request_duration_milliseconds

• Traffic:
• • http_requests_total[5m]
  • container_network_receive_bytes_total
  • apiserver_request_total

• Errors:
• • http_requests_total{status=~”5..”}[5m]
  • Http_requests_total[5m]
  • kube_pod_container_status_restarts_total

• Saturation:
• • node_memory_MemTotal_bytes
  • node_memory_MemAvailable_bytes
  • node_memory_MemTotal_bytes
  • kubelet_volume_stats_used_bytes

Prometheus collects an overwhelming number of metrics. Visiting http://localhost:9090/api/v1/targets/metadata returns JSON-formatted metadata about all of the metrics. This illustrates how many metrics there are; you would not be expected to review this data for available metrics.

To learn more about the metrics, you can begin typing in the expression box and have it autocomplete based on available metrics, or you can review the exporter’s documentation. 

Each exporter (application that exports Prometheus data) has its own document, including Node Exporter, Kube-State-Metrics, cAdvisor (used by kubelet), and etcd. Once you know the metrics you’re interested in, you can further use PromQL (Prometheus Query Language) to write queries, create dashboards, and set up alerts based on this data.

Plan how to handle alerts

Once you’ve collected your key metrics, the next step is to decide how to act on them, and this is where alerts come in. The key idea here is that clarity is everything. Each alert should identify four things:

• What is happening
• What action to take
• Who should get the alert
• How should they be notified 

The first question to ask is: “Is this alert relevant?” Without answering this question, teams can be flooded with so much noise that important alerts are missed, a phenomenon known as alert fatigue that leads to slow response times and missed incidents. 

Another way to look at an alert’s relevance is to consider what would happen if the alert condition is missed or persists for an extended period. Would it be not noticeable, lead to performance degradation, or cause an outright outage?

Alert categorization

To help prioritize alerts, they are often categorized into severity levels, usually SEV1 (highest) to SEV4 (lowest). While the criteria for each level may differ between teams and organizations, the main idea is that SEV1 alerts are critical issues needing immediate action, while SEV4 issues are the lowest priority and may not require immediate action.

The table below describes these categories and provides examples of each.

You can start by defining alert thresholds, such as when memory exceeds 90% for more than five minutes or when a pod restarts more than three times in five minutes. This process will evolve over time as you learn what normal usage is for your environment. 

Once an alert fires, the next step is to decide what to do. Every alert should have clear and actionable steps. If there is no action to take, then there is no reason for it to be an alert. 

An example of clear steps would be logging on to the server and running a series of predetermined commands to identify which processes use the most memory. Based on what is discovered, processes could be killed, resources scaled up, etc. For Kubernetes specifically, a useful alert would be if a pod restarts multiple times over a short period. This means a pod is trying to restart but has a persistent issue preventing it from running. The response could be to review logs, increase resource limits, or even cordon off the node for deeper investigation. 

The next thing to identify is who should get the alert and how they should be notified. Each alert should have an owner: a team or individual directly responsible for resolving it. For example, a manager may be responsible for the system’s overall health, and sending alerts to the engineering team would facilitate a quicker response than sending them to the manager, who will ultimately send them to the engineering team.

Finally, consider how the owner should be notified. There are many tools for informing a team or individuals; the key is to decide what works for your situation. The notification could be an email, Slack message, PagerDuty notification, etc.