Kubernetes High Availability for Production: Essential Considerations

Worker node HA strategies

Worker node high availability forms the foundation of a resilient Kubernetes cluster. While the control plane manages the cluster state, worker nodes execute the actual workloads. A comprehensive worker node HA strategy must address both planned maintenance and unexpected failures while ensuring application availability.

Node pools and zone distribution

Node pools represent groups of worker nodes with similar configurations and purposes. In a high-availability setup, nodes should be distributed across multiple availability zones to protect against zone-level failures. This distribution isn’t just about spreading nodes—it’s about maintaining enough capacity in each zone to handle workload redistribution during failures.

The key here is capacity planning. Each zone should maintain enough spare capacity to handle the redistribution of workloads from a failed zone. For example, if you’re running at 66% capacity across three zones, losing one zone would allow the remaining two zones to absorb the redistributed workloads without oversubscription.

# Example node pool configuration in cloud provider
apiVersion: compute.cloud.example/v1
kind: NodePool
metadata:
  name: production-workload
spec:
  replicas: 6  # 2 nodes per zone
  distribution:
    zones: 
      - zone-1
      - zone-2
      - zone-3
  nodeConfig:
    machineType: e2-standard-4
    labels:
      pool: production
      environment: prod

Pod disruption budgets

Pod disruption budgets (PDBs) are Kubernetes’ way of ensuring application availability during voluntary disruptions. The concept revolves around maintaining a minimum number of healthy pods during operations like node drains or cluster upgrades.

PDBs work in conjunction with the Kubernetes scheduler and respect pod topology spread constraints. They’re particularly important for stateful applications where pod placement can impact performance and reliability. The PDB mechanism helps maintain application SLAs by preventing too many pods from being unavailable simultaneously.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: critical-service-pdb
spec:
  minAvailable: 2    # or use maxUnavailable: 1
  selector:
    matchLabels:
      app: critical-service

Recovery strategies during node failures

Node failure recovery involves multiple mechanisms working together. The process begins with failure detection, moves through workload evacuation, and ends with node replacement, if necessary. This process must be automated to minimize service disruption.

The node lifecycle in failure scenarios follows several stages:

1. Detection: The node problem detector identifies an issue.
2. Isolation: Kubernetes marks the node as unschedulable.
3. Evacuation: Pods are rescheduled to healthy nodes.
4. Recovery: Automated repair or replacement occurs.

Each layer of the worker node HA strategy builds upon the others to create a resilient infrastructure capable of handling various failure scenarios while maintaining application availability. Regular testing of these mechanisms, particularly through chaos engineering practices, helps validate the effectiveness of your HA implementation.

Application-level high availability

HA at the application level involves a complex interplay among infrastructure, application design, and operational patterns. Unlike infrastructure HA, which focuses on platform resilience, application HA demands a deep understanding of workload characteristics, data consistency, and business continuity needs. The fundamental goal is to maintain service availability despite component failures, network issues, or planned maintenance.

Stateless vs. stateful considerations

The distinction between stateless and stateful applications forms the base of HA strategy. Stateless applications follow the principles of immutable infrastructure: They can be created, destroyed, or replaced without data loss concerns. Each request contains all the information needed to process it, making horizontal scaling and failover straightforward.

Stateful applications, however, maintain data that must persist across sessions and survive pod restarts. This persistence requirement introduces complexity in several areas:

• Data synchronization between replicas
• Consistent ordering of operations
• Backup and recovery procedures

Pod anti-affinity rules

Anti-affinity represents a critical concept in Kubernetes scheduling logic. It enforces constraints that prevent pods from colocating on the same infrastructure components, thereby improving fault tolerance. There are two types of anti-affinity:

• Required anti-affinity: Hard rules that must be satisfied
• Preferred anti-affinity: Soft rules that the scheduler attempts to satisfy

The scheduler evaluates these rules against topology keys: labels that represent failure domains like nodes, racks, or availability zones. This mechanism ensures proper workload distribution across your infrastructure’s failure domains.

Deploying an HA database

Let’s examine a complete example of deploying a highly available PostgreSQL database on Kubernetes using AWS infrastructure. 

The below StatefulSet configuration demonstrates several critical HA concepts: 

• Running multiple replicas across availability zones
• Configuring proper health checks
• Setting up streaming replication
• Managing persistent storage

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: postgres
spec:
 # Headless service for stable DNS names for each pod
 serviceName: postgres
 # Three replicas: one primary and two replicas spread across AZs
 replicas: 3
 selector:
   matchLabels:
     app: postgres
 template:
   metadata:
     labels:
       app: postgres
   spec:
     # Used for AWS IAM roles to access AWS services like KMS, S3 etc.
     serviceAccountName: postgres-sa  
     # Required for postgres user permissions on mounted volumes
     securityContext:
       fsGroup: 999
       
     # Init container runs before main container to handle primary/replica setup
     initContainers:
     - name: init-postgres
       image: postgres:16
       command:
       - /bin/bash
       - -c
       - |
         # postgres-0 becomes primary, others become replicas
         if [[ $(hostname) == postgres-0 ]]; then
           echo "Initializing primary node"
         else
           echo "Initializing replica node"
           # Create replica using streaming replication
           pg_basebackup -h postgres-0.postgres -D /var/lib/postgresql/data/pgdata -U replicator -v -P --wal-method=stream
         fi
       env:
       - name: PGPASSWORD
         valueFrom:
           secretKeyRef:
             name: postgres-secret
             key: replication-password
       volumeMounts:
       - name: postgres-data
         mountPath: /var/lib/postgresql/data

     containers:
     - name: postgres
       image: postgres:16
       env:
       - name: POSTGRES_PASSWORD
         valueFrom:
           secretKeyRef:
             name: postgres-secret
             key: password
       # Specify PGDATA path for PostgreSQL data directory
       - name: PGDATA
         value: /var/lib/postgresql/data/pgdata
       ports:
       - containerPort: 5432
       
       # Resource requests and limits suitable for production workload
       resources:
         requests:
           cpu: "2"
           memory: "8Gi"
         limits:
           cpu: "4"
           memory: "16Gi"
           
       # Health checks for container lifecycle management
       readinessProbe:
         exec:
           command: ["pg_isready", "-U", "postgres", "-h", "localhost"]
         initialDelaySeconds: 5
         periodSeconds: 10
       livenessProbe:
         exec:
           command: ["pg_isready", "-U", "postgres", "-h", "localhost"]
         initialDelaySeconds: 30
         periodSeconds: 10
         
       # Volume mounts for data and configuration
       volumeMounts:
       - name: postgres-data
         mountPath: /var/lib/postgresql/data
       - name: postgres-config
         mountPath: /etc/postgresql/postgresql.conf
         subPath: postgresql.conf

     # Ensure pods are distributed across AZs
     topologySpreadConstraints:
     - maxSkew: 1
       topologyKey: topology.kubernetes.io/zone
       whenUnsatisfied: DoNotSchedule
       labelSelector:
         matchLabels:
           app: postgres

     # Prevent multiple pods on the same node
     affinity:
       podAntiAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
         - labelSelector:
             matchLabels:
               app: postgres
           topologyKey: kubernetes.io/hostname

 # EBS volume configuration for each pod
 volumeClaimTemplates:
 - metadata:
     name: postgres-data
   spec:
     accessModes: [ "ReadWriteOnce" ]
     # Using our custom storage class for EBS gp3
     storageClassName: ebs-gp3-postgres
     resources:
       requests:
         storage: 100Gi

Data protection for HA

Data protection in Kubernetes extends beyond simple backup and restore operations. It requires a comprehensive strategy that ensures data consistency, minimal RPO/RTO, and seamless integration with application lifecycle management. While Kubernetes provides basic volume snapshots, enterprise environments demand more sophisticated solutions for true high availability.

Backup strategies

Traditional backup methods often prove inadequate in Kubernetes environments due to the inherent complexity of container ephemerality and state distribution. Applications running on Kubernetes typically span multiple components, making it essential to maintain consistency across these distributed elements during backup operations. Cross-namespace dependencies and resource relationships further complicate the backup process, requiring sophisticated orchestration to ensure complete and consistent captures.

For example, Trilio‘s approach addresses these challenges through application-aware backups. This method considers the entire application stack as a single unit, ensuring that all components are captured in a consistent state. The system intelligently handles dependencies between different parts of the application, maintaining referential integrity throughout the backup process.

Implementing automated backups

Automated backup implementation requires consideration of scheduling patterns and resource impact. Full backups provide complete state capture but consume significant resources, while incremental backups offer efficiency at the cost of slightly more complex recovery procedures. The key lies in balancing these approaches based on application activity patterns and business requirements.

Recovery time optimization

Recovery time optimization involves balancing speed with reliability. Parallel restore operations can significantly reduce recovery time but must be managed carefully to prevent resource contention.

The recovery process must be thoroughly tested and validated. This includes verifying application health post-recovery, ensuring data consistency, and validating service connectivity. Performance validation ensures that the recovered application meets its operational requirements. Regular recovery testing helps identify potential issues before they impact real recovery scenarios.

Testing and validation

Testing high availability isn’t just about technical validation—it’s about ensuring business continuity. Real-world production environments rarely fail in expected ways, and the testing approach must reflect this reality.

Chaos engineering practices

Start small and build confidence gradually. Begin with testing in non-peak hours and in isolated environments. For example, instead of immediately failing an entire region, start by testing how your applications handle a single pod failure, then a node failure, and then gradually increase the scope. Document every unexpected behavior—these are your most valuable insights.

Monitoring critical components

Focus on monitoring what matters to your business, not just what’s easy to measure. For example, if you’re running an ecommerce platform, monitoring pod CPU usage is important, but tracking successful checkout completions is critical. Build your monitoring strategy around your customer’s journey.

A practical approach used by successful organizations is to create monitoring dashboards that business stakeholders can understand. While technical metrics are important, being able to show the direct impact on business operations makes conversations about reliability investments much more productive.

Key metrics and alert rules

The most effective alerts are those that demand action. Many organizations suffer from alert fatigue because they alert on everything they can measure. Instead, focus on symptoms, not causes. For example, don’t alert on high CPU usage—alert on degraded customer response times.

Here’s an example of real-world prioritization:

• Critical: Customer-impacting issues (service outages, failed transactions)
• High: Issues that could become customer-impacting (degraded performance, reduced redundancy)
• Medium: System health indicators (resource usage trending up)
• Low: Informational (successful automated recoveries)

Multi-region failover testing

The hardest part of multi-region failover isn’t the technical implementation—it’s the human element. Regular failover testing often reveals that team members aren’t familiar with the procedures, documentation is outdated, or critical access credentials have expired.

Run semi-annual “game days” where teams practice failover procedures. Include not just the technical team but also customer support, business stakeholders, and communication teams. The goal isn’t just to test the technology but to practice the entire organizational response.