Kubernetes on OpenStack – Best Practices for High Availability

Deploying K8s cluster with Kubespray

Deploying the K8S cluster

We are now ready to start the deployment of the K8s cluster. Source the OpenStack variables and run the cluster deployment Ansible playbook:

$ source k8s-project-openrc.sh
$ ansible-playbook -i inventory/mycluster/hosts.yaml \
-u ubuntu --become cluster.yml

Now sit back and take a coffee break—the K8s cluster deployment may take up to 30 minutes because the playbook will need to download lots of packages to complete the deployment. 

After successfully completing the Kubespray playbook, you should get the following output without any failed tasks:

The deployment will create a kubectl config on the controller node with cluster authentication information. You can do an SSH connection into the controller node1 and copy the file into your home directory. You can also download the config file remotely into your Ansible controller and then use kubectl to interact with the K8s cluster.

ubuntu@node1:~$ sudo cp /etc/kubernetes/admin.conf .kube/config 

ubuntu@node1:~$ kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   17h   v1.29.5
node2   Ready    control-plane   17h   v1.29.5
node3   Ready    control-plane   17h   v1.29.5
node4   Ready              17h   v1.29.5
node5   Ready              17h   v1.29.5
node6   Ready              17h   v1.29.5

ubuntu@node1:~$ kubectl get pods -n kube-system
NAMESPACE     NAME                                          READY   STATUS    RESTARTS      AGE
kube-system   coredns-69db55dd76-q7qtl                      1/1     Running   0             2m9s
kube-system   coredns-69db55dd76-vtxfv                      1/1     Running   0             15m
kube-system   csi-cinder-nodeplugin-5dv49                   3/3     Running   0             15m
kube-system   csi-cinder-nodeplugin-6bzr2                   3/3     Running   0             15m
kube-system   csi-cinder-nodeplugin-8b9s8                   3/3     Running   0             15m
kube-system   dns-autoscaler-6f4b597d8c-v6h5l               1/1     Running   0             15m
kube-system   kube-apiserver-node1                          1/1     Running   1             19m
kube-system   kube-apiserver-node2                          1/1     Running   1             18m
kube-system   kube-apiserver-node3                          1/1     Running   1             17m
.
.

Doing a test deployment

We can now start deploying applications on top of the K8s cluster. As an example, we will do a deployment of an Nginx web server and check the pod created by the Nginx deployment:

ubuntu@node1:~$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

ubuntu@node1:~$ kubectl get pods -l app=nginx
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7854ff8877-g6b24   1/1     Running   0          13s

Tearing down the cluster (if required)

If, for any reason, we want to tear down the K8s cluster—maybe to start again—we can use the reset.yml playbook of Kubespray to do that. This will remove all the installed packages and remove the K8s cluster configurations:

$ ansible-playbook -i inventory/mycluster/hosts.yaml \
-u ubuntu --become reset.yml

Best practices for running K8s environments

Plan capacity and prepare the environment

Whenever you are doing a production-grade deployment, there are certain prerequisite steps that should be taken before going live. The first one is the planning and the preparation phase. 

Start by figuring out the type of application workload you will be deploying on your cluster. What is the profile of your applications? Are these CPU intensive or memory intensive? What are the I/O requirements? You need to plan your compute resources accordingly and choose the appropriate instance type with the right CPU, RAM, and disk IO specifications. 

Most languages have their own profiling tools or libraries. For Java, you can use tools like YourKit, JProfiler, or VisualVM that can be integrated into applications to collect profiling data. Go applications have pprof, which offers CPU and memory profiling. For Python, cProfile, line_profiler, and Pyinstrument are popular options.

Another thing you need to account for is the resources required for the K8s components themselves (e.g., kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, etc.).

If you are offering persistent volumes, you should have sufficient storage capacity and  the ability to increase this capacity. You also need to define storage classes based on the I/O performance requirement of the applications (HDDs or SSDs). 

Choose the right deployment methods

The right deployment method depends on which ecosystem you are already working with and your team’s skill set. If you have a deeper understanding of Kubernetes internals and networking concepts, you can use kubeadm as the officially developed and maintained tool by the Kubernetes team. You get fine grained control over the cluster configuration and allows greater customization according to your specific needs.

If your team is proficient in automation tools like Ansible and if you want a repeatable, consistent deployment with speed and efficiency then the recommendation is to go with an automated deployment tool as explained above. With this, although you get slightly less flexibility compared to kubeadm, you can reduce manual efforts and chances of errors using IAC tools.

Scale horizontally and/or vertically

As infrastructure grows, you eventually come to a decision point when it comes to scaling up your resources. You have the option of either scaling horizontally (scaling out), which means adding more nodes to your deployment cluster, or scaling vertically (scaling up), which is adding more compute resources to your existing nodes.

When you have stateless applications, you can scale horizontally to handle increased load by distributing the load across multiple instances while also improving fault tolerance and availability. With more nodes, you get better redundancy. A small downside is that on each node there is some compute capacity reserved for running the OS and K8s components, which slightly decreases your overall cluster capacity.

With stateful applications where individual instances need more resources, you can scale vertically for better resource optimization. It is simpler and easier to manage a smaller number of resources, but that comes with the downside of reduced fault tolerance.

Select a deployment method for high availability

Separate nodes: control, worker, etcd

In a high-availability cluster, you cannot have any single point of failure. This means that you should have separate nodes for control and worker nodes. 

The control plane components should be replicated across multiple nodes. The minimum recommendation is three nodes to maintain a quorum, i.e., the minimum number of nodes needed for the cluster to agree on updates and maintain data consistency. 

You must set up a load balancer in front of the cluster API servers. You should have at least N+1 spare worker nodes to handle the failure of a single node. For even greater levels of availability, you can decouple etcd from the control plane nodes. This topology will reduce the impact of losing any control node where etcd is run as an external cluster to the control plane. 

High-availability Kubernetes cluster topology with external etcd cluster (source)

Deploying across different availability zones

Most large cloud operators offer multiple regions to keep failure zones separate. OpenStack also has the concept of availability zones (AZs) that are logically defined zones within a single physical data center; each zone has independent power, cooling, and network infrastructure. Deploying OpenStack services across multiple AZs enhances fault tolerance within a data center.

For the cluster control plane, where availability is a major concern, you should replicate the control components (API server, scheduler, etcd, and cluster controller manager) across at least three different AZs.

Backup and disaster recovery options

High availability is obviously important for production clusters, and you should also have a backup and disaster recovery plan in case the cluster itself faces a catastrophic failure. This section will discuss some of the disaster recovery options and their effectiveness. 

etcd backup

In a K8s cluster, etcd is the primary key-value data store that replicates all the Kubernetes cluster states. Backing up etcd is crucial to perform disaster recovery in case of any critical failure.

You can use the etcdctl utility on one of the control nodes to take snapshots of the etcd key-value store. You need to specify the endpoint and the certificate files for connecting with the etcd cluster:

ubuntu@node1:~$ sudo etcdctl --endpoints https://node1:2379 \
--cacert=/etc/ssl/etcd/ssl/ca.pem \
--cert=/etc/ssl/etcd/ssl/member-node1.pem \
--key=/etc/ssl/etcd/ssl/member-node1-key.pem \
snapshot save etcd-backup.db
.
.
Snapshot saved at etcd-backup.db

Make sure that the etcd-backup.db file is stored securely offsite. Also, back up on a regular frequency so that you have the least amount of data loss in case of disaster. Please note with the etcd backup, you are only able to recover the K8s cluster configuration and state—not the pods’ persistent data.

OpenStack Cinder snapshots

The second option is to use the snapshot option of OpenStack Cinder volumes. You can preserve the state of the controller and worker node volumes using this option, which is different from etcd database backup. The volume snapshots can be used to clone new volumes or perform recovery to the most recent snapshots in case of any node corruption or failure. 

The snapshots are colocated with the volume backends. In the case of failure of the storage backend itself, you would lose both the node volume and its snapshot.

# Source the OpenStack credentials
$ source k8s-project-openrc.sh
$ openstack volume snapshot create controler1-backup controller1-volume
.
.

Trilio K8S backup and recovery

The two backup and recovery options above offer only partial restoration. The etcd backup only captures the K8s state, and with Cinder snapshots, you can restore the storage volumes by creating new VMs and attaching the snapshots.

A third option is to use Trilio for OpenStack backup and recovery. Trilio can fully orchestrate disaster recovery even in the case of complete OpenStack cloud failure.

Trilio is natively installed alongside OpenStack services and captures the complete state of the OpenStack project—including disk images, instance metadata, Cinder volumes and metadata, ssh-key pairs, network security groups, network subnets, ports, and MAC addresses—and stores all this data as a qcow2 image offsite in NFS or S3 storage. Trilio can perform automated full and incremental backups, and you can perform the disaster recovery on the same or any other offsite OpenStack cluster. 

OpenStack deployment with the Trilio data protection service (source)