Kubernetes Best Practices, Tips and Tricks

Security context

K8s provides security contexts that you can use to restrict containers, kernel capabilities, and access to the filesystem. You can use this to limit privilege escalation and force the containers to run as specific non-root users. You can also define specific permissions for the mounted filesystems.

Example

Create a container to run with specific non-root user and group permissions, disallow privilege escalation, ensure that the file systems are mounted with ownership of a specific group, and check that the root file systems are mounted as read-only. 

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 2000
    fsGroup: 3000
  containers:
  - name: secure-container
    image: my-secure-app:latest
    securityContext:
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true

OpenShift security context

RedHat OpenShift enforces a high level of security by limiting pods to the fewest privileges possible. The platform prevents containers from protected functions like root access and shared file system access. The OpenShift Restricted Security Context Constraints (SCC) ensure that containers are run using arbitrarily assigned user IDs, root filesystems are mounted as read-only, and the container network interface is isolated from the host system. 

Application security

When using container images in K8s, you should always be using trusted sources like the official repository from the Docker Hub or your own private repo. Alternatively, you can also download images from verified publishers like Nginx, VMware, or HashiCorp that publish high-quality, secure images.

You should be using the most recent version of images that include the latest security patches. For better consistency across the application environment, pin specific versions of the image and avoid the latest tag.

Example

Create an Nginx deployment using the official docker hub repository of Nginx and a specific mainline release version. 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        # Image from the official Docker Hub NGINX repo
        image: nginx:1.27.2  # Specific version tag
        ports:
        - containerPort: 80

CIS Kubernetes benchmarks

The Center for Internet Security (CIS) provides security guidelines and benchmarks for operating systems, applications, and cloud operations. These controls are the recommended security practices, created using real-world experiences and consensus review of a global community of subject matter experts. 

Adopting the recommendations of the CIS Benchmark for Kubernetes is an essential step in verifying the security of the K8s environment. The benchmark provides an extensive list of checks and recommended practices for API server configuration, etcd configuration, kubelet and node security, network policies, and pod security.

Although running tests manually is one option, the CIS recommended checks can be automated using open-source utility tools like Kube-Bench. Kube-Bench can run the checks documented in the CIS K8s benchmark and provide a pass/fail report for each individual configuration check. 

A sample CIS check run using Kube-Bench (Source)

Deployments and replica sets

In a K8s live production environment, you need to ensure the availability of pods and apply update strategies to manage the life cycle of your applications. Using deployments, you can define rolling updates and rollbacks, allowing you to modify applications without downtime. With replica sets, you can maintain a stable set of pod replicas at any time. You can define a sufficient number of replicas required for the efficient running of your application and maintain availability in case of node failure.

Example

Here, we will create a nginx deployment with a rolling update strategy and three replicas. The deployment will ensure that only one pod is unavailable during updates and will create one additional pod during updates to reduce downtime:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3  # Number of replicas
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # pods unavailable during update
      maxSurge: 1        # additional pods above number of replicas
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.27.2  # NGINX image
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10

Logging and monitoring

K8s is a complex platform with lots of moving components. The system can generate hundreds of thousands of logs, events, and metrics. Centralized observability and monitoring solutions are highly recommended for proper visibility, diagnosis, and troubleshooting.

You can use a log collection tool like Fluentd to collect logs from K8s, transform them into your required format, and ship them to the Elasticsearch backend. Elasticsearch is a real-time distributed and scalable search engine that can index and search through a large volume of data. In addition to the above, Kibana can be deployed alongside Elasticsearch for visualization and dashboards.

For collecting metrics, you can use Prometheus along with Grafana for visualizing and alerting based on these metrics.

CI/CD and GitOps

In large and complex infrastructure operations, it is highly recommended that automation tools and version control systems be used. With automation, you can reduce the risks of human errors, improve operational efficiency, and ensure consistent deployments across your infrastructure. With version control systems, you can enforce best practices of change management, tracking, auditing, approval, and rollback. 

You can use version control platforms like GitHub or GitLabs to manage your YAML manifests. To execute your pipeline deployments, you can use Jenkins as a general-purpose automation tool with a rich set of plugins to integrate with different platforms. Another option is ArgoCD, which has native integrations with Git and Kubernetes.

OpenShift Pipelines

Redhat OpenShift Pipelines is a cloud-native CI/CD system built for decentralized teams working on microservices architecture. Organizations already using OpenShift can leverage Pipelines because it is natively integrated into Kubernetes. This makes it easier to manage pipelines, as they are handled using the same Kubernetes APIs and tools without the need for external CI/CD tools.

Static code analysis

When you have a large team working on multiple aspects of your application and deploying various infrastructure components using infrastructure as code (IaC), it is highly recommended to use automated security scanning tools to perform code analysis. With multiple engineers with different skill levels, there are bound to be mistakes, like exposing secrets in plaintext, misconfigured permissions, and missing RBAC controls.

You can use open-source tools like Terrascan to perform automated code analysis. Terrascan includes over 500 policies to scan your code against standards like CIS Benchmarks. You can integrate Terrascan with your CI/CD process to automate the code analysis process.

Example

Consider the following deployment manifest for Apache:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-deployment
  labels:
    role: webserver
spec:
  replicas: 2
  selector:
    matchLabels:
      role: webserver
  template:
    metadata:
      labels:
        role: webserver
    spec:
      containers:
      - name: frontend
        image: httpd
        ports:
        - containerPort: 80

Scanning this with Terrascan would list the policy violations detected in the code:

$ terrascan scan -i k8s -f apache-deployment.yaml 
Violation Details -
    
 Description    : CPU Limits Not Set in config file.
 File           : wordpress-deployment.yaml
 Line           : 28
 Severity       : MEDIUM
 ------------------------------------------------------------------
 
 Description    : No readiness probe will affect automatic recovery in case of unexpected errors
 File           : apache-deployment.yaml
 Line           : 28
 Severity       : LOW
 ------------------------------------------------------------------
 
 Description    : Apply Security Context to Your Pods and Containers
 File           : apache-deployment.yaml
 Line           : 28
 Severity       : MEDIUM
 ------------------------------------------------------------------
 
 Description    : Image without digest affects the integrity principle of image security
 File           : apache-deployment.yaml
 Line           : 28
 Severity       : MEDIUM
 ------------------------------------------------------------------
 
 Description    : Prefer using secrets as files over secrets as environment variables
 File           : apache-deployment.yaml
 Line           : 28
 Severity       : HIGH
 ------------------------------------------------------------------

Disaster recovery and backups

For business-critical applications it is essential to have a proper disaster recovery plan. Failure is always a possibility due to a system failure, human error, or malicious activity. As part of a set of best practices, you should always have a backup of your critical data and components that you can quickly restore in case of any type of failure. 

Backup of etcd

Etcd is a distributed key-value store that holds all K8s objects and cluster states. For production environments, it is important to deploy etcd as a multi-node cluster with a minimum of three nodes (five recommended). You should also take regular backups of the etcd state to recover from any type of loss or corruption of data. 

Etcd supports built-in snapshots as follows. The snapshots should be stored somewhere offsite that is separate from the main production environment:

ubuntu@node1:~$ sudo etcdctl --endpoints https://node1:2379 \
--cacert=/etc/ssl/etcd/ssl/ca.pem \
--cert=/etc/ssl/etcd/ssl/member-node1.pem \
--key=/etc/ssl/etcd/ssl/member-node1-key.pem \
snapshot save etcd-backup.db
.
.
Snapshot saved at etcd-backup.db

Backup of persistent volumes

The etcd snapshots will only save the configuration state of the k8s cluster, not the application data. Applications like databases use persistent volumes to store data. You can take snapshots of the storage volumes if the infrastructure supports backups (like AWS EBS snapshots or Azure Disk snapshots). 

Trilio for Kubernetes

The two backup options discussed above have some limitations. The etcd snapshot only saves the configuration state of the K8s cluster. The volume snapshots are dependent on the storage infrastructure and require a manual recovery process to bring the k8s cluster and applications back up.

The third option is to use Trilio for Kubernetes (T4K). With T4K, you can fully orchestrate your backup and recovery process even in case of failure of a complete K8s cluster. With T4K, you can protect metadata, operators, container image registry, virtual machines, and persistent volumes. You can perform partial disaster recovery (like metadata only) or complete disaster recovery on any K8s cluster whether on-prem or on-cloud.