Reference Guide: Optimizing Backup Strategies for Red Hat OpenShift Virtualization

Kubernetes environments face unique security challenges due to their distributed nature and dynamic workloads. The complexity of Kubernetes clusters, with multiple nodes, pods, and containers, creates an expanded attack surface that requires specialized security measures. Common risks include unauthorized access to cluster resources, container vulnerabilities, and network security issues.

Kubernetes security tools address these challenges by providing automated scanning, monitoring, and enforcement capabilities. These tools identify misconfigurations, detect vulnerabilities in container images, and enforce security policies across the cluster. Their primary purposes are to reduce manual effort in securing complex environments, provide real-time threat detection, and ensure compliance with industry standards.

These tools integrate with your existing workflows, enabling you to build security into your development and deployment processes. This integration enables continuous security validation throughout the application lifecycle, from code commit to production deployment, helping maintain a strong security posture.

This article explores open-source options and discusses best practices for securing Kubernetes deployments.

Summary of key Kubernetes security tool concepts

Concepts

Summary

The four “C”s of Kubernetes security

The four “C”s are cloud, cluster, container, and code. Cloud security focuses on the underlying infrastructure. Cluster security addresses API server protection and access controls. Container security involves image scanning and runtime protection. Code security encompasses secure development practices and vulnerability management. This layered approach ensures comprehensive protection.

Open-source Kubernetes security tools

Popular tools like Kube-bench, Trivy, Falco, Cilium, and Checkov each play specific roles in enhancing Kubernetes security, from benchmark testing to network observability.

Kubernetes security best practices

Best practices include hardening cluster components through automated patching and encryption, implementing strong access controls with pod security policies, enhancing network security using network policies and eBPF, securing the supply chain with SBOM and image signing, and establishing comprehensive monitoring with logging and runtime security tools. These practices form a multi-layered defense strategy.

The four “C”s of Kubernetes security

Kubernetes security follows a multi-layered approach, often based on a paradigm called the four “C”s: cloud, cluster, container, and code. Each layer requires specific security measures to collectively create a comprehensive defense strategy.

Cloud security measures

Cloud security measures focus on protecting the underlying infrastructure hosting Kubernetes clusters. This includes securing cloud platforms’ network, storage, and computing resources.

It’s important to understand that cloud security is a shared responsibility. While cloud providers secure the underlying infrastructure, users are responsible for securing their resources and configurations. This includes:

  • Implementing adequate IAM configurations, adhering to the principle of least privilege
  • Continuous monitoring and logging of cloud resources to detect security anomalies
  • Regular security audits and compliance checks

For example, consider this logical setup for a Kubernetes cluster’s security group in AWS:

  • Allow inbound traffic on ports only from the organization’s IP range for secure API server access.
  • Grant only required traffic between Kubernetes cluster nodes within the security group.
  • Implement AWS Systems Manager (SSM) access for secure node management.
  • Restrict all other inbound and outbound traffic.

This configuration ensures that cluster nodes can communicate internally, access necessary AWS services securely, and limit external access to the API server only.

Cluster-level protections

Cluster-level protections address security concerns within the Kubernetes control plane and worker nodes. This involves hardening the API server, controller manager, and scheduler components. 

Critical measures include enabling role-based access control (RBAC), implementing network policies to regulate pod-to-pod communication, and using admission controllers to enforce security policies. Regular updates and patches for Kubernetes components are also essential to mitigate known vulnerabilities. 

Cluster hardening techniques should also include:

  • Ensuring TLS encryption for all API traffic within the cluster
  • Utilizing the built-in Kubernetes certificate management API for automating the TLS certificate lifecycle
  • Implementing pod security admission controllers to put various security restrictions on pods.

Here’s an example of a network policy that restricts pod-to-pod communication:

				
					apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      role: backend
      app: inventory
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
          app: inventory
    ports:
    - protocol: TCP
      port: 6379
				
			

This policy ensures that only frontend pods with the label app: inventory can communicate with backend pods on port 6379, enhancing the security of inter-service communication within the cluster.

Automated Kubernetes Data Protection & Intelligent Recovery

Perform secure application-centric backups of containers, VMs, helm & operators

Use pre-staged snapshots to instantly test, transform, and restore during recovery

Scale with fully automated policy-driven backup-and-restore workflows

Container security practices

Container security practices focus on ensuring the integrity and isolation of containerized applications. This starts with using minimal base images and regularly scanning containers for vulnerabilities. Implementing container runtime security tools helps detect and prevent malicious activities during execution.

Key practices include:

  • Running containers with non-root users and limiting container privileges
  • Implementing container image signing and verification processes to ensure authenticity and integrity
  • Applying the principle of least privilege by running containers with minimal capabilities

Here’s an example of a Dockerfile that implements a few container security best practices:

				
					FROM python:3.9-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.9-slim
RUN useradd -m appuser
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY app.py .
USER appuser
ENV PYTHONUNBUFFERED=1
EXPOSE 8080
CMD ["python", "app.py"]
				
			

This Dockerfile uses a multi-stage build to minimize the final image size, runs the application as a non-root user, and sets secure defaults, reducing the attack surface of the containerized application.

Code-level safeguards

Code-level safeguards involve implementing security best practices during application development and deployment. This includes secure coding practices, such as input validation and error handling. 

Best practices include:

  • Integrating static and dynamic code analysis tools into the CI/CD pipeline to identify vulnerabilities early in development
  • Implementing secrets management solutions to securely store and access sensitive information like API keys and passwords
  • Using infrastructure as code (IaC) with security scanning tools to maintain consistent and secure configurations across environments
  • Implementing mutual TLS (mTLS) for encrypting network traffic between services

Here’s an example of Python code that demonstrates several code-level security practices:

				
					import psycopg2
from psycopg2 import pool

# Create a connection pool
connection_pool = psycopg2.pool.SimpleConnectionPool(
    1, 20, "dbname=myapp user=postgres password=secret host=db.example.com"
)

def get_user(user_id):
    try:
        conn = connection_pool.getconn()
        with conn.cursor() as cur:
            cur.execute("SELECT * FROM users WHERE id = %s", (user_id,))
            user = cur.fetchone()
        return user
    except psycopg2.Error as e:
        print(f"Database error: {e}")
        return None
    finally:
        if conn:
            connection_pool.putconn(conn)

# Usage example with input validation
def get_user_by_id(user_id_input):
    try:
        user_id = int(user_id_input)
        return get_user(user_id)
    except ValueError:
        print("Invalid user ID format")
        return None
				
			

This code uses parameterized queries to prevent SQL injection, implements error handling to avoid exposing sensitive information, and uses a connection pool to manage database connections efficiently. The input validation in the get_user_by_id function adds an extra layer of security by ensuring that the user input is in the expected format.

Watch this 1-min video to see how easily you can recover K8s, VMs, and containers

Open-source Kubernetes security tools

The Kubernetes ecosystem offers various security tools that address different aspects of cluster and application security. These tools provide valuable capabilities for enhancing your cluster’s security posture without significant financial investment.

Described below are some popular open-source tools and their practical applications in Kubernetes environments.

Kube-bench: CIS benchmark testing

Kube-bench ensures that Kubernetes clusters adhere to security best practices. It automates checking cluster configurations against the Center for Internet Security (CIS) Kubernetes Benchmark, a set of globally recognized security standards.

Kube-bench scans Kubernetes components such as the API server, etcd, kubelet, and container runtime for misconfigurations and security vulnerabilities. It provides detailed reports highlighting areas where the cluster falls short of CIS recommendations, allowing administrators to identify and remediate potential security risks quickly.

In practice, Kube-bench can be run as a job within a Kubernetes cluster or on individual nodes. For example, here is code to check a master node:

				
					apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
spec:
  template:
    spec:
      hostPID: true
      containers:
        - name: kube-bench
          image: aquasec/kube-bench:latest
          command: ["kube-bench", "run", "--targets", "master"]
          volumeMounts:
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-kubelet
          hostPath:
            path: "/var/lib/kubelet"
				
			

This job will generate a report highlighting deviations from the CIS benchmark, allowing you to quickly create a security report on your cluster configuration.

Trivy: Comprehensive vulnerability scanner

Trivy is a comprehensive, open-source vulnerability scanner for containers and other artifacts. It’s designed to detect vulnerabilities in container images, file systems, and git repositories as well as misconfigurations in IaC files and Kubernetes resources.

Trivy scans for known vulnerabilities in operating system packages and application dependencies and can identify misconfigurations in Kubernetes manifests. It provides detailed reports that include severity levels, descriptions, and references to the Common Vulnerabilities and Exposures (CVE) database, allowing you to prioritize and address the most critical issues first. You can integrate Trivy into your CI/CD pipeline or run it directly in your cluster.

Here’s an example of how to run Trivy as a cron job to regularly scan your cluster for vulnerabilities:

				
					apiVersion: batch/v1
kind: CronJob
metadata:
  name: trivy-scanner
spec:
  schedule: "0 1 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: trivy-scanner
          containers:
          - name: trivy
            image: aquasec/trivy:latest
            args:
            - k8s
            - --report=summary
            - all
            - --severity=HIGH,CRITICAL
          restartPolicy: OnFailure
				
			

This cron job will run Trivy daily at 1 AM, scanning all resources in the cluster for HIGH and CRITICAL severity vulnerabilities. The results can be forwarded to your log store and event management (SIEM) tool for further analysis and alerting.

Learn about the features that power Trilio’s intelligent backup and restore

Falco: Runtime security

Falco is a cloud-native runtime security tool that provides real-time threat detection for Kubernetes clusters. It monitors system calls at the kernel level, allowing it to detect and generate alerts for suspicious activities across containers, pods, and cluster components.

Falco uses a flexible rules engine to define security policies and detect anomalous behavior. It can identify issues such as privileged container executions, unexpected network connections, or modifications to sensitive files. Falco’s ability to provide context-rich alerts makes it invaluable for incident response and threat hunting in Kubernetes environments.

To deploy Falco using Helm, you can use the following commands:

				
					helm repo add falcosecurity https://falcosecurity.github.io/charts 
helm repo update 
helm install --replace falco --namespace falco --create-namespace --set tty=true falcosecurity/falco
				
			

​Once deployed, Falco will start monitoring your cluster and generate alerts based on its default ruleset, which can be customized to fit specific security requirements.

Cilium: Network security and observability

Cilium is an open-source networking, security, and observability platform for Kubernetes. It leverages extended Berkeley Packet Filter (eBPF) technology to provide robust network security and visibility features at the kernel level.

Cilium offers advanced network policy enforcement, allowing fine-grained control over pod-to-pod communication. It also provides deep visibility into network flows, making troubleshooting connectivity issues and detecting potential security threats easier.

To install Cilium using Helm, you can use the following commands:

				
					helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system
				
			

After installation, you can use Cilium to implement network policies, like this:

				
					apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-frontend-to-backend"
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
				
			

This policy allows traffic only from frontend pods to backend pods, demonstrating Cilium’s ability to enforce fine-grained network segmentation.

Checkov: Infrastructure-as-code scanning

Checkov is an infrastructure-as-code (IaC) scanning tool that helps identify misconfigurations and security risks in Kubernetes manifests, Helm charts, and other IaC files before deployment to the cluster.

Checkov analyzes manifest/IaC files for security best practices, compliance standards, and potential vulnerabilities. It can detect issues such as exposed ports, missing resource limits, or overly permissive security contexts in pod specifications.

Here’s how to scan a Kubernetes manifest with Checkov:

				
					> checkov -f deployment.yaml

Check: CKV_K8S_13: "Memory requests should be set" 	
FAILED for resource: Deployment.example-deployment 	
File: /deployment.yaml:1-18
 
... 

Check: CKV_K8S_8: "Liveness Probe Should be Configured" 	
FAILED for resource: Deployment.example-deployment 	
File: /deployment.yaml:1-18 
				
			

This command will analyze the deployment.yaml file and report any identified security issues or misconfigurations, allowing you to address these problems early in the deployment cycle.

Kubernetes security best practices

Securing Kubernetes environments requires a comprehensive approach that addresses challenges across multiple infrastructure layers. These best practices include cluster hardening, access controls, network security, and supply chain protection. 

Note: The following best practices are primarily geared toward self-managed Kubernetes clusters. While many principles apply universally, managed Kubernetes services cloud vendors often handle some of these aspects automatically or require different approaches.

Harden cluster components

Hardening cluster components is a critical aspect of securing a Kubernetes environment. These security measures at the cluster level can minimize the attack surface for potential threats.

Here are three specific steps for hardening your Kubernetes cluster components.

Automate security patching

Regularly updating and patching Kubernetes components is integral for maintaining a secure cluster. Kube-bench, an open-source tool, automatically checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

To implement automated security patching with kube-bench:

  1. Integrate kube-bench into your CI/CD pipeline to run regular checks.
  2. Set up alerts for failed checks or deviations from the benchmark.
  3. Automate the application of security patches based on kube-bench findings.

Here’s an example of running kube-bench as part of a CI/CD job:

				
					security_check:
  stage: test
  script:
    - docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t aquasec/kube-bench:latest run --outputfile results.json --targets=master
  only:
    - schedules
  artifacts:
    paths:
      - results.json
				
			

This CI job runs kube-bench against the master node and generates a report that can be used to trigger automated further patching processes.

Encrypt etcd data at rest using LUKS or dm-crypt

In self-managed clusters, etcd is a fundamental component that requires strong protection. Encrypting etcd data at rest prevents unauthorized access to sensitive information in case of physical theft or unauthorized access to the storage system.

Considerations for encrypting etcd:

				
					# Example using HashiCorp Vault for key management
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-etcd-encryption
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com"
    roleName: "etcd-encryption-role"
    objects: |
      - objectName: "etcd-encryption-key"
        secretPath: "secret/data/etcd-encryption"
        secretKey: "key"
				
			
  • Perform load/stress tests to ensure that encryption doesn’t significantly impact etcd performance.
				
					# Benchmark etcd performance with encryption 
etcd --data-dir=/path/to/encrypted/etcd-data benchmark --target-leader --conns=100 --clients=1000
				
			

Enable and configure Kubernetes audit logging

Audit logging enables visibility into cluster activities, detects security incidents, and meets compliance requirements. Kubernetes audit logging records API requests and their associated metadata, providing a detailed trail of actions performed within the cluster.

For example, to enable and configure Kubernetes audit logging:

1. Create an audit policy file that defines what events should be logged

				
					# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: ""
resources: ["pods", "services", "configmaps"]
				
			

2. Configure the API server to use the audit policy and specify the log backend:

				
					# kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit.log
				
			

3. Set up log rotation and archiving to manage log file sizes:

				
					# logrotate configuration
/var/log/kubernetes/audit.log {
    rotate 5
    daily
    compress
    missingok
    notifempty
}
				
			

Implement strong access controls

Use Pod Security Admission to enforce security contexts

Pod Security Admission is a built-in admission controller that enforces pod security standards at the namespace level. It defines conditions that resources (pods, deployments, etc. ) must meet to be allowed to be deployed into the cluster.

Here’s an example of configuring Pod Security Admission:

				
					apiVersion: v1
kind: Namespace
metadata:
  name: my-secure-namespace
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
				
			

This configuration enforces the baseline policy, audits against the restricted policy, and warns about violations of the restricted policy.

Next, create a pod that adheres to the security standards:

				
					apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
  namespace: my-secure-namespace
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: your-secure-app:latest
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
				
			

This pod configuration adheres to the baseline security standards by running as a non-root user, using the default seccomp profile, and dropping all capabilities.

Integrate with external secret management systems

External secret management systems provide a secure way to store, access, and manage sensitive information such as API keys, passwords, and certificates.

Here is an example of the SecretProviderClass that uses external HashiCorp Vault:

				
					apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.default:8200"
    roleName: "database-role"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/myapp/database"
        secretKey: "password"
				
			

Regularly rotate Kubernetes certificates and service account tokens

Rotating certificates and tokens helps mitigate the risk of compromised credentials. For example, here are the steps to rotate Kubernetes certificates and tokens:

1. Check certificate expiration:

				
					kubeadm certs check-expiration
				
			

2. Rotate certificates:

				
					kubeadm certs renew all
				
			

3. To rotate service account tokens, use token expiration:

				
					apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: build-secret
  annotations:
    kubernetes.io/service-account.name: build
    kubernetes.io/service-account.expiration: "2024-12-31T12:00:00Z"
				
			

Enhance network security

Network security in Kubernetes involves controlling traffic flow between pods and monitoring network activities for potential threats.

Apply network policies for fine-grained traffic control

Network policies enable fine-grained control over pod-to-pod and pod-to-external-service communication.

Here’s an example of a network policy that allows incoming traffic only from specific pods:

				
					apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 8080
				
			
This policy only allows incoming traffic on port 8080 from pods labeled with role: frontend to pods labeled with app: api.

Leverage eBPF for Advanced Network Monitoring and Security

extended Berkeley Packet Filter (eBPF) provides robust network monitoring and security capabilities in Kubernetes. It offers deep insights into system performance, security threats, and network traffic without the need for intrusive modifications to applications or the kernel itself. Here’s an example using Cilium, an eBPF-based networking solution:
				
					# Install cilium
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system

#Enable Hubble for network visibility:
cilium hubble enable

#Use Hubble to monitor network flows:
hubble observe --follow

# Implement an eBPF-based network policy:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-policy"
spec:
  endpointSelector:
    matchLabels:
      app: myservice
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: client
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/v1/.*"
				
			

This Cilium policy allows only GET requests to paths starting with /api/v1/ from pods labeled with org: client.

Secure the supply chain

Securing the Kubernetes supply chain involves ensuring the integrity and authenticity of all components from development to deployment. This process focuses on creating transparency in the software lifecycle and verifying the provenance of container images.

Implement software bill of materials (SBOM) generation and verification

Implementing software bill of materials (SBOM) generation and verification is vital to securing the Kubernetes supply chain. SBOMs provide an inventory of all components in a software package, including dependencies, versions, and licensing information. This transparency lets you quickly identify and respond to vulnerabilities and comply with licensing requirements.

Tools like Syft can automate SBOM generation during the build process, while Grype can scan SBOMs for known vulnerabilities, integrating seamlessly into CI/CD pipelines.

SBOM lifecycle in a Kubernetes environment

SBOM lifecycle in a Kubernetes environment

Use tools like Sigstore for signing and verifying container images

Sigstore offers tools for signing and verifying container images, enhancing the integrity and authenticity of software artifacts in Kubernetes environments. Using Cosign, a key component of Sigstore, you can cryptographically sign container images during the build process and verify them before deployment. This signing process creates a tamper-evident record of the image’s origin and contents. 

Combined with Kubernetes admission controllers, you can enforce policies that only allow the deployment of signed and verified images, significantly reducing the risk of running compromised or unauthorized software in your clusters.

Container image signing and verification with Sigstore

Container image signing and verification with Sigstore

Trilio’s security approach to Kubernetes

Implementing Kubernetes security best practices forms the foundation of a robust security posture, and specialized tools can further enhance your cluster’s protection. Trilio offers a comprehensive approach tailored for Kubernetes environments, addressing data protection and disaster recovery.

Comprehensive backup and recovery

Trilio’s solution provides application-aware backups of Kubernetes resources, including persistent volume claims and Helm releases. This granular approach ensures that all components of your applications are consistently protected. 

Recovering to any Kubernetes cluster enhances disaster recovery capabilities, allowing for quick restoration in case of cluster-wide failures or during migration scenarios.

Enhanced security for data protection

The security of backup data can’t be overlooked, especially in the context of ransomware threats. Trilio addresses this by offering encryption of backups on any storage platform and supporting immutable backups. This prevents unauthorized changes to backup data, which is required to maintain a reliable recovery point after security incidents.

Integration with security automation

Integrating security practices into the development lifecycle is essential for maintaining a solid security posture. Trilio supports integration into CI/CD pipelines, enabling secure data management practices during development and testing. This integration helps ensure that security considerations are addressed throughout the application lifecycle, from development to production deployment.

Learn about a lead telecom firm solved K8s backup and recovery with Trilio

Conclusion

Kubernetes security tools offer significant benefits in protecting your containerized environments. These tools automate vulnerability detection, enforce security policies, and provide real-time monitoring, greatly enhancing your ability to maintain a secure Kubernetes infrastructure. With tools like Kube-bench, Trivy, Falco, Cilium, and Checkov, you can systematically address various aspects of security, from compliance checking to runtime threat detection.

A layered security approach is crucial for comprehensive Kubernetes protection. This strategy involves securing every level of your infrastructure, from the underlying cloud platform to the application code running in containers. By implementing security measures at each layer—cloud, cluster, container, and code—you create multiple lines of defense against potential threats. This depth of protection is essential in the complex and dynamic environment of Kubernetes, where traditional perimeter-based security is insufficient.

Remember, cluster security is an ongoing process, not just a day-one implementation. To enhance your Kubernetes security posture, assess your current security measures against the four “C”s framework. Identify areas for improvement, and gradually integrate security tools into your workflows.

Table Of Contents

Like This Article?

Subscribe to our LinkedIn Newsletter to receive more educational content

Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.