OpenShift Backup Best Practices

Step 4. Backup etcd

The etcd database keeps track of the state of all resources in the cluster. Making a backup of the etcd database is a bit more involved than doing other resources because you need to debug into one of the control plane nodes and run scripts to dump a database backup.

One good example of where an etcd restore might come in handy is if you need to restore numerous deployments, but the teams that manage those deployments made changes via the UI and do not have the most up-to-date version of the YAML file. A drawback here is that restoring the etcd database could have unintended consequences and is usually a last resort, so this is also where a dedicated backup and restore tool like Trilio comes in handy.

In the latest version of OpenShift (4.15), automated backups of etcd are still in tech preview, which means they are not supported and require extra steps to make them work.

You can find full documentation on how to create etcd backups in Red Hat’s OpenShift Documentation. Trilio also provides an etcd plugin to help with the backup and restore process.

Security and compliance considerations

These are three primary considerations when it comes to security:

• Use RBAC to limit/audit access: Institute security precautions regarding who can access your data. OpenShift has native role-based access control (RBAC) features that allow you to restrict access to your data and audit any actions performed on it. 
• Make backups immutable: Backups should be immutable, meaning that they cannot be altered or deleted once created. This is a preventative measure against ransomware altering or encrypting your backups to hold your data hostage.
• Encrypt your data: Use encryption keys to encrypt your data as it is being backed up. You should be able to use your own encryption keys as part of this process to ensure that only you can decrypt your backups.

These three considerations set the context for the following two best practices.

Use role-based access controls (RBAC) to limit and audit access

Role-based access control (RBAC) is a built-in user and group permissions system for OpenShift. It enforces policies that limit which users or services can access which resources. 

In OpenShift, there are two types of roles: a user/group-level role and a cluster-wide role. Your approach to RBAC controls will depend on your organization’s needs. If you have multiple tenants that manage their own resources, you will want to create per tenant users or groups to manage the resources. On the other hand, if you have a dedicated team managing your backups, you may want to create a cluster-wide role that has restricted access to all namespaces.

Here is an example of a role that defines what actions can be taken on what resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: development
  name: dev-team-role
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]

When you create a role, you will need a RoleBinding to associate a user or group to a role:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-team-binding
  namespace: development
subjects:
- kind: Group
  name: dev-team
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: dev-team-role
  apiGroup: rbac.authorization.k8s.io

In this example, these two files set up permissions so that members of the dev-team group can take actions (verbs) on the specific resources (get, list, watch, create, etc).

In addition to controlling access to your resources, you can also keep an audit trail of who does what within your cluster. Here is an example of an audit log entry that shows the system:admin user listed the available Trilio policies:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "e54527ce-02d1-426e-82e9-79f651dfa9da",
  "stage": "ResponseComplete",
  "requestURI": "/apis/triliovault.trilio.io/v1/policies?limit=500",
  "verb": "list",
  "user": {
    "username": "system:admin",
    "groups": [
      "system:masters",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "10.0.91.226"
  ],
  "userAgent": "oc/4.15.0 (linux/amd64) kubernetes/62c4d45",
  "objectRef": {
    "resource": "policies",
    "apiGroup": "triliovault.trilio.io",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2024-05-22T01:37:53.130266Z",
  "stageTimestamp": "2024-05-22T01:37:53.133197Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}

Trilio creates RBAC roles that help manage the cluster at both the cluster and namespace levels. This ensures that each user can only see their own resources within Trilio’s interface. You can read more about how Trilio uses RBAC on its documentation page.

Make backups immutable and encrypt them

After protecting who has access to your data, you need to protect the data itself against the possibility of someone unauthorized accessing it. 

There are three steps to this best practice:

1. Pick a storage target such as S3.
2. Encrypt your backups as they are being made.
3. Make the backup immutable.

The first step is to decide what off-site location you want to use to store your backups. You should pick a provider that can enforce immutability and ensure that your data cannot be modified or deleted once saved. Amazon S3 can be a good choice for this.

Second, you want to encrypt your backup data. The biggest concern here is encrypting sensitive information from your database. If you are creating your backups manually, you have options such as GnuPG to encrypt your files. This works but is clunky, and as part of a best practice, you will want to streamline your process.

With Trilio, you can enable encryption when you create the backup and use keys provided by the user, which are saved as secrets on your cluster. When you create a backup with Trilio, it makes a snapshot of a PV and converts it to the QCOW2 image format. Using a widely used format like QCOW2 makes the data easier to move and integrate into other processes. Trilio uses LUKS encryption, a common and well-tested technology, to protect the QCOW2 images.

The last part is to make your backup immutable. Trilio’s documentation provides more information about immutable backups.

One last, often overlooked aspect of this is data sovereignty. Depending on your location and its applicable compliance regulations, there may be restrictions on where your data can reside or under what conditions it needs to be kept. Consult any applicable regulatory websites for information on data protection and storage.

Test data restoration

Part of any backup plan is verifying that the data is good and can be restored quickly. This is achieved by continuously testing your backup process and having it well documented so it is easy to follow during an ongoing data incident.

Testing your restore process regularly and quickly identifies issues such as incomplete backups, corrupted data, missing data, or any general oversight in the backup process.

Create a backup/test schedule

When you make backups and how long you keep them can be explored in an article all its own. In this section, we will scratch the surface.

You want to establish a process for restoring your data that reflects your backup type and schedule. How often and how you backup will determine your testing schedule.

When talking about how to schedule your backup, other than a full backup, there are two types of backups to consider. A full backup is just like it sounds: a backup of all your data. How long this backup takes depends on the amount of data you have to protect. This can put a strain on systems resources and take hours.

A backup tool is useful when you want to optimize how you do backups after a full backup.

After a full backup, your two best options are to make an incremental backup or a differential backup. Both an incremental and differential backup will take less time than running a full backup and take up less disk space but in different ways:

• An incremental backup saves only the data that has changed since the last full or partial backup. It takes minimal space since it only saves the data that has changed since the last backup. However, restoring can be complex since you have to restore the full backup and each incremental backup. 
• A differential backup saves the data that has changed since the last full backup. It takes more space because it always includes the data that has changed since the last full backup. Restoring is easier because you only need the last full and incremental backup.

A common schedule might be to create full backups overnight on Sundays and then incremental or partial backups every night so that you are never too far away from doing a full recovery if needed.

If you are doing this manually, you will want to integrate your tooling into a cron job within OpenShift. The full documentation can be found in the OpenShift documentation for Jobs/CronJobs.

Trilio uses jobs and cron jobs as part of how it manages both scheduled backups and the retention policy for how long they are kept. You can explore their Backup Retention Policy docs for more information.

Knowing that your backup process works will build your team’s confidence, and you will know how long it would take to recover should you need to.

Continuous restores

One approach is to have a system that can continuously back up your data and then restore it to other clusters. This process not only benefits your cluster in the event of a disaster but also helps build test environments and a path from development to QA and production.

Continuous restore in action

With Trilio, you can implement the practice of continuous recovery, where your data is backed up and nearly instantly restored to another cluster. This process can reduce the time it takes to recover from a loss because you have a cluster ready to go. You can read more about continuous recovery on Trilio’s website. 

Document the process

Finally, it is a best practice to document the entire process. This step is often overlooked because the person responsible for performing the restores is often also the only person responsible for documenting the process. Taking the time to document a process will often get deprioritized for more pressing work. 

For your documentation to be successful, it has to be able to be completed by someone who can follow the plan step by step. That individual must be able to perform a backup, restore a backup, and verify that the data is correct.

Here are some questions you’ll want to answer while writing your documentation.