Since virtualization became a generally available feature in OpenShift with version 4.5, it has been seen as a compelling alternative to traditional virtualization platforms. It lets users harness the advantages of both containerization and virtualization in a consolidated environment. Earlier this year, Red Hat also announced the availability of a new OpenShift edition, Red Hat OpenShift Virtualization Engine, which focuses exclusively on running virtualization workloads.
Storage design is often the make-or-break factor in Red Hat OpenShift virtualization deployments. Carefully planning storage resources is essential because they play a significant role in the virtualization environment’s scalability and performance.
One key advantage of OpenShift’s virtualization approach is its intelligent storage integration. Instead of creating separate frameworks and methodologies for managing virtual machine storage, Red Hat OpenShift virtualization extends native Kubernetes concepts such as storage classes, persistent volumes (PVs) and persistent volume claims (PVCs) to virtual machines. This approach simplifies storage provisioning and the management of virtual machines.
Summary of key Red Hat OpenShift virtualization storage concepts
The following table provides an overview of the key areas related to Red Hat OpenShift virtualization storage discussed in this article.
Focus area | Description |
Managing storage through native Kubernetes objects | Red Hat OpenShift virtualization provides a streamlined way to manage storage for virtual machines by leveraging native Kubernetes API objects like PVCs, PVs, and storage classes, abstracting the complexities of the underlying infrastructure. |
Mapping VMware storage to Red Hat OpenShift virtualization storage | Familiar VMware storage terminologies, like datastores and VMDKs, can be translated to their OpenShift virtualization equivalents (storage classes and PVCs), which provides a clear path for users transitioning from VMware environments. |
Configuring persistent storage for virtual machines | OpenShift simplifies common storage operations for VMs, including attaching new disks by creating a new PVC and expanding existing ones by simply updating the size in the associated PVC. |
Connecting virtual machines to external storage | Virtual machines can be easily connected to external storage volumes using standard protocols like NFS, Fibre Channel (FC), and iSCSI, which are exposed to the VM through a persistent volume. |
Cloning virtual machines | Red Hat OpenShift virtualization offers various techniques for cloning existing virtual machines, such as CSI volume and Smart cloning. |
Creating snapshots | A point-in-time snapshot of a VM’s disk can be generated by creating a volume snapshot object of its underlying PVC, which provides a critical capability for backup and rapid restoration. |
Automated Application-Centric Red Hat OpenShift Data Protection & Intelligent Recovery
Managing storage through native Kubernetes objects
When dealing with virtualization in OpenShift, it’s helpful to understand that the platform simplifies things by representing virtual machine storage as native OpenShift objects. This approach effectively hides the complexity of the storage backend and means that administrators can manage storage using the same familiar tools and concepts they already use without worrying about the underlying infrastructure.
The following are the major components involved when dealing with storage in OpenShift:
- Persistent volume claims (PVCs): Workloads declare their storage requirements through PVCs, which specify details like capacity, access modes, and (optionally) preferred storage classes.
- Persistent volumes (PVs): These represent actual storage resources in the cluster, supporting various on-premises and cloud-based storage platforms and backend solutions, such as NFS, iSCSI, FC, etc.
- Storage classes: These act as blueprints defining the different types of storage available in the cluster. Storage classes can group PVs with similar characteristics (performance tiers, availability zones, or provider types). The cluster workloads can then request storage from specific classes through their PVCs.
- Container storage Interface (CSI): OpenShift leverages CSI drivers to integrate with diverse storage backends. This interface translates PV/PVC requests into provider-specific operations, supports any CSI-compatible storage solution, and enables Red Hat OpenShift virtualization to utilize the full range of supported storage options.
- Storage provisioners: Each storage class utilizes a specific provisioner that handles volume creation and management, converts PVC requests into concrete storage resources, and works through CSI to interface with the actual storage systems.
Understanding the storage provisioning workflow in OpenShift
More details about how OpenShift manages storage can be found here.
Automated Red Hat OpenShift Data Protection & Intelligent Recovery
Perform secure application-centric backups of containers, VMs, helm & operators
Use pre-staged snapshots to instantly test, transform, and restore during recovery
Scale with fully automated policy-driven backup-and-restore workflows
Mapping VMware storage to Red HatOpenShift Virtualization Storage
Many organizations today are moving their virtual machine workloads from VMware to OpenShift. To make this process as smooth as possible, it’s essential to understand the different terminology and storage concepts used by both platforms.
For virtualization administrators, translating their existing knowledge of VMware concepts, such as datastores and VMDKs, to the equivalent concepts in OpenShift is the key to a successful migration. This is particularly important because VMware is such a widely used virtualization platform.
The table below offers a comparison of key storage concepts in both VMware and OpenShift virtualization. It’s designed to be a quick guide for understanding how features on one platform correspond to those on the other.
Feature | VMware | OpenShift |
Connection to backend storage | Uses VMkernel-level drivers that are tightly integrated with the hypervisor. | Uses the CSI standard, which allows for a more vendor-agnostic approach. |
VM disk storage location | Datastore; ESXi formats storage using VMFS and creates a shared datastore accessible to all virtual machines. | Persistent volume claim (PVCs) and persistent volumes (PVs) abstract the underlying storage. |
Policy-based storage management | vVols, Storage Policy Based Management (SPBM). | Storage classes. These define storage features like quality of service, throughput, and the underlying technology. |
Dynamic provisioning | Typically a more manual process where administrators preconfigure datastores and VMDKs or use templates and scripts for automation. | Uses storage classes and a CSI provisioner to dynamically provision storage on demand when a PVC is created. |
Non-disruptive VM storage migration | Storage vMotion | Storage live migration |
Storage availability and backend | vSAN (vVols), NFS, SAN (Fiber Channel or iSCSI), raw device mapping | PV and PVC paradigms for managing storage; CSI for seamless integration with various storage backends and software-defined storage solutions |
Accessing legacy storage arrays | Compatible with all major storage vendors | All major storage vendors have CSI-compliant driver |
Storage-accelerated cloning | Supported via vStorage APIs for Array Integration (VAAI) | Supported via CSI drivers |
Disk resizing | Supported | Supported |
Disk/VM snapshots | Supported | Supported |
Storage distributed resource scheduler (SDRS) | Supported | N/A |
Migrating from VMware to OpenShift requires aligning the existing VM disk types with the right OpenShift storage classes.
Storage provisioning approaches for OpenShift virtualization
There are multiple approaches through which virtual machines can consume storage in OpenShift. Each method has its own merits and trade-offs; carefully considering the infrastructure requirements is essential before making a decision.
The three primary methods for providing storage to virtual machines in OpenShift are:
- External storage access via CSI drivers: This approach utilizes an external storage solution, such as a traditional SAN or NAS array, with a vendor-provided Container Storage Interface (CSI) driver.
- Internal storage access via an SDS: This method leverages software-defined storage (SDS) products, like OpenShift Data Foundation, to create a shared storage pool from the local disks of the OpenShift cluster nodes.
- Shared-disk file systems: This approach provides a shared file system accessible to all virtual machines, which is often a suitable solution for specific application types.
The three approaches are discussed in more detail below.
Accessing external storage with CSI
This method utilizes external storage infrastructure, such as a traditional SAN or NAS environment, by connecting to the OpenShift cluster using a vendor-provided Container Storage Interface (CSI) driver. This is a good option for organizations using enterprise storage arrays and wanting to use them for virtual machines.
The primary benefit is maintaining centralized storage management outside OpenShift while providing virtual machines access to advanced features like hardware-accelerated snapshots and high-availability from the storage array. This method is preferred when you want to integrate with and benefit from your existing storage environment.
Accessing external storage via CSI
Most CSI drivers come packaged as container images. While these drivers aren’t inherently aware of the OpenShift environment, they can be deployed by a cluster administrator. To make a CSI-compatible storage backend work with OpenShift, administrators must install several components that act as a bridge between OpenShift and the storage driver. All the major storage vendors, such as Dell and HPE, provide these drivers.
Accessing internal storage via SDS
This method uses an SDS solution that pools storage directly from the local disks on your cluster nodes. It creates a hyper-converged architecture where the same nodes that run your virtual machines and containers also provide all the storage.
A key advantage of this approach is its deep integration with OpenShift, which streamlines storage management. This lets you handle administrative tasks directly through the cluster’s control plane. Furthermore, since storage capacity and performance scale automatically with the addition of new nodes, this approach is ideal for consolidating computing and storage infrastructure into a single, unified platform.
The most notable solutions in this space are OpenShift Data Foundation and Portworx.
Accessing storage via SDS CSI
When using an SDS solution, the storage can be either a local block device or a block volume provisioned via any existing enterprise storage array. The SDS solution manages these block devices and creates an abstracted storage pool for virtual machine disks.
Shared disk filesystems
This approach uses a shared disk file system that allows multiple virtual machines to read and write to the same logical disk concurrently. Solutions like the IBM Fusion Access for SAN and the Veritas Infoscale Operator fall in this category. They are often deployed for mission-critical applications, such as clustered databases or high-performance computing (HPC) workloads, that require shared, high-speed access to a single block storage resource. The relevant CSI driver provides the framework for managing these shared-disk resources within Red Hat OpenShift Virtualization. This approach is used in specific scenarios and requires careful configuration to ensure data consistency and integrity.
Configuring persistent storage for virtual machines in OpenShift
When configuring storage for a virtual machine, it’s important to remember that each storage backend has unique features, and its volumes can be mounted in either read-only or read/write mode. End users can specify their desired access modes in the persistent volume claim (PVC).
The following access modes are applicable to a persistent volume:
- ReadWriteOnce (RWO): This mode allows the volume to be mounted in read/write mode by a single node at a time. It is commonly used for block devices and is supported by most storage providers.
- ReadWriteOncePod (RWOP): This mode allows the volume to be mounted in read/write mode by a single pod on a single node. It uses the SELinux mount feature, which is typically used for block devices, and works with many storage providers.
- ReadOnlyMany (ROX): You can mount the volume in read-only mode on multiple containers simultaneously. However, not all storage backends support this mode.
- ReadWriteMany (RWX): This mode lets you simultaneously mount the volume in read/write mode on multiple containers. Like ROX, not all storage backends support this mode.
It’s critical to note that the VM live migration feature in OpenShift Virtualization specifically requires the VM’s persistent volumes to be in RWX mode. Virtual machines cannot live-migrate RWO volumes. The core issue is that since these volumes are exclusively mounted to one node, they would be inaccessible to the destination node, which is essential for a successful migration.
Now that we’ve covered the concepts behind OpenShift Virtualization storage, we’ll proceed to see how you can perform the major storage operations for virtual machines. The demonstrations in the upcoming sections assume that you’ve set up all the prerequisites for OpenShift Virtualization and have virtual machines running in your cluster. If you haven’t already, follow this guide to start with virtual machines in OpenShift.
Hot insertion and removal of disks
OpenShift allows the addition and removal of disks without stopping virtual machines. However, a key point to remember is that this process relies on a specific type of controller. Each virtual machine is equipped with a virtio-scsi controller, which allows hot-plugged disks to use a SCSI bus. This approach is highly scalable and can support over 4 million disks.
The standard VirtIO disk driver is not used for hot-plugging because it is not scalable. Each VirtIO disk requires a dedicated PCI Express (PCIe) slot inside the virtual machine. The number of these slots is limited, and they must be reserved in advance for other devices, meaning that a slot might not be available on demand when you need to add a new disk. The virtio-scsi controller solves this limitation by using just a single PCIe slot to manage many disks.
Before creating a PVC which can be attached to a running VM, it’s often helpful to first inspect the available storage classes in the cluster. This will allow you to select the appropriate storage type that meets your virtual machine’s requirements.
$ oc get storageclasses -o name
storageclass.storage.k8s.io/ocs-external-storagecluster-ceph-rbd-virtualization
storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbd
storageclass.storage.k8s.io/ocs-storagecluster-ceph-rgw
storageclass.storage.k8s.io/ocs-storagecluster-cephfs
Let’s create a PVC so that we can hot-plug a disk to our running VM.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hotplug-demo
spec:
accessModes:
- ReadWriteMany
storageClassName: ocs-external-storagecluster-ceph-rbd-virtualization
volumeMode: Block
resources:
requests:
storage: 5Gi
Apply the manifest to create the PVC and verify.
$ oc create -f demo-pvc.yaml
persistentvolumeclaim/hotplug-demo created
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hotplug-demo Bound pvc-eabc98112-3f04-4e6e-9581-8329dc926439 5Gi RWX ocs-external-storagecluster-ceph-rbd-virtualization 4s
This new volume can now be hot plugged to a virtual machine as follows. The following command will attach hotplug-demo to the rhel9-vm VM.
$ oc get vm
NAME AGE STATUS READY
rhel9-vm 7m Running True
$ virtctl addvolume rhel9-vm --volume-name=hotplug-demo
Successfully submitted add volume request to VM rhel9-vm for volume hotplug-demo
Similarly, disks can be unplugged on the fly as follows.
$ virtctl removevolume rhel9-vm --volume-name=hotplug-demo
Successfully submitted remove volume request to VM rhel9-vm for volume hotplug-demo
The availability and unavailability of the disk can be verified by accessing the virtual machine and listing the available block devices.
Resizing virtual machine disks
Red Hat OpenShift virtualization enables the resizing of PVCs. When the size of the PVC is changed, OpenShift automatically resizes the associated PV. Because most VM disks are based on PVCs, resizing a disk by resizing its PVC is possible, but it is not possible to shrink PVCs.
Each storage class has a field that specifies whether volume expansion is supported. When volume expansion is available, the storage class resource allows the allowVolumeExpansion field to be set to true.
$ oc get storageclass ocs-external-storagecluster-ceph-rbd-virtualization -o yaml | grep allowVolumeExpansion
allowVolumeExpansion: true
The PVC can be expanded by editing its resource file and modifying the spec.resources.requests.storage parameter.
$ oc edit pvc/hotplug-demo
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hotplug-demo
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: ocs-external-storagecluster-ceph-rbd-virtualization
volumeMode: Block
volumeName: pvc-eabc98112-3f04-4e6e-9581-8329dc926439
Once the PVC has been expanded, the disk might need to be rescanned inside the host OS. Most operating systems report the new size but do not automatically expand the partitions and file systems. The disk must be resized to use the additional space with the appropriate tools from within the operating system.
Connecting virtual machines to external storage
When a PVC is created to request a volume for a workload, OpenShift prepares a PV and then binds it to the PVC. This approach differs from VMware’s storage concept because administrators define each PVC for VMs in OpenShift instead of relying on a shared datastore. Cluster administrators must grasp this paradigm shift to PVCs within OpenShift to ensure a smooth migration to the platform during VMware migrations.
With dynamic provisioning, OpenShift creates persistent volumes on demand. A cluster provisioner controls the back-end storage. However, not all storage types have a provisioner; in some organizations, dynamic provisioning is unavailable because the back-end storage is managed manually or used by applications that OpenShift does not control. In those environments, OpenShift uses static provisioning. Storage administrators prepare volumes in the back-end storage and then OpenShift administrators declare these volumes in PV resources. When you create the PVC, Kubernetes binds one available PV to the claim.
A VM can be connected to external storage via the following methods:
- Connecting the storage directly from inside the VM by using VM operating system tools via protocols such as NFS and iSCSI
- Preparing a static PV for the external storage, create a PVC that binds to the PV, and then attaching that PVC as a disk inside the VM
For the sake of this demo, we’ll focus on the second method. The following demo shows how static PVs can be provisioned via NFS and FC protocols and attached to VMs.
This code shows the resource file for a PV that describes the connection to an NFS share:
apiVersion: v1
kind: PersistentVolume
metadata:
name: static-nfs-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
claimRef:
name: demo-nfs-pvc
namespace: vm-pool
nfs:
path: /exports/demo
server: 192.168.150.200
The details of a PV should be provided in the claimRef section to ensure that it is reserved for a specific PVC. By default, OpenShift does not reserve PVs, so if a developer creates a PVC, OpenShift might bind it to an available PV.
For NFS, OpenShift only supports the Filesystem mode. As the default value for the volumeMode parameter is Filesystem, it can be skipped. The details mentioned under the nfs section must be provided by the storage administrator after making the relevant configurations.
Similarly, if the cluster nodes have Fiber Channel (FC) connectivity to a SAN array, static PVs can also be configured via FC protocol. The following example shows the resource file for a PV that describes the connection to an FC volume. Only block mode is supported for PVs provisioned via FC.
apiVersion: v1
kind: PersistentVolume
metadata:
name: static-fc-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
volumeMode: Block
claimRef:
name: demo-fc-pvc
namespace: vm-pool
fc:
targetWWNs:
- "50060e80209e25b3"
lun: 0
Apply the manifest to create the static PV:
$ oc apply -f static-pv.yaml
persistentvolume/static-fc-pv created
$ oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS
static-fc-pv 10Gi RWO Retain Available
Create the PVC to bind to the existing PV:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: demo-fc-pvc
namespace: vm-pool
spec:
resources:
requests:
storage: 10Gi
accessModes:
- ReadWriteOnce
volumeMode: Block
Create the PVC and confirm that it binds to the PV:
$ oc apply -f static-fc-pvc.yaml
persistentvolumeclaim/demo-fc-pvc created
$ oc get pvc -n vm-pool
NAME STATUS VOLUME CAPACITY ACCESS MODES [.......]
demo-fc-pvc Bound static-fc-pv 10Gi RWO
The bound PVC can then be hot plugged to a VM via the OpenShift console or CLI, as discussed in the earlier section on hot insertion and removal of disks.
Managing virtual machine snapshots
With Red Hat OpenShift virtualization, you can take snapshots of both running and stopped virtual machines. To ensure data consistency for live snapshots, OpenShift communicates with a QEMU guest agent that quiesces the file systems before the snapshot is taken. If this agent is not installed, you can still take a live snapshot, but the file systems might not be consistent.
The ability to take snapshots relies heavily on the storage backend. The Container Storage Interface (CSI) driver enables OpenShift to manage this feature on the storage side. If a disk is located on a backend that does not support snapshots, OpenShift will not be able to perform the snapshot operation.
Creating snapshots
You can verify that the underlying storage provider supports snapshots by checking the available snapshot classes.
$ oc get volumesnapshotclasses
NAME DRIVER DELETIONPOLICY AGE
ocs-storagecluster-rbdplugin-snapclass openshift-storage.rbd.csi.ceph.com Retain 3d
Create the snapshot manifest as follows, which defines a VirtualMachineSnapshot object and the name of the source VM.
apiVersion: snapshot.kubevirt.io/v1beta1
kind: VirtualMachineSnapshot
metadata:
name: snap-demo
spec:
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: rhel9-vm
Apply the snapshot creation manifest, monitor the status of the snapshot, and wait until it is ready for use.
$ oc apply -f snap-demo.yaml
virtualmachinesnapshot.snapshot.kubevirt.io/snap-demo created
$ oc wait rhel9-vm snap-demo --for condition=Ready
$ oc get vmsnapshot
NAME SOURCEKIND SOURCENAME PHASE READYTOUSE CREATIONTIME ERROR
snap-demo VirtualMachine rhel9-vm Succeeded true
17s
Restoring snapshots
Once the snapshot has been successfully created, it can be used to restore the VM.
apiVersion: snapshot.kubevirt.io/v1beta1
kind: VirtualMachineRestore
metadata:
name: demo-snap-restore
spec:
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: rhel9-vm
virtualMachineSnapshotName: snap-demo
Apply the restore manifest to restore the VM’s state.
$ oc create -f snap-restore.yaml
virtualmachinerestore.snapshot.kubevirt.io/demo-snap-restore created
$ oc get vmrestore
NAME TARGETKIND TARGETNAME COMPLETE RESTORETIME ERROR
demo-snap-restore VirtualMachine rhel9-vm true 11s
Cloning virtual machines
Cloning is the process of creating a copy of a virtual machine. Typically, an administrator prepares a single VM as a template, which is then used to deploy multiple identical machines quickly. This method is fast and reliable because it eliminates the need to manually install and configure the operating system for each new VM.
A clone of a VM can be created by using the VirtualMachineClone resource type.
apiVersion: clone.kubevirt.io/v1alpha1
kind: VirtualMachineClone
metadata:
name: demo-clone
spec:
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: rhel9-vm
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: rhel9-vm-clone
Apply the manifest to create a clone of the VM:
$ oc create -f demo-clone.yaml
virtualmachineclone.clone.kubevirt.io/demo-clone created
Backing up virtual machines in OpenShift
Along with snapshots and clones, OpenShift supports the creation of full backups of virtual machines. While these three data protection methods can seem similar, they serve distinct purposes. Snapshots capture a point-in-time image of a VM for quick recovery from a minor incident, clones are used to deploy multiple identical VMs from a template rapidly, and backups provide complete, restorable, independent copies of a virtual machine for long-term retention and disaster recovery.
Specialized tools are used to perform these backup operations. OpenShift API for Data Protection (OADP), Red Hat’s native data protection solution, is deeply integrated with the OpenShift platform. OADP version 1.4.1 supports backing up and restoring VM workloads in Red Hat OpenShift Virtualization. It uses components like Velero, an open-source tool for backing up Kubernetes resources, and Kopia, which provides file-level data protection. OADP’s primary strength lies in its ability to natively protect the application layer, including all associated Kubernetes objects, container images, and persistent volumes.
An alternative popular solution is Trilio, a specialized backup solution for OpenShift. Trilio offers a more comprehensive approach by providing a unified platform to protect application containers and virtual machines. This is a significant advantage for hybrid and mixed-workload environments because it allows administrators to manage data protection for containerized and virtualized workloads from a single, centralized interface. Trilio simplifies virtual machine backups in OpenShift.
Best practices for managing storage in Red Hat OpenShift virtualization
The following are some of the best practices for managing virtual machine storage in Red Hat OpenShift virtualization:
- Select an appropriate storage architecture: The core decision is to choose a storage approach that aligns with your needs. This could be external storage (leveraging existing arrays via CSI), an internal SDS solution (for a hyper-converged model), or shared-disk file systems (for specialized applications).
- Define appropriate storage classes: Define and create storage classes based on the performance requirements of your workloads. Use the appropriate storage class to ensure that virtual machines are provisioned with the correct storage performance tier for their workloads.
- Request necessary storage capacity: When creating persistent volume claims (PVCs), request the required capacity to avoid wasting resources. You can often expand a volume’s size later on, making this a cost-effective approach that aligns with the on-demand nature of cloud-native environments.
- Choose a suitable storage backend: Not all storage backends offer the same features. Creating efficient snapshots and clones depends on the underlying storage and its Container Storage Interface (CSI) driver, which is essential for rapid VM deployment and data protection.
- Utilize snapshots: Snapshots provide a fast and efficient way to create new VM clones for development and testing. This method is far quicker than duplicating entire disks and is ideal for spinning up multiple, identical environments on demand.
- Backup: Snapshots are not a replacement for a comprehensive backup strategy. Specialized tools like Trilio create independent, restorable copies of virtual machines that can be stored off-site for long-term retention and disaster recovery.
Conclusion
Ultimately, the move to Red Hat OpenShift Virtualization offers a fresh perspective on storage management. It shifts the focus from isolated VMs to integrating them seamlessly with cloud-native workflows. This unified approach allows administrators to handle storage for virtual machines and containers using the same tools and best practices. By making deliberate choices about your storage architecture and following recommendations for data protection, you can ensure that virtualized workloads are reliable and fully prepared for a modern, hybrid cloud environment.
Like This Article?
Subscribe to our LinkedIn Newsletter to receive more educational content