Implementing persistent storage for stateful applications running in containerized environments like OpenShift is a challenging problem. There is little doubt about the flexibility of containers, but their inherent disposability can lead to all sorts of issues for stateful workloads. To persist data for such workloads, you often need to rely on specialized storage solutions that can be set up in OpenShift.
One of the most popular solutions in this regard is OpenShift Data Foundation (ODF), which is Red Hat’s integrated, software-defined storage solution for OpenShift clusters. Built on Ceph, Rook, and NooBaa, it delivers persistent block, file, and object storage directly inside the Kubernetes ecosystem.
This article is meant for engineers who already understand container storage fundamentals and want a clear, practical checklist for deploying and running ODF in real OpenShift environments. If you see daily workload, performance tuning, and production uptime challenges, you’ll want to read on.
OpenShift Data Foundation summary deployment and operations checklist
The following table provides an overview of the key ODF deployment-to-operations lifecycle stages and their purpose.
Stage | Step | Purpose |
Deployment | Verify infrastructure and sizing | Confirm platform compatibility, and provision nodes with minimum CPU/RAM. |
Prepare local storage | Discover raw block devices and create persistent volume mappings. | |
Deploy storage system | Initialize the ODF cluster with proper capacity and select the right nodes. | |
Validate storage services | Verify that block, file, and object storage classes are operational. | |
Operations | Configure day 2 operations | Implement data lifecycle management and monitoring capabilities. |
Extend data Protection | Integrate CSI snapshots into developer workflows to allow for instant rollbacks and low-overhead data cloning for CI/CD testing. |
Automated Application-Centric Red Hat OpenShift Data Protection & Intelligent Recovery
Why OpenShift needs a dedicated storage solution like ODF
When you move from simple proofs of concept to production-grade OpenShift clusters, the “stateless” dream usually hits a wall. Most enterprise applications like databases, logging stacks, or private registries require persistent data. While Kubernetes has made great strides with the Container Storage Interface (CSI), relying on generic, external storage often creates a massive bottleneck for Day 2 operations.
Here is why a dedicated, software-defined storage layer like OpenShift Data Foundation (ODF) is a technical necessity for OpenShift:
Eliminating “external provider” friction
In a standard setup, whenever a developer needs a persistent volume (PV), the cluster has to talk to an external SAN or a cloud-managed disk. This often leads to “impedance mismatch.” You end up managing storage through one console and your containers through another. Bringing storage inside the cluster lets you treat disk capacity just like CPU and RAM—as a native resource that the OpenShift scheduler actually understands.
The RWX problem (ReadWriteMany)
Most cloud-native storage providers excel at providing ReadWriteOnce (RWO) access, where a single pod accesses a single disk. However, the moment you need a shared file system for web uploads or log aggregation (ReadWriteMany), external providers often struggle with performance or complex mounting logic. ODF uses Ceph under the hood to provide high-performance file, block, and object storage simultaneously, so you don’t have to hunt for a different storage vendor every time a new app requirement pops up.
Solving the node-failure bottleneck
In a typical cloud setup, if a worker node goes down, you usually have to rely on the cloud provider’s API to detach and then reattach a disk to a new node. That “volume-in-use” or “stuck re-mounting” phase is detrimental to meeting your recovery time objective (RTO).
By running an integrated storage mesh, the data isn’t sitting on some distant, external array; it’s replicated across the cluster’s own nodes. If a node drops, the replacement pod doesn’t have to wait for a slow hardware handshake or network-wide data “travel” time. The bits are already present on the surviving peers in the storage cluster, turning what used to be a multi-minute recovery into a few seconds of container startup.
Consistent experience across clouds
If you’re running a hybrid deployment, for example, OpenShift on-prem (VMware / bare metal) alongside public cloud instances in AWS, the underlying storage behavior is entirely different. One uses EBS, the other uses vSphere CNS. A dedicated storage layer like ODF abstracts that hardware away. Your YAML manifests stay the same regardless of where the cluster lives because the storage API is consistent.
Automated Red Hat OpenShift Data Protection & Intelligent Recovery
Perform secure application-centric backups of containers, VMs, helm & operators
Use pre-staged snapshots to instantly test, transform, and restore during recovery
Scale with fully automated policy-driven backup-and-restore workflows
OpenShift Data Foundation architecture
Under the hood, ODF consists of a sophisticated stack of proven open-source projects, primarily Ceph, Rook, and NooBaa, orchestrated to behave as a single storage entity.
The ODF technical stack
To get why ODF is so resilient, you have to look at the three specific engines running the show:
- Ceph: This is the foundational layer. It’s a distributed, software-defined storage cluster that treats a group of independent nodes like a single, massive pool of capacity. It handles all the background work, such as data replication, self-healing, and ensuring that even if you lose a disk (or an entire rack), your applications don’t feel the impact.
- Rook: If Ceph is the engine, Rook is the driver. Managing a raw Ceph cluster in a containerized environment is notoriously difficult. Rook acts as the Kubernetes operator, automating the deployment, scaling, and upgrades of Ceph. It translates your high-level intents (like “I need a 50 GB volume”) into actual Ceph configurations.
- NooBaa (Multicloud Object Gateway): This additional unit handles the object storage abstraction. It allows you to provide an S3-compatible interface to your developers regardless of where the data actually sits, whether it’s on local NVMe drives or tucked away in a public cloud bucket. It effectively decouples the application’s storage calls from the underlying infrastructure provider.
Consolidating block, file, and object storage
The real power of ODF lies in how it partitions a single set of raw physical disks into three distinct storage types. You don’t need a different vendor for each; you just use a different StorageClass:
- Block storage (RBD): For databases like Postgres or MongoDB, every millisecond of latency is a dealbreaker. ODF handles these IOPS-heavy workloads via Ceph’s RADOS Block Device (RBD). ODF uses Ceph’s RADOS Block Device (RBD) to create and map block volumes directly to a pod using the ReadWriteOnce (RWO) mode. This gives the single pod exclusive, high-speed access to the volume, with superior performance.
- File storage (CephFS): When you have a shared file system where multiple pods across different nodes need to read and write to the same directory simultaneously (ReadWriteMany/RWX), ODF uses CephFS. It’s a distributed file system that scales much better than traditional legacy NFS setups in a cloud-native environment.
- Object storage (RGW and NooBaa): ODF leverages Ceph’s RADOS Gateway (RGW) and NooBaa to provide S3-compliant object storage. This is a good option for modern applications that communicate via APIs instead of mounting drives. ODF uses Ceph’s RADOS Gateway (RGW) for local high-speed object needs and NooBaa for flexible, policy-based bucket management across hybrid environments.
The CSI abstracts the entire storage backend, so your developers don’t have to understand how Ceph works. They can request a standard persistent volume claim (PVC), and the ODF operator handles all the work in the background, including provisioning the volume, handling encryption, and ensuring data replication. It turns storage into a self-service utility, so the infra team isn’t stuck manually carving out LUNs or managing mount points every time a new app goes live.
ODF deployment modes
ODF offers significant flexibility in how you roll out the storage layer. You’re essentially choosing between a hyper-converged model or a decoupled, external backend.
Internal mode
In this setup, ODF runs entirely within the OpenShift cluster as a set of pods. It uses the Local Storage Operator (LSO) to “claim” and aggregate disks, whether they are local NVMe drives, SAN volumes, or cloud-native disks such as EBS or vSphere volumes. This approach is practical when:
- Your cluster’s storage growth isn’t yet precisely mapped, and you need to scale resources dynamically.
- You don’t have the luxury of dedicated infrastructure nodes or “storage-only” hardware.
- Provisioning additional node instances, especially on bare-metal servers, is a major logistical or financial hurdle.
External mode
In an external deployment, ODF connects to a standalone Red Hat Ceph Storage cluster outside the OpenShift environment. This approach treats storage as an independent service and is recommended when:
- The storage requirements are massive enough that they would compete with applications for CPU and RAM if run internally.
- You are managing a multi-cluster environment in which several OpenShift clusters need to consume storage from a single central source.
- A dedicated storage team is managing the hardware, allowing you to decouple the storage lifecycle from your Kubernetes maintenance windows.
The following figure illustrates the architecture of ODF, covering both internal and external mode deployment strategies, and shows how the CSI drivers bridge the gap between your application pods and the underlying storage.
Architecture of OpenShift Data Foundation
Validating infrastructure and sizing: Preparing a cluster for OpenShift Data Foundation
Deploying ODF isn’t a “one-size-fits-all” task. You can’t just follow a generic guide, because a bare-metal setup with local NVMe drives requires an entirely different mindset and a different provisioner than a managed cloud environment like ROSA or GCP. Therefore, it is recommended to audit your platform’s specific storage constraints and supported modes to avoid hitting a configuration dead end mid-install. The underlying infrastructure should dictate the architecture for you.
Platform requirements
The following table summarizes the supported deployment modes and recommended storage provisioners across the major infrastructure providers:
Platform | Supported Modes | Storage Provisioner / Requirements |
Amazon EC2 | Internal only | Supports gp2-csi and gp3-csi; gp3 is recommended for high throughput. D2/D3 instances are best for high IOPS. |
Bare Metal | Internal and external | Requires Local Storage Operator (LSO). Production nodes need at least 0.5 TiB per storage device. |
VMware vSphere | Internal and external | Recommended for vSphere 8.0+. Uses vsphere-volume for vSAN/VMFS or LSO for direct VMDK/DirectPath access. |
Microsoft Azure | Internal only | Requires a storage class providing Azure Disks via the azure-disk provisioner. |
Google Cloud | Internal only | Requires GCE Persistent Disks via the gce-pd provisioner. |
IBM Power / Z | Internal and external | Requires LSO for local SSD/SAN. IBM Z supports external mode specifically when Ceph runs on x86. |
ROSA (HCP) | Internal only | Strictly uses AWS EBS volumes via the gp3-csi provisioner. |
Validating the backend for external mode deployment
For external mode deployments, you need to validate the integration of RedHat Ceph Storage cluster with ODF. If your Red Hat Ceph Storage (RHCS) version isn’t perfectly aligned with your ODF release, the CSI drivers will likely run into API mismatches or silent communication failures. Before you commit to an external architecture, verify that your specific Ceph version is validated for the ODF 4.20 lifecycle.
To ensure that your external Red Hat Ceph Storage (RHCS) cluster is officially supported for your ODF release, use the Red Hat Supportability and Interoperability Checker:
- Service Type: Set this to ODF as Self-Managed Service.
- Version Selection: Select the appropriate ODF version from the dropdown.
- Compatibility Mapping: Under the Versions tab, click Supported RHCS Compatibility to see the validated Ceph releases for that build.
The following figure illustrates the utility of the Red Hat Interoperability Checker in verifying infrastructure compatibility.
Validating ODF design via the Red Hat Interoperability Checker
Resource requirements overview
ODF services carry their own overhead, so you need to account for the overhead that ODF requires on top of your existing OpenShift footprint. These numbers represent what the ODF services themselves consume; they don’t include the resources needed for your actual application workloads.
A standard production rollout typically follows the “rule of three,” where services are spread across three nodes (one per failure domain) to satisfy pod-placement rules. The following table breaks down the aggregate requirements for the ODF base services.
Deployment Mode | Total CPU (Logical) | Total Memory (RAM) | Storage Devices |
Internal (Standard) | 30 units | 72 GiB | 3 devices |
Internal (Minimum) | 24 units | 72 GiB | 3 devices |
External | 4 units | 16 GiB | N/A |
Compact (3-node) | 24 units | 72 GiB | 3 devices |
In addition, if you plan to enable the NFS service for external access, you’ll need to reserve an additional 3 CPUs and 8 GiB of RAM.
When calculating these requirements, keep in mind that 1 CPU unit in the Red Hat documentation maps directly to a Kubernetes CPU unit:
- For non-hyperthreaded CPUs, 1 unit = 1 physical core.
- For hyperthreaded CPUs, 2 units = 1 physical core.
It’s always safer to overprovision slightly. If your worker nodes are already running at high utilization, ODF might struggle to maintain data replication speeds during peak IO periods.
Storage device guidelines
To keep your cluster performant and reliable, keep these in mind:
- The “rule of three”: Always expand your cluster in multiples of three. By placing one node in each failure domain (or rack), you ensure that ODF can satisfy its internal pod-placement and data-replication rules.
- Dedicated disks: Each node requires at least two disks, one dedicated solely to the operating system and the others reserved for ODF.
- Device limits: While you can scale up, it is generally recommended to keep the number of devices per node to 12 or fewer. This will let you avoid hitting the cloud provider’s limits on attached volumes and keep data recovery times manageable if a node ever goes down.
- Configure failure domains: To ensure data resilience, configure ODF to recognize failure domains (such as different racks or zones). By implementing anti-affinity rules, the system ensures that three-way replication spreads data across these domains, allowing the cluster to survive a hardware failure without data loss.
Capacity and scaling
The way you provision storage depends on whether you are using dynamic cloud volumes or local physical disks:
- Dynamic storage (cloud): You can typically request sizes of 0.5 TiB, 1 TiB, 2 TiB, 4 TiB, or 8 TiB. Note: You can only expand capacity in increments of the size you selected at installation. If you start with 2 TiB disks, your expansion sets must also be 2 TiB.
- Local storage (bare metal / VMware): You can use disks up to 16 TiB. For the sake of cluster balance and predictable performance, ensure that all disks in a set are the same size and type (e.g., all 1.9 TB NVMe).
Deploying OpenShift Data Foundation
We’re now going to see how to set up ODF services in internal mode in our OpenShift cluster. For this demo, the OpenShift cluster is running on VMware.
Please ensure that the following are true:
- Each of the selected nodes must have at least one raw block device available for use by ODF.
- The block devices must be empty and must not contain any existing configurations.
- For VMware, ensure that the disk.EnableUUID option is set to TRUE for each VM.
Operator installation
Log into your OpenShift Web Console with cluster-admin privileges and follow these steps to get the necessary operators running:
- OpenShift Data Foundation Operator:
- Navigate to Operators > OperatorHub.
- Filter for OpenShift Data Foundation.
- Select the Operator, click Install, and accept the default settings (stable channel, cluster-wide).
- Local Storage Operator:
- Return to OperatorHub and filter for Local Storage.
- Select the operator and click Install, again using the default settings.
Preparing local storage for ODF
Next, we need to tell the LSO how to “claim” the disks on your nodes so they can be handed over to ODF. This involves two main tasks: discovery and provisioning.
Discovery
- Disk discovery
You need to identify which disks are actually available across your worker nodes without manually logging into every machine:
- Go to Installed Operators > Local Storage > Local Volume Discovery.
- Create a Local Volume Discovery instance and select the specific nodes you’ve designated for ODF.
- The Operator will scan these nodes. You can verify the results via the CLI to see which drives are ready for use:
- Identifying reliable device IDs
When you look at the discovery results, you’ll see disks listed with standard paths like /dev/sdb.
A word of caution: Avoid using /dev/sdX paths. These are not stable; if a node reboots, the OS might discover drives in a different order, potentially breaking your storage configuration.
Best practice: Always use the unique Device ID (found in the discovery details). These IDs are persistent and tied to the hardware itself, ensuring that your cluster remains stable across reboots and maintenance windows.
$ oc get localvolumediscoveryresults NAME AGE discovery-result-ocpnode-1 1m discovery-result-ocpnode-2 1m discovery-result-ocpnode-3 1m
$ oc describe localvolumediscoveryresults discovery-result-ocpnode-1 |less [...............] Device ID: /dev/disk/by-id/wwn-0x6000c29c01d91ed7b7109f82c40d42e7 Fstype: Model: Virtual disk Path: /dev/sdc Property: Rotational Serial: 6000c29c01d91ed7b7109f82c40d42e7 Size: 107374182400 Status: State: Available [.................]
Once you’ve mapped out your physical drives and secured those persistent Device IDs, the final phase is bridging that hardware into the OpenShift ecosystem. This is where the Local Volume Set meets the ODF Storage System to create the actual usable storage classes for your applications.
Provisioning the local volumes
With your Device IDs in hand, you need to bind them into a storage class that ODF can consume.
- Create Local Volumes: In the console, navigate to Installed Operators > Local Storage > Local Volume and click Create Local Volume.
- Map the Paths: Assign a name to your volume set. Under StorageClassDevices > devicePaths, paste the unique Device IDs you identified during the discovery phase. If you have multiple drives, use Add devicePaths to include them all.
- Define the Class: Set your storageClassName and ensure the volumeMode is set to Block. Leave the remaining defaults and hit create.
- Verification: Once the process finishes, verify that a new StorageClass exists and that individual Persistent Volumes (PVs) have been automatically generated for each physical disk.
The following figure demonstrates the configuration and provisioning of local storage devices for use within the cluster.
Configuration of local storage devices via the Local Storage Operator
$ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE lso kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 1h $ oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-3cfceaf 200Gi RWO Delete Bound openshift-storage/ocs-deviceset-lso-0-data-4fgmfr lso 1h local-pv-9ccedf9f 200Gi RWO Delete Bound openshift-storage/ocs-deviceset-lso-0-data-5zz5lk lso 1h local-pv-a862d6ec 200Gi RWO Delete Bound openshift-storage/ocs-deviceset-lso-0-data-15bsz7 lso 1h local-pv-b98bce49 1200Gi RWO Delete [.........................]
Deploying the ODF storage system
Now that the underlying local volumes are ready, you can initialize the core ODF cluster:
- Initialize: Go to Installed Operators > OpenShift Data Foundation > Storage System and click Create Storage System.
- Select Strategy: Choose Full deployment and Use an existing StorageClass. Select the StorageClass you just created in the previous step.
- Capacity & Nodes: Enter your Requested Capacity and select the specific worker nodes where your disks are physically attached.
- Finalize: You can typically stick with the default Security, Network, and Data Protection settings for a standard rollout. Proceed to create the system.
The following figures outline the creation of the storage system and the initial cluster configuration within the platform.
Storage system creation
Defining backing storage for ODF cluster
Defining backing storage deployment options
Defining ODF cluster capacity
Defining storage system parameters
It takes a few minutes for the openshift-storage project to pull images and spin up the Ceph mon, osd, and mgr pods. You can monitor the progress by watching the pods in that namespace. Once the storage cluster reports a Ready status, your environment is live.
$ watch oc get storagecluster,pods -n openshift-storage
Once the storage cluster has been created successfully, the command will report Ready status.
$ oc get storagecluster -n openshift-storage NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 10m Ready 2026-02-11T11:3:06Z 4.18.16
This can also be verified from the console.
Successful ODF cluster creation
Validating ODF storage services: Block, file, and object use cases
Upon successful deployment, ODF provides four distinct storage classes, each designed for specific architectural needs.
$ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE lso kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 1h ocs-storagecluster-ceph-rbd openshift-storage.rbd.csi.ceph.com Delete Immediate true 1h ocs-storagecluster-ceph-rgw openshift-storage.ceph.rook.io/bucket Delete Immediate false 1h ocs-storagecluster-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 1h openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 1h
The recommended use cases of these storage classes are listed below.
| Storage Class | Backing Technology | Primary Use Case |
| ocs-storagecluster-ceph-rbd | Ceph Block Device | High-performance RWO workloads (databases, transactional apps) |
| ocs-storagecluster-cephfs | Ceph File System | Shared, distributed RWX storage (logging, data aggregation, file share services, backup archives) |
| openshift-storage.noobaa.io | Multicloud Object Gateway | S3-compatible API endpoint for abstracting multiple cloud stores |
| ocs-storagecluster-ceph-rgw | Ceph RADOS Gateway | On-premises object storage for data-intensive S3 applications |
Once the cluster is healthy, you can start putting that storage to work. The ocs-storagecluster-ceph-rbd storage class is typically your workhorse for high-performance, block-backed volumes.
Examining the RBD storage class
To understand exactly how ODF is handling your data requests, you can inspect the storage class definition. This gives you a look at the CSI parameters and the underlying Ceph pool configuration.
$$ oc describe sc ocs-storagecluster-ceph-rbd Name: ocs-storagecluster-ceph-rbd IsDefaultClass: No Annotations: description=Provides RWO Filesystem volumes, and RWO and RWX Block volumes,storageclass.kubernetes.io/is-default-class=true Provisioner: openshift-storage.rbd.csi.ceph.com Parameters: clusterID=openshift-storage,csi.storage.k8s.io/controller-expand-secret-name=rook-csi-rbd-provisioner,csi.storage.k8s.io/controller-expand-secret-namespace=openshift-storage,csi.storage.k8s.io/fstype=ext4,csi.storage.k8s.io/node-stage-secret-name=rook-csi-rbd-node,csi.storage.k8s.io/node-stage-secret-namespace=openshift-storage,csi.storage.k8s.io/provisioner-secret-name=rook-csi-rbd-provisioner,csi.storage.k8s.io/provisioner-secret-namespace=openshift-storage,imageFeatures=layering,deep-flatten,exclusive-lock,object-map,fast-diff,imageFormat=2,pool=ocs-storagecluster-cephblockpool AllowVolumeExpansion: True MountOptions: ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: none
The output reveals several critical operational details:
- IsDefaultClass: No: While it’s a primary class, it won’t be used for every generic PVC unless you explicitly mark it as the default for the cluster.
- Provisioner: This uses openshift-storage.rbd.csi.ceph.com, signaling that the Ceph RBD CSI driver is managing the lifecycle of these volumes.
- Parameters: Key values like pool=ocs-storagecluster-cephblockpool and fstype=ext4 tell you exactly where the data lives in Ceph and how the filesystem is formatted upon creation.
- AllowVolumeExpansion: True: This is vital for Day 2 operations; you can grow your volumes on the fly without unmounting them from your pods.
- Reclaim Policy: Delete: By default, ODF will clean up the backend Ceph image as soon as the PVC is deleted in OpenShift to prevent “orphaned” storage.
Let’s validate the volume provisioning and create a test volume using a standard manifest:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: ocs-storagecluster-ceph-rbd
volumeMode: Filesystem
resources:
requests:
storage: 1GiApply the manifest and verify the volume creation.
$ oc apply -f test-pvc.yaml persistentvolumeclaim/test-pvc created $ oc get pvc -n default NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE block-pvc Bound pvc-34c76ey6-7845-4e46-8f01-04f8a69f3d43 1Gi RWO ocs-storagecluster-ceph-rbd 12s $ oc get pv |grep test-pvc pvc-34c76ey6-7845-4e46-8f01-04f8a69f3d43 1Gi RWO Delete Bound default/block-pvc ocs-storagecluster-ceph-rbd 35s
At this point, the backend storage is fully carved out and ready. Your application pods can simply reference the test-pvc in their volume definitions and start consuming storage.
Examining the CephFS storage class
For workloads that need to share a common directory across multiple nodes, like distributed logging or a central content repo, you need ReadWriteMany (RWX) support. This is the primary use case for ocs-storagecluster-cephfs. While RBD is perfect for high-speed block storage, it’s limited to a single node at a time; CephFS, however, is a distributed file system that manages concurrent read/write operations across your entire cluster.
If you pull the metadata for this storage class, you’ll see a different backend driver:
$ oc describe storageclass ocs-storagecluster-cephfs Name: ocs-storagecluster-cephfs IsDefaultClass: No Annotations: description=Provides RWO and RWX Filesystem volumes Provisioner: openshift-storage.cephfs.csi.ceph.com Parameters: clusterID=openshift-storage,csi.storage.k8s.io/controller- expand-secret-name=rook-csi-cephfs-provisioner,csi.storage.k8s.io/controller- expand-secret-namespace=openshift-storage,csi.storage.k8s.io/node-stage- secret-name=rook-csi-cephfs-node,csi.storage.k8s.io/node-stage-secret- namespace=openshift-storage,csi.storage.k8s.io/provisioner-secret-name=rook-csi- cephfs-provisioner,csi.storage.k8s.io/provisioner-secret-namespace=openshift- storage,fsName=ocs-storagecluster-cephfilesystem AllowVolumeExpansion: True MountOptions: ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: none
The output reveals the following:
- Cephfs uses openshift-storage.cephfs.csi.ceph.com. Instead of mapping a raw block device, this provisioner creates subvolumes within the Ceph File System (fsName=ocs-storagecluster-cephfilesystem).
- The most important feature here is RWX support. This allows you to scale your deployment across multiple replicas while ensuring that every pod sees the same directory structure and files.
- Like RBD, it supports immediate volume binding and online expansion. You don’t have to wait for a pod to be scheduled before provisioning the volume, and you can grow the shared space as your data grows.
To consume storage from CephFS, we’ll need to create a PVC that explicitly requests RWX mode from the CephFS storage class.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: ocs-storagecluster-cephfs
resources:
requests:
storage: 1GiExamining the object storage classes
When you’re dealing with large scale archives, media or unstructured data that doesn’t need a traditional filesystem hierarchy, you move into object storage. Instead of mounting a drive, your apps talk to an S3 API endpoint. This is the “flat” storage model that scales almost infinitely.
For object storage, ODF gives you the flexibility to choose your provider based on where you want that data actually to live.
Ceph RADOS Gateway (RGW)
If you need high-performance object storage that stays strictly on your local NVMe or SSD tiers, Ceph RGW is the play. It’s a RESTful gateway that sits directly on top of your Ceph cluster, providing a native S3 interface without any external dependencies.
- Storage Class: ocs-storagecluster-ceph-rgw
- When to use it: Local database backups, high-speed logging, or any application where data sovereignty and low latency are the top priorities
Multicloud Object Gateway (NooBaa)
For environments that bridge local data with the public cloud, NooBaa (MCG) is the standard. It acts as an abstraction layer, a gateway that can consume storage from AWS, Azure, or other S3-compatible backends and present them as a single, unified bucket to your developers.
- Storage Class: openshift-storage.noobaa.io
- When to use it: Hybrid-cloud tiering, disaster recovery syncs to the cloud, or when you want to decouple your application’s S3 endpoint from the underlying physical hardware
Provisioning with object bucket claims (OBC)
For consuming object storage, the equivalent to a PVC is an object bucket claim (OBC). When you create an OBC, the ODF operator (NooBaa or RGW) creates a ConfigMap and a secret in the OBC namespace. These two resources act as the bridge between
apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: archive-storage-obc namespace: my-obc-project spec: generateBucketName: app-archive storageClassName: openshift-storage.noobaa.io
$ oc create -f obc.yaml objectbucketclaim.objectbucket.io/archive-storage-obc created $ oc get objectbucketclaims NAME STORAGE-CLASS PHASE AGE archive-storage-obc openshift-storage.noobaa.io Bound 6s oc get objectbuckets NAME STORAGE-CLASS ... PHASE obc-default-archive-storage-obc openshift-storage.noobaa.io ... Bound
As soon as the OBC resource is created, the following resources are created to handle the lifecycle and security of the storage:
- Object bucket (OB): This is the cluster-scoped representation of the actual S3 bucket. It tracks the backend metadata and ensures that the physical storage exists on either NooBaa or Ceph RGW.
- ConfigMap (CM): The CM contains the S3 endpoint (host and port) and the specific bucket name. Since this is non-sensitive routing data, it’s stored in plain text for easy injection.
- Secret: The secret contains the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. ODF generates these unique credentials specifically for the object claim, ensuring that your application follows the principle of least privilege.
$ oc extract configmap/archive-storage-obc --to=- # BUCKET_HOST s3.openshift-storage.svc # BUCKET_PORT 443 # BUCKET_NAME archive-storage-obc-14f6ce1f-11ac-67ec-b8a5-1a74427cfb1 [.....] oc extract secret/archive-storage-obc --to=- # AWS_ACCESS_KEY_ID YZDsbNPnG1o23AGDZprt # AWS_SECRET_ACCESS_KEY xjreDRhswx3lfrdA8WqzoUxpiRYuyjc2uPlWpMw3
The environment variables from the configuration map and secret can then be referenced inside the application manifest to consume object storage.
Configuring ODF Day 2 operations
Once your storage is provisioned and the applications are running, your focus shifts to data lifecycle management. In ODF, snapshots and clones are the two primary tools for ensuring data recoverability and streamlining your dev/test workflows without overhead.
Data recovery with volume snapshots
A volume snapshot is a point-in-time copy of your data. Think of it as a “safety net” before performing risky operations, such as database migrations or application upgrades. If something goes sideways, you can restore the persistent volume to its exact state at the moment the snapshot was taken.
The Ceph RBD and CephFS storage classes have CSI drivers that provide the snapshot functionality for PVCs.
$ oc get volumesnapshotclasses NAME DRIVER ocs-storagecluster-rbdplugin-snapclass openshift-storage.rbd.csi.ceph.com ocs-storagecluster-cephfsplugin-snapclass openshift-storage.cephfs.csi.ceph.com
You can create a snapshot of a PVC by referencing the source PVC in the persistentVolumeClaimName parameter and specifying the volume snapshot class for Ceph RBD or CephFS.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: pvc-data-snapshot
spec:
volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass
source:
persistentVolumeClaimName: pvc-dataAfter the volume snapshot is created, a new PVC can be created from it. The volume snapshot must be referenced in the dataSource parameter of the new PVC.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-data-restore
labels:
app: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: ocs-storagecluster-ceph-rbd
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: pvc-data-snapshot
resources:
requests:
storage: 1GiTesting and debugging with volume clones
While snapshots are for recovery, clones are for utility. A volume clone is a duplicate of an existing persistent volume and is invaluable when you need to debug a production issue using real data without actually touching the production environment.
You can create a clone of a PVC by referencing the source PVC in the dataSource parameter. You need to reference the source PVC from which to take the clone. Once the clone has been created, it can be used just like any other PVC.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-pvc-clone
labels:
app: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: ocs-storagecluster-ceph-rbd
dataSource:
kind: PersistentVolumeClaim
name: data-pvc
resources:
requests:
storage: 1GiDynamic volume expansion
One of the most common Day 2 requests is the need for more space as a database or log repository grows. In ODF, you can expand a PVC on the fly without having to delete the volume or restart your application.
This capability is controlled by the allowVolumeExpansion attribute in the StorageClass. In ODF, both the RBD and CephFS storage classes have this set to true by default.
To trigger an expansion, you modify the spec.resources.requests.storage value in your existing PVC manifest.
$ oc edit pvc/data
[......]
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 200Gi
storageClassName: ocs-storagecluster-cephfs
volumeMode: Filesystem
volumeName: pvc-652f16c-2d09-4dc1-ae09-27034a5ded8c
[......]Monitoring the ODF cluster
To truly maintain a healthy ODF environment, you have to move beyond just provisioning and into active performance monitoring. Red Hat OpenShift provides a deep, integrated observability stack that surfaces these storage-specific metrics directly within the administrator console.
Monitoring ODF cluster health
You can also see performance data for metrics such as IOPs, throughput and latency.
Monitoring the performance of the ODF cluster
To check the health of your storage pools, navigate to Storage > Data Foundation in the Administrator perspective of the OpenShift web console. Under the Storage Systems tab, click on your specific storage system to view the Status card, which provides a real-time health indicator and any active alerts for the underlying Ceph pools.
Monitoring the health of storage pools
Extending ODF data protection with Trilio
For complex enterprise environments, the resilient storage layer provided by ODF is improved by complementing it with Trilio, an application-centric data protection solution. ODF handles the physical availability of data, while Trilio provides the management framework needed for full-spectrum disaster recovery and application mobility.
Application-centric data protection
Trilio extends ODF’s basic CSI snapshot capabilities by capturing more than just raw data volumes. It groups persistent volumes, Kubernetes metadata (such as ConfigMaps, Secrets, and Services), and associated objects into a single, restorable Application Vault. This ensures that when you restore an application, it comes back as a fully functional unit, not just a collection of disconnected disks.
High-speed local backups
One unique advantage of Trilio’s integration is the ability to use ODF’s native Multicloud Object Gateway (NooBaa) as the primary storage target for Trilio’s backups. By targeting a local S3-compatible bucket, backup and recovery traffic remains internal to the cluster. This significantly reduces recovery time objectives (RTO) by avoiding the latency and egress costs associated with external cloud storage.
Seamless multi-cluster mobility
Beyond simple backups, Trilio enables true application mobility across the hybrid cloud. By abstracting the underlying ODF storage layer, Trilio allows you to migrate entire workloads between different OpenShift clusters, for instance, moving a stateful application from an on-premises datacenter to AWS. Because Trilio packages the data and the application’s “brain” (the YAML manifests) together, the application can be re-provisioned on any ODF-backed cluster with minimal configuration changes.
Recommended practices for running ODF
The following are some of the recommended practices for running ODF:
- Finalize topology before deployment: Evaluate the workload requirements early to decide between internal and external mode deployments.
- Allocate dedicated resources: Validate that every node meets the specific CPU and RAM requirements for OSDs to prevent out-of-memory events and maintain consistent I/O throughput.
- Configure failure domains: Implement affinity and anti-affinity rules to spread data across different racks or availability zones, ensuring that three-way replication can survive a hardware failure.
- High speed storage: Use high-speed storage devices as a backend for improved performance.
- Automate via OBCs: Standardize on Object Bucket Claims (OBC) for all S3 traffic to automate credential rotation and ensure application portability.
- Validate snapshots: Periodically test your recovery path by spinning up a new PVC from an existing snapshot to confirm data integrity.
Learn How To Best Backup & Restore Virtual Machines Running on OpenShift
Conclusion
Building a production-grade Kubernetes strategy means moving away from fragmented, hardware-dependent storage. Red Hat OpenShift Data Foundation (ODF) addresses this need by providing a software-defined abstraction that treats block, file, and object storage as native, first-class citizens. Regardless of the storage requirements of the workload, ODF provides a single, unified interface that scales with the cluster.
The real value here is decoupling application logic from the underlying infrastructure. Automating the provisioning and lifecycle of these storage classes means that cluster administrators no longer have to manually manage individual disk mappings. This creates a consistent storage experience across on-premises datacenters and the public cloud, ensuring that data remains as portable and agile as the containerized applications themselves.
Like This Article?
Subscribe to our LinkedIn Newsletter to receive more educational content