Modern applications are heavily data-driven, especially if they incorporate AI/ML—they consume and generate terabytes or even petabytes of data every day. Running such applications requires access to different storage services, typically object or block storage. For example, an online analytical processing solution may require fast block storage, while an image analysis service may require scalable storage for the images.
Using the open-source cloud computing platform OpenStack for such data-intensive workloads requires understanding the platform’s storage services, so you can address these different storage needs.
This article covers OpenStack’s storage services, including Swift, Cinder, Manila etc. It also discusses Ceph, a universal backend that can be used for all OpenStack services. The article ends with a high-level coverage of OpenStack storage integration, best practices, performance, security, and monitoring.
Summary of key OpenStack storage services
OpenStack has a decoupled and pluggable storage architecture, meaning that there are multiple services that you can use for any storage type and backend, depending on the requirement. We cover these in more detail later in this article, but here’s a table that breaks down this information into something more manageable. It provides an overview of storage services to choose from, depending on the feature and use case.
| Feature | Nova | Cinder | Swift | Manila | Ceph (not an OpenStack service) |
| Storage type | Ephemeral | Block | Object | File | Block, object, file |
| Snapshot management | Yes | No | Yes | Yes | |
| Multiple Backends | Yes | Yes | No | Yes | Yes |
| Protocols | RESTful API | iSCSI, Fibre Channel, NFS, etc. | HTTP/HTTPS | NFS, SMB, CIFS | HTTPS, RADOS, iSCSI, NFS, CephFS |
| Multi-tenancy | Yes | Yes | Yes | Yes | Yes |
A brief history of OpenStack
OpenStack started as a joint project between the US National Aeronautics and Space Administration (NASA) and Rackspace. The idea was to create an “infrastructure service” platform that abstracted away computing, networking, and storage to run NASA.net and other applications.
Over the years, OpenStack has been leveraged to run telecommunications backhaul networks, global in-vehicle entertainment systems, and streaming platforms. It is one of the preferred choices for private clouds and an option for those companies wanting a private on-premises answer to a public cloud platform like AWS.
OpenStack is governed and managed by the OpenStack Foundation, which changed its name to the OpenInfra Foundation in 2021. The foundation hosts many other open-source projects that create infrastructure technology to meet modern application needs like containers, AI/ML, CI/CD, edge computing, etc. Today, the OpenInfra Standard, which uses the Linux OpenStack Kubernetes Infrastructure (LOKI) projects, drives modern infrastructure.
Since 2018, OpenStack has grown in use by 350%, with 45 million compute cores running in production. Nvidia and many other companies also use OpenStack to handle GPU AI workloads.
OpenStack storage concepts
OpenStack provides multiple services that cater to different storage requirements. Apart from the default Nova-provisioned block storage—which is ephemeral or non-persistent—there is Swift, Cinder, and Manila, the most common storage services that provide persistent storage capabilities. OpenStack storage services provide three common storage types: object, block, and file storage. OpenStack also supports many storage backends that support all of these types.
OpenStack storage types
Ephemeral storage is what VM instances in OpenStack use by default. Nova manages it and is part of the instance provisioning process, where it creates a boot volume from the Glance image. “Ephemeral” means that once you terminate a VM, all data on that disk is also lost. Simply put, this type of storage is ideal for temporary data such as OS files, cache, etc.
With persistent block storage, the data stored remains or lives beyond the instance’s lifecycle. These volumes are provisioned independently and are typically part of a server or hardware decoupled from the server where the instance is running. You can use this type of block storage for databases, critical files, etc. The Cinder service provides persistent block storage.
The building blocks of OpenStack storage
OpenStack supports the usual object, block, and file storage types. As mentioned earlier in the article, various OpenStack services provide these storage types through APIs. In this section, we will cover the basics of these storage types and cover the services in more detail later.
Object storage
Object storage is a persistent storage solution for storing static data at scale with high availability and redundancy. Data is stored and retrieved using HTTP(S) URLs instead of the traditional I/O read-write usually seen with block storage. Object storage solutions use distributed architecture and store multiple copies of data across many servers and disks for redundancy. It is excellent for unstructured data such as files, PDFs, images, videos, backups, etc.
The service that enables object storage in OpenStack is called Swift.
File-based storage
File-based storage or shared file systems are also persistent storage solutions that allow storage to be shared across multiple machines or instances simultaneously. These file systems, or shares, can be mounted on the VM instances for data storage, sharing, and exchange.
OpenStack’s shared file system service is called Manila.
Block storage
Block storage is the usual hard disk storage that we are familiar with on our bare-metal servers, laptops, etc. It is formatted into a filesystem and requires an operating system for reading and writing data.
Block storage is available in both ephemeral and persistent forms in OpenStack. The volumes created by Nova are ephemeral, whereas the ones created by a service like Cinder are persistent. If you are familiar with public cloud platforms, note that persistent block storage is similar to AWS EBS or Azure disks.
Find out how Vericast solved K8s backup and recovery with Trilio
OpenStack storage services
OpenStack provides a rich ecosystem of storage services that cater to various needs within a cloud infrastructure. These services offer scalability, flexibility, and high availability, making them suitable for a wide range of workloads. The following are the primary storage types available in OpenStack.
Nova-provisioned block storage
Nova is the OpenStack service that provides compute instances or virtual machines. Nova-provisioned block storage is ephemeral and tightly coupled with the instance’s lifecycle. In this approach, the VM image is copied from Glance and stored in /var/lib/nova/_base if it doesn’t exist. Next, an instance is created in /var/lib/nova/uuid with a thin provisioned overlay QCOW2 file referencing the base image copied from Glance. Changes to the instance are written to this layer and are lost when the instance is terminated or stopped and started.
This combination of a reference to the base image and the overlay file forms the boot disk, which contains the root file system or the operating system from which the instance boots. The flavor of an instance defines the size of the overlay file.
Illustration of the Nova provisioned block storage (source)
In addition to the root volume, flavors can provide an additional ephemeral block device. It is represented as a raw block device with no partition table or file system. A cloud-aware operating system can discover, format, and mount such a storage device. Nova defines the default file system for different operating systems as ext4 for Linux distributions, VFAT for non-Linux and non-Windows operating systems, and NTFS for Windows. However, it is possible to configure other file system types.
Cinder
Cinder is the name of the service that provides persistent block storage in OpenStack, primarily for compute instances. The storage created is called a volume and has an independent lifecycle, so data written to it persists beyond the lifecycle of instances. You can mount and unmount it as needed.
Illustration of Cinder’s operation (source)
You can use Cinder volumes as a primary root disk or secondary bootable disk and mount it to only one instance at a time. If needed, you must unmount and mount on another instance.
You can configure a persistent volume for the bootable disk containing the root file system and use it to provide a persistent store for OS files. Its state is maintained even if you stop or terminate the instance (preserving the disk).
Cinder provides the following resources:
- Volumes are attached to instances for boot disk or additional disk.
- Snapshots are read-only copies of the volume at a certain point in time. You can use snapshots to create volumes from them.
- Backups are copies of the volume stored in Swift.
Swift
The object storage service in OpenStack is called Swift. It provides a RESTful API to store and retrieve objects such as documents, pictures, audio and video files, etc. If you are familiar with the public cloud, think of it as a service similar to Amazon S3. Objects are stored in a hierarchy similar to S3, where an account has containers where objects are stored. Swift is scalable, highly available, and provides object redundancy.
Illustration of how data is stored in Swift (source)
Swift maintains data redundancy by storing multiple copies of an object across different locations, servers, and disks. It uses a concept called Rings to do this. The Ring uses zones, devices, partitions, and replicas to keep track of data locations. Each partition is replicated at least three times across the cluster, and the Ring knows where these copies are. It also decides which devices to use should a failure occur.
This provides high availability and data durability, which are necessary for object storage solutions. Swift creates new copies to maintain the desired number of replicas if a server or disk fails. Swift replication automatically handles hardware failures and ensures data consistency. It uses checksums to ensure the data integrity of all copies across the cluster and can detect and repair corrupted data automatically. Along with replication, Swift offers erasure coding as an alternative, reducing storage overhead while maintaining data durability.
Swift uses a distributed architecture, so you can scale the storage capacity in a cluster by adding more nodes, a process typically called horizontal scaling. You can also scale the storage nodes vertically if needed.
Swift is ideal for storing large amounts of unstructured data, backups, archival requirements, large datasets, VM images, etc. The RESTful API enables easy application integration and is compatible with Amazon S3 API for seamless interoperability.
Swift can use Keystone for authentication and authorization, providing role-based access control capabilities. It also supports multiple accounts and users, providing isolated storage environments for tenants.
Manila
Manila is the name of the service that provides shared file system capabilities in OpenStack. It also uses a distributed architecture. Manila can be extended with multiple backends to support a specific vendor or file system requirement.
Illustration of Manila architecture (source)
It is also a RESTful API service, and users can interact with it directly using OpenStack Horizon, CLI, or API. Additionally, developers can use the API to build apps that provide complex logic and automation.
Manila can also provide self-service share provisioning and access control of file shares to clients running in other private and public clouds or in bare metal environments. It is recommended that Keystone be used for authentication and authorization.
Modern applications require shared storage for various use cases, and services like Manila cater to these use cases. Here are some of these use cases:
- Collaboration applications require shared storage for sharing content across multiple users or applications for collaboration.
- Project management applications also require shared storage for documents and artifacts related to projects.
- Content management systems also use shared storage for media such as images, videos, etc. WordPress is an excellent example of this.
- VDI or Virtual Desktop Infrastructure solutions also use shared file systems to store users’ home directories, application data, user profiles, etc.
- HA clusters of Windows and Linux also need shared storage to set up clustering.
Ceph
Ceph is an open-source solution providing object, block, and file storage capabilities. It is a software-defined solution popular with cloud platforms such as OpenStack and cloud-native platforms like Kubernetes. Ceph nodes sit on top of commodity hardware and use intelligent daemons to orchestrate large numbers of nodes, which communicate with each other to replicate and distribute data dynamically.
Illustration of the Ceph stack and the technologies it supports (Source)
Ceph is based on the Reliable Autonomic Distributed Object Store (RADOS), which provides features such as intelligent data distribution, self-healing, high availability with no single point of failure, and self-managing capabilities. Data stored is replicated for reliability and resiliency, and the cluster can be scaled to store petabytes of data with thousands of clients reading and writing data simultaneously.
A typical Ceph cluster has a number of Ceph monitors, managers, and object storage daemons (OSDs) equal to the number of replicas of the object being stored. OSDs are responsible for replicating data across nodes. Ceph uses the Controlled Replication Under Scalable Hashing (CRUSH) algorithm for data safety and scaling. Ceph clients and OSDs use it to track where the object is stored in the cluster.
In the context of OpenStack, Ceph is used as backend storage for services such as Swift, Cinder, Manila, etc. It provides two interfaces that various OpenStack services can use for storage needs:
- Swift uses the RADOS REST Gateway to store objects.
- Services such as Glance, Cinder, and Nova use the RADOS block device interface for block storage requirements.
Learn about the features that power Trilio’s intelligent backup and restore
Integrating storage with other OpenStack services
Based on their use cases, OpenStack storage services integrate with other OpenStack services to provide complete solutions. For example, OpenStack Cinder is used with OpenStack Nova for persistent block storage. Let’s quickly examine how various storage services integrate in the areas of compute, networking, and identity.Compute
Cinder volumes can be attached to instances using the CLI or the horizon interface. To attach a volume using the CLI, use the OpenStack server add volume command. Here is an example:openstack server add volume <server_id> <volume_id> --device <drive>Note that “server” means “instance” here.
Networking
The Swift and Glance services rely on networking to access and store objects. Objects in Swift are accessed using HTTPS through a URL. The public URL of a Swift container or object uses the following format:https://<swift-host>/v1/<account>/<container>/<object>For example, the URL to access an object called swift.png would look like this:
https://swift.oscloud.net/v1/1234567890/architectures/swift.pngSimilarly, for a public Glance image, the URL would be:
http://glance.oscloud.net/v1/images/71c675ab-d94f-49cd-a114-e12490b328d9
Identity
Keystone, OpenStack’s identity service, is the recommended authentication provider for storage services, especially Swift, the object storage service. To use Keystone for authentication in Swift, you must install the KeystoneMiddleware package. It includes the auth_token middleware for authentication token validation and the keystoneauth middleware for authorization and mapping of keystone roles to ACLs of Swift. Keystone supports per-container ACL.Performance recommendations
Storage can quickly become a performance bottleneck if not optimized and managed well. It can also lead to an increase in storage costs. The following are some steps you can take to manage OpenStack storage performance.Storage tiering
Storage tiering helps in optimizing storage capacity, cost, and performance. An example is providing an SSD-drive-based tier for frequently accessed data and a magnetic-drive-based tier for less frequently accessed data, such as backups or archival data. Swift provides storage policies that you can use to create tiering. You do this by creating multiple object rings. Each ring has a policy defining durability, performance, storage backend, etc. You must apply storage policies at the container level. Cinder provides volume types that you can use to create a tiered storage offering. A volume type is a collection of properties or capabilities representing a particular tier. For example, a premium volume type uses a backend with NVMe storage for higher performance, while a standard volume type uses a backend with SSD storage. You must specify the properties or capabilities as key/value pairs. You must also have a suitable backend to support them. Similarly, Manila provides share types, which you can create to characterize a share with specific capabilities. For example, you could have a premium share type and a standard share type.Cache optimization
Follow these recommendations for cache optimization in OpenStack storage:- Use a caching solution like memcache to improve the performance of Swift proxy servers.
- Enable write caching for Cinder volumes, which can make a substantial difference for write-intensive scenarios.
Securing OpenStack storage services
Since OpenStack is multi-tenant, multiple projects and clients can use a cluster, requiring you to address regulatory and data privacy concerns. Here are some ways to ensure data privacy.Encryption
Encryption should be enabled on projects with sensitive data or when required for compliance. While OpenStack services support encryption at rest, you must also take measures to encrypt data in transit and in memory as well as the application code, if applicable.- Swift supports data encryption at rest using the keymaster and encryption middleware filters in the proxy server pipeline. This capability is optional and not enabled by default. However, if you need encryption for data in transit, you should encrypt it on the client side before sending it to Swift. Encrypting before sending it to Swift ensures that data is encrypted both in transit and at rest.
- Similarly, Cinder supports the creation of encrypted volumes, including the option to select the encryption algorithm and key size. If a volume is encrypted, the backups are also encrypted.
- Manila supports creating a security service to control access to shares. A share can be associated with one of the following security service types: LDAP, Kerberos, or Microsoft Active Directory. You should do this on a per-project basis, separate from the Keystone authentication used for role-based access control to the Manila API.
RBAC
You should use OpenStack Keystone to provide all storage services with role-based access control (RBAC) capabilities. Additionally, Keystone can authenticate users to gain access to Swift objects.Monitoring OpenStack storage services
Finally, you need to know what happens with the various storage services available in OpenStack. You can do this by monitoring metrics such as disk usage, object size, utilization, etc. OpenStack has a native monitoring-as-a-service project called Monasca. It is an agent-based monitoring tool that you can also extend using plugins. It can show metrics like disk usage, CPU usage, network errors, etc. Ceilometer is another OpenStack project that can help with monitoring. While it is not a full-fledged monitoring tool, it allows you to measure and collect telemetry information that you can consume through APIs or third-party tools. Due to the lack of a mature unified monitoring tool or solution within OpenStack, it is common for organizations to use third-party monitoring tools that provide OpenStack-specific solutions. Remember to do your due diligence on what storage services are available and what level of metrics they can monitor before purchasing. Here are some of the popular third-party monitoring tools:- ManageEngine Applications Manager
- Dynatrace
- Opsview Cloud or OpsView Monitor
- The Elasticsearch, Logstash, and Kibana (ELK) stack
Swift monitoring using native middleware
The Swift object storage service has middleware and other tools for monitoring Swift-specific parameters. The approach is not very user-friendly and requires some integration with a monitoring system.| Middleware | Swift-specific parameters |
| Swift Recon |
|
| Swift-Informant |
|
| Statsdlog |
|
OpenStack storage best practices
Finally, here’s a rundown of some of the best practices for these OpenStack storage services around planning, performance, and redundancy.Planning
For efficient capacity planning and to get the best return on investment or ROI from your OpenStack storage solutions, consider these practices:- Use monitoring tools such as Prometheus to monitor metrics and gain insight into the scalability of your OpenStack storage. This comes in handy for future capacity planning and scaling.
- Use horizontal scaling methods to minimize the stress for accurate planning.
- Plan for storage segregation based on region and availability zones before deployment.
Performance
Consider the following recommendations for Swift:- Choosing the right storage backend significantly impacts performance. For example, Ceph or a specialized storage backend such as NetApp can perform better than NFS.
- Use SSDs to achieve faster response times for frequently accessed data.
- For a large number of connections, use multiple proxy servers.
- Swift supports storage policies, which can control parameters such as the number of replicas, disk types, such as SSD or magnetic, data placement, etc. You can use storage policies to balance performance and cost and offer SLAs.
- When using Ceph as the backend for Glance, use RAW images; this saves time and greatly improves the instance boot time.
Redundancy
Redundancy is a function of the redundancy of the storage services and the data itself. Deploy the APIs of the respective storage services in a highly available configuration for service redundancy. The general requirement is to have a VIP or virtual IP address, add the API resource to Pacemaker, configure the service API to listen or bind to the VIP, and then configure other OpenStack services to use the HA API. Here are some additional redundancy recommendations:- Use multiple proxy servers in Swift for redundancy.
- Use volume replication for Cinder. For example, you can create a primary and secondary model based on Ceph RBD mirroring.
- Use RAID on nodes for protection against failures (when not using a storage backend with HA capabilities).
- Use storage solutions like Ceph that are natively highly available.
Automated Kubernetes Data Protection & Intelligent Recovery
Enjoy native OpenStack integration, documented RESTful API, and native OpenStack CLI
Restore Virtual Machines with Trilio’s One-Click Restore
Select components of your build to recover via the OpenStack dashboard or CLI
Conclusion
OpenStack provides a comprehensive suite of storage services that meet many different use cases of modern applications. Most cloud-native applications need object storage for storing unstructured data like backups, archives, and media files, which can be provided by Swift, the object storage service of OpenStack. Workloads like databases need high-performance persistent storage for storing data and other related files that the block storage service Cinder can provide. It can meet any use case that requires block storage directly attached to a VM. Sometimes, data must be shared between multiple VMs for collaboration or applications relying on traditional file system protocols, which can be done by the shared file system service Manila. Finally, you can use Glance to store VM images.
Together, these services offer scalable, highly available, robust, and flexible storage solutions tailored to meet the specific requirements of diverse use cases in modern applications and infrastructure.
Trilio provides a comprehensive backup and recovery solution for backing up and recovering Cinder, Ceph, and Nova ephemeral volumes. Additionally, Ceph-based Cinder volume backups use less storage and consume less network bandwidth to move the backups, providing optimized use of compute resources.
- Summary of key OpenStack storage services
- A brief history of OpenStack
- OpenStack storage concepts
- OpenStack storage services
- Integrating storage with other OpenStack services
- Performance recommendations
- Securing OpenStack storage services
- Monitoring OpenStack storage services
- OpenStack storage best practices
- Conclusion
Like This Article?
Subscribe to our LinkedIn Newsletter to receive more educational content
