Mastering OpenStack with Kolla Ansible: An Automation and High-Availability Guide

Service orchestration

For deployment and upgrades, Kolla uses the Ansible automation engine. Ansible offers the benefit of agentless deployment, using only SSH and Python on the deployment hosts. It also simplifies the deployment by utilizing standardized human-readable YAML-based configuration files. 

Inventory

Kolla utilizes a standard Ansible inventory file to deploy OpenStack services. The inventory file specifies which service (control, compute, or storage) should be deployed on which host. Kolla provides a sample inventory file for performing a quick deployment on a single host for evaluation. For production and high availability, you can use the multinode deployment and configure which services get deployed on which hosts.

Configuration files

Kolla has a very simplified configuration structure based on one inventory file, as explained above. There is also the /etc/kolla/globals.yaml file, which is used to tweak the OpenStack deployment based on your requirements. You can configure options like which base distro to use, which internal and external virtual IPs to use for APIs, and what the network interfaces are on the hosts.

Comparison of Kolla Ansible and OpenStack-Ansible

The OpenStack cloud can be deployed via various methods, including manually via source code or using distribution packages. In addition to Kolla Ansible, various automated deployment tools exist. Here, we will compare Kolla with a related tool, OpenStack-Ansible, which also utilizes Ansible for service orchestration.  

Kolla Ansible is a typical choice for those who look for a fast, lightweight deployment, have prior experience with containerized applications, and want to take advantage of rolling upgrades. Alternatively, if you want full control of the deployment with a high level of customization, are ready to go through a steep learning curve, and would like to perform bare-metal deployment or utilize system containers for isolation, you can choose OpenStack-Ansible.

More details on OpenStack-Ansible configuration and recommendations can be found here. 

Multinode deployment via Kolla Ansible

For production deployments, the different services of OpenStack are configured in high availability and spread across different nodes. There are specific services, like the MariaDB Galera cluster, that require a minimum of three nodes to maintain a quorum and avoid split-brain scenarios. The control plane services like RabbitMQ, Keystone, Nova, Neutron, and Glance APIs are deployed on multiple nodes. The load balancing and failover are typically managed via a combination of HAProxy and Keepalived.

In our current article, we will deploy three OpenStack control nodes and two compute + storage nodes. The deployment will be managed via a dedicated Kolla Ansible node, which can be used for the complete cloud lifecycle, from first deployment to OpenStack upgrades. It can also clean up the deployment if you would like to start over. The beauty of Kolla is that the deployment can be scaled up effortlessly by adding additional compute and storage nodes to the inventory file and running the deployment again. 

Kolla Ansible OpenStack multinode deployment architecture

Deployment prerequisites

We will use Ubuntu 24.04 host machines for the OpenStack deployment. Some steps need to be taken before starting the Kolla Ansible deployment:

• All the nodes must be synced with an NTP server, and the timezone should be set to the same as per your zone.
• The public SSH key of the deployment host root user must be copied into the /root/.ssh/authorized_keys file of all nodes.
• The Docker engine must be installed on all nodes.
• The nodes must have proper names configured based on their roles.
• All the nodes, including the deployment machine, should have the node names and IPs in the /etc/hosts file as follows:

172.16.16.11 control01
172.16.16.12 control02
172.16.16.13 control03
172.16.16.21 compute01
172.16.16.22 compute02

Prepare the deployment host

All the deployments and upgrades will be done from the Kolla deployment host. It is recommended that Kolla Ansible be installed in a Python virtual environment so that the required dependencies can be met and they do not conflict with the system packages. All the installations will be done via the root user to avoid any permissions-related issues.

Prerequisite packages: Install the prerequisite packages of Kolla Ansible on the deployment host.

# apt install git libffi-dev gcc libssl-dev \
libdbus-glib-1-dev python3-dev python3-venv

Python virtual environment: Create and enable the Python virtual environment. Update pip to the latest version.

# python3 -m venv ~/kolla-venv
# source ~/kolla-venv/bin/activate
(kolla-venv) :~# pip install -U pip

Install Kolla Ansible: Get the Kolla Ansible repository and install Kolla using pip.

(kolla-venv) :~# pip install git+https://opendev.org/openstack/kolla-ansible@master

Install the Kolla dependencies: Install dependencies using this command.

(kolla-venv) :~# kolla-ansible install-deps

Kolla configurations: Copy the Kolla template configuration globals.yml and passwords.yml into the /etc/kolla directory.

(kolla-venv) :~# cp -r kolla-venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla

Kolla inventory: Kolla provides sample inventory files, all in one, and multinode. Copy the multinode inventory file into your home directory.

(kolla-venv) :~# cp kolla-venv/share/kolla-ansible/ansible/inventory/multinode .

The inventory file is in the format of Ansible ini inventory. It contains the nodes’ group roles (control, compute, network, storage). We need to update the inventory per our deployment, specifying the node names under the group/role. The hostnames must be resolvable from the deployment host. The following is the initial section of the multinode inventory file with nodes defined under each role.

[control]
control01
control02
control03

[network]
control01
control02

[compute]
compute01
compute02

[monitoring]
control01

[storage]
compute01
compute02

[deployment]
localhost       ansible_connection=local
...
...

Service passwords: The template passwords.yml file has blank passwords. We need to generate passwords to be used during the deployment as follows.

(kolla-venv) :~# kolla-genpwd

Kolla configuration: We next update the Kolla /etc/kolla/globals.yml configuration as follows. We specify the base distro, the virtual IP addresses used by HAProxy for the internal and external API endpoints, the network interface for API services, and the external network interface for Neutron.

......
kolla_base_distro: "ubuntu"
......
kolla_internal_vip_address: "172.16.16.8"
......
kolla_external_vip_address: "172.16.16.9"
......
network_interface: "eth0"
......
neutron_external_interface: "eth1"
......

Prechecks: Before starting the OpenStack deployment, we need to verify that all requirements are met on the nodes by running prechecks. They verify factors like Docker environment, software versions, network configurations, time synchronization, etc. We need to address any reported issues before moving further by running the kolla prechecks command.

(kolla-venv) :~# kolla-ansible prechecks -i multinode

OpenStack deployment: We are now ready for the OpenStack deployment. The following deploy command will download the required containers from the container registry and install and configure all OpenStack services.

(kolla-venv) :~# kolla-ansible deploy -i multinode

Validate configuration: After deployment, we can check that all service configurations are as expected using the validate-config command.

(kolla-venv) :~# kolla-ansible validate-config -i multinode

Verify the OpenStack deployment

We need to install the Python OpenStack client to interact with the deployment via CLI using pip in the virtual environment.

(kolla-venv) :~# pip install python-openstackclient -c https://releases.openstack.org/constraints/upper/master

The OpenStack CLI requires cloud endpoints and credentials to connect to the OpenStack cloud. The following command will generate the OpenStack RC files in /etc/Kolla.

(kolla-venv) :~# kolla-ansible post-deploy

We can now authenticate and interact with the cloud. The following are examples of how to get the services endpoint list, list of users, and list of projects created in the cloud.

(kolla-venv) :~# source /etc/kolla/admin-openrc.sh
(kolla-venv) :~# openstack endpoint list -c "Service Name" -c URL
+--------------+-------------------------------------------+
| Service Name | URL                                       |
+--------------+-------------------------------------------+
| keystone     | http://172.16.176.8:5000                  |
| heat-cfn     | http://172.16.176.8:8000/v1               |
| glance       | http://172.16.176.9:9292                  |
| heat-cfn     | http://172.16.176.9:8000/v1               |
| heat         | http://172.16.176.9:8004/v1/%(tenant_id)s |
| keystone     | http://172.16.176.9:5000                  |
| glance       | http://172.16.176.8:9292                  |
| nova         | http://172.16.176.9:8774/v2.1             |
| heat         | http://172.16.176.8:8004/v1/%(tenant_id)s |
| neutron      | http://172.16.176.8:9696                  |
| placement    | http://172.16.176.9:8780                  |
| nova         | http://172.16.176.8:8774/v2.1             |
| neutron      | http://172.16.176.9:9696                  |
| placement    | http://172.16.176.8:8780                  |
+--------------+-------------------------------------------+

(kolla-venv) root@kolla-jump:~# openstack user list
+----------------------------------+-------------------+
| ID                               | Name              |
+----------------------------------+-------------------+
| 0e145dfe43c4445683c67ea7950f948f | placement         |
| 0e3589e5900b4bca99ecd5651e3bd0e3 | heat              |
| 40748512c7d94e0c8b648028f3e3688d | nova              |
| 81b57a0b74f4457281a0f14499e8e484 | glance            |
| b5c7d93f1aad4a2aa89a58de17d48c27 | admin             |
| c79d966e2c91490ab47215c5977edce6 | heat_domain_admin |
| f2fdf9d45ae146e587e0ae14f0cc98f3 | neutron           |
+----------------------------------+-------------------+

(kolla-venv) root@kolla-jump:~# openstack project list
+----------------------------------+---------+
| ID                               | Name    |
+----------------------------------+---------+
| 54da4ec1cff54403a45494cfad131584 | service |
| a7d85f6d137c42568c891e6b47ef1181 | admin   |
+----------------------------------+---------+

Enhancing Kolla Ansible with Trilio

In a production environment, it is recommended to have a disaster recovery strategy. OpenStack provides some native capability to make backups and snapshots, but these are not context-aware and lack automatic off-site backup and recovery capability. The data protection of OpenStack can be enhanced via Trilio, which seamlessly integrates into OpenStack and provides agentless backups and complete disaster recovery.  Trilio supports various OpenStack deployments, including Kolla Ansible. 

You can follow the steps from the Trilio deployment manual for Kolla Ansible. Clone the Trilio repository and append the deployment scripts, inventory, and passwords to the Kolla Ansible deployment. Next, configure the required Trilio parameters and perform the deployment using the kolla-ansible deploy command.