We’re getting quite a few questions about how to set the admin password in OpenStack Windows instances, so let’s clarify the available options.
nova get-password
The secure and proper way to set passwords in OpenStack Windows instances is by letting Cloudbase-Init generate a random password and post it encrypted on the Nova metadata service. The password can then be retrieved with:
nova get-password <instance> [<ssh_private_key_path>]
You need to boot your instance with a SSH keypair (exactly like you would do on Linux for SSH public key authentication). In this case the public key is used to encrypt the password before posting it to the Nova HTTP metadata service. This way nobody will be able to decrypt it without having the keypair’s private key.
This option is also well supported in Horizon, but not enabled by default. To enable it, just edit openstack_dashboard/local/local_settings.py and add:
OPENSTACK_ENABLE_PASSWORD_RETRIEVE = True
To retrieve the password in Horizon, select “RETRIEVE PASSWORD” from the instance dropdown menu:
Browse for your private key:
Click “DECRYPT PASSWORD” (de decryption will occur in the browser, no data will be sent to the server) and retrieve your password:
nova boot –meta admin_pass
In case a password automatically generated is not suitable, there’s an option to provide a password via command line. This is NOT RECOMMENDED due to the security implications of sharing clear text passwords in the metadata content.
In this case the password is provided to the Nova instance via metadata service and assigned by Cloudbase-Init to the admin user:
nova boot --meta admin_pass="<password>" ...
Given the previously mentioned security concerns this feature is disabled by default in Cloudbase-Init. In order to enable it inject_user_password must be set to true in the cloudbase-init.conf and cloudbase-init-unattend.conf config files:
inject_user_password = true
Password change in userdata script
The userdata can contain any PowerShell content (note the starting #ps1 line to identify it as such), including commands for creating users or setting passwords, providing a much higher degree of flexibility. The same security concerns for clear text content apply as above.
The main limitation is that it does not work with Heat or other solutions that already employ the userdata content for other means.
Passwordless authentication
Nova allows X509 keypairs to support passwordless authentication for Windows. This is highly recommended as it does not require any password, similarly to SSH public key authentication on Linux. The limitations of this option is that it works only for remote PowerShell and WinRM and not for RDP.
OpenStack is a great technology, but it can be a bit cumbersome to deploy and manage without the proper tools. One easy solution to address this issue is to deploy OpenStack services using pre-built Docker containers.
Kolla is a set of deployment tools for OpenStack, consisting in the Kolla project itself, for generating OpenStack Docker images, and “deliverables” projects, to deploy the Docker containers and thus OpenStack. The most mature deliverable is kolla-ansible, which, as the name implies, uses Ansible playbooks to automate the deployment. The project documentation can be found here.
Hyper-V setup
On the Windows host, we need a VM to host the Linux OpenStack controller. For this purpose I created an Ubuntu 16.04 VM with 8GB of RAM, 4 virtual cores and 20GB of disk. All the controller services run here and are deployed with Kolla in Docker containers. Last but not least, the same Hyper-V also serves as a compute host for the OpenStack deployment. This is achieved by installing our Cloudbase OpenStack components. Additional Hyper-V compute nodes can be added later as needed.
Networking setup
On the Hyper-V host, I am going to need 2 virtual switches that are going to be connected to the OpenStack controller VM. ext-net is the external network, it is bridged to the Windows physical external interface. I will use this network also for the management of the VM. data-net is the data network, which can be a simple private virtual switch for now (an external one is needed only when adding more compute nodes).
On the OpenStack Controller VM there are 3 interfaces. The first two, eth0 and eth1 are connected to the external network. The former is used for management (SSH, etc) and the latter is used by OpenStack for external traffic, managed by Open vSwitch. Finally, eth2 is the data/overlay network. It is used for tenant traffic between the instances and the Neutron components in the controller.
eth1 and eth2 do not have an IP and are set as “manual” in /etc/network/interfaces. The reason for this is that they are managed by OpenvSwitch. Also on these interfaces I need to enable MAC address spoofing (“Advanced Features” tab on the adapter).
The scripts that I will be using configures the Linux network interfaces automatically so I don’t need to bother with that now. The only interface I have already configured is eth0 so I can SSH into the machine.
OpenStack controller deployment
I am going to clone a repository that contains the scripts for the Kolla Openstack deployment, which can be found here. At the end of the deployment it will also create some common flavors, a Cirros VHDX Cinder image, a Neutron virtual router and 2 networks, one external (flat) and one private for tenants (VLAN based).
git clone https://github.com/cloudbase/kolla-resources.git
cd kolla-resources
To begin with, we are going to configure the management and external network details by setting some variables in deploy_openstack.sh:
vim deploy_openstack.sh
# deploy_openstack.sh
MGMT_IP=192.168.0.60
MGMT_NETMASK=255.255.255.0
MGMT_GATEWAY=192.168.0.1
MGMT_DNS="8.8.8.8"
# neutron external network information
FIP_START=192.168.0.80
FIP_END=192.168.0.90
FIP_GATEWAY=192.168.0.1
FIP_CIDR=192.168.0.0/24
TENANT_NET_DNS="8.8.8.8"
# used for HAProxy
KOLLA_INTERNAL_VIP_ADDRESS=192.168.0.91
As you can see, I am using the same subnet for management and external floating IPs.
Now I can run the deployment script. I am using the Linux “time” command to see how long the deployment will take:
time sudo ./deploy_openstack.sh
The first thing this script will do is to pull the Docker images for each OpenStack service. The great thing with Kolla is that you just need to create the images once, sparing significant time during deployment. This reduces significantly potential errors due to updated dependencies as the container images already contain all the required components. The images that I am going to use during the deployment are available here. Feel free to create your own, just follow the documentation.
After the deployment is finished, I have a fully functional OpenStack controller. It took around 13 minutes to deploy, that’s quite fast if you ask me.
real 12m28.716s
user 3m7.296s
sys 1m4.428s
By running “sudo docker ps” I can see all the containers running.
Admin credentials can be sourced now:
source /etc/kolla/admin-openrc.sh
The only thing left to do is to deploy the OpenStack Hyper-V components.
Nova Hyper-V compute node deployment
First, I’m going to edit the Ansible inventory to add my Hyper-V host (simply named “hyperv-host” in this post) as well as the credentials needed to access it:
An HTTPS WinRM listener needs to be configured on the Hyper-V host, which can be easily created with this PowerShell script.
Now, I’m going to run the scripts that will fully deploy and configure Nova compute on Hyper-V. The first parameter is the data bridge that I configured earlier, data-net. The third and fourth parameters are are the Hyper-V credentials that FreeRDP will need to use in order to access the Hyper-V host when connecting to a Nova instance console.
Next, I need to set trunk mode for my OpenStack controller. There are two reasons for this: first, I have a tenant network with type VLAN, and second, the controller is a VM in Hyper-V, so the hypervisor needs to allow VLAN tagged packets on the controller VM data interface. Start an elevated PowerShell and run:
The FreeRDP console access from Horizon works as well. I can also access the VM directly from Hyper-v if needed.
Useful information
What if you need to modify the configuration of an OpenStack service running in a container? For example, lets say you want to enable another ML2 type driver. It’s quite easy actually.
In this case i need to edit the ml2_conf.ini file:
sudo vim /etc/kolla/neutron-server/ml2_conf.ini
After I am done with the editing, the only thing left to do is to restart the Neutron server container:
sudo docker restart neutron_server
Done. As you can see, Kolla keeps all the OpenStack configuration files in /etc/kolla.
Conclusions
Containers can help a lot in an OpenStack deployment. Footprint is small, dependency induced regressions are limited and with the aid of a tool like Ansible, automation can be managed very easily.
What’s next? As I mentioned at the beginning, kolla-ansible is just one of the “deliverables”. kolla-kubernetes is also currently being developed and we can already see the benefits that the Kubernetes container orchestration can bring to OpenStack deployments, so looking forward for kolla-kubernetes to reach a stable status as well!
Windows Server 2016 is gaining significant traction in OpenStack and other clouds, thanks to the support for Windows Docker containers and lots of other new features.
While there’s no OpenStack Windows Server 2016 image directly available for download, the good news is that our automated build scripts will do all the work for you. All you need is a Windows Server 2016 ISO.
The automated build tools are publicly available on GitHub, allowing the generation of virtual (Hyper-V, KVM, VMware ESXi) or bare metal (MAAS, Ironic) images, including Cloudbase-Init, VirtIO drivers (KVM), latest Windows updates, etc.
You can kickstart the image generation on any Windows host (e.g. Windows 10, Windows Server 2016, etc) with Hyper-V enabled and the Windows ADK installed.
git clone https://github.com/cloudbase/windows-openstack-imaging-tools
cd windows-openstack-imaging-tools
Edit the create-windows-online-cloud-image.ps1 in the Examples directory to match your environment and requirements.
If you need to make any changes to the image generation (e.g. adding storage or networking drivers for bare metal servers), you have an extensive Readme which will guide through the entire process.
# This needs an elevated PowerShell prompt
./create-windows-online-cloud-image.ps1
For KVM, a frequent use case, the tool supports the latest Fedora VirtIO drivers, improving considerably the stability and performance of the OS.
We’ve added support for RemoteFX for Windows / Hyper-V Server 2012 R2 back in Kilo, but the highly anticipated Windows / Hyper-V Server 2016 comes with some new nifty features for which we’re excited about!
In case you are not familiar with this feature, it allows you to virtualize your GPUs and share them across virtual machine instances by adding virtual graphics devices. This leads to a richer RDP experience especially for VDI on OpenStack, as well as the benefit of having a GPU on your instances, enhancing GPU-intensive applications (CUDA, OpenCL, etc).
If you are curious, you can take a look at one of our little experiments. We’ve run a few GPU-intensive demos on identical guests with and without RemoteFX. The difference was very obvious between the two. You can see the recording here.
One of the most interesting feature RemoteFX brings in terms of improving the user experience, is device redirection. This allows you to connect your local devices (USBs, smart cards, VoIP devices, webcams, etc.) to RemoteFX enabled VMs through your RDP client. For a detailed list of devices you can redirect through your RDP session can be found here.
Some of the new features for RemoteFX in Windows / Hyper-V Server 2016 are:
4K resolution option
1GB dedicated VRAM (availble choices: 64MB, 128MB, 256MB, 512MB, 1GB) and up to another 1GB shared VRAM
Support for Generation 2 VMs
OpenGL and OpenCL API support
H.264/AVC codec investment
Improved performance
One important thing worth mentioning is the fact that RemoteFX allows you to overcommit your GPUs, the same way you can overcommit disk, memory, or vCPUs!
All of this sounds good, but how can you know if you can enable RemoteFX? All you need for this is a compatible GPU that passes the minimum requirements:
it must support DirectX 11.0 or newer
it must support WDDM 1.2 or newer,
Hyper-V feature must installed.
If you pass these simple requirements, all you have to do to enable the feature is to run this PowerShell command:
Install-WindowsFeature RDS-Virtualization
Hyper-V has to be configured to use RemoteFX. This can be done by opening the Hyper-V Manager, opening up Hyper-V Settings, and under Physical GPUs, check the Use this GPU with RemoteFX checkbox.
For more information about RemoteFX requirements and recommended RemoteFX-compatible GPUs, read this blog post.
In order to take advantage of all these features, the RDP client must be RemoteFX-enabled (Remote Desktop Connection 7.1 or newer).
Please do note that the instance’s guest OS must support RemoteFX as well. Incompatible guests will not be able to fully benefit from this feature. For example, Windows 10 Home guests are not compatible with RemoteFX, while Windows 10 Enterprise and Pro guests are. This fact can easily be checked by looking up the Video Graphics Adapter in the guest’s Device Manager.
After the RDS-Virtualization feature has been enabled, the nova-compute service running on the Hyper-V compute node will have to be configured as well. The following config option must be set to True in nova-compute‘s nova.conf file:
[hyperv]
enable_remotefx = True
In order to spawn an instance with RemoteFX enabled via OpenStack, all you have to do is provide the instance with a few flavor extra_specs:
os_resolution: guest VM screen resolution size.
os_monitors: guest VM number of monitors.
os_vram: guest VM VRAM amount. Only available on Windows / Hyper-V Server 2016.
There are a few things to take into account:
Only a subset of resolution sizes are available for RemoteFX. Any other given resolution size will be met with an error.
The maximum number of monitors allowed is dependent on the requested resolution. Requesting a larger number of monitors than the maximum allowed per requested resolution size will be met with an error.
Only the following VRAM amounts can be requested: 64, 128, 256, 512, 1024.
On Windows / Hyper-V Server 2012 R2, RemoteFX can only be enabled on Generation 1 VMs.
The available resolution sizes and maximum number of monitors are:
For Windows / Hyper-V Server 2012 R2:
Here is an example of a valid flavor for RemoteFX:
# nova flavor-create <name> <id> <ram> <disk> <vcpus>
nova flavor-create m1.remotefx 999 4096 40 2
nova flavor-key m1.remotefx set os:resolution=1920x1200
nova flavor-key m1.remotefx set os:monitors=1
nova flavor-key m1.remotefx set os:vram=1024
We hope you enjoy this feature as much as we do! What would you use RemoteFX for?
Platform and cloud interoperability has come a long way. IaaS and unstructured PaaS options such as OpenStack and Kubernetes can be combined to create cloud-native applications. In this port we’re going to show how Kubernetes can de deployed on an OpenStack cloud infrastructure.
Setup
My setup is quite simple, an Ocata all-in-one deployment with compute KVM. The OpenStack infrastructure was deployed with Kolla. The deployment method is not important here, but Magnum and Heat need to be deployed alongside other OpenStack services such as Nova or Neutron. To do this, enable those two services form /etc/kolla/global.yml file. If you are using Devstack, here is a local.conf that is deploying Heat and Magnum.
Kubernetes deployment
The Kubernetes cluster will consist of 1 master node and 2 minion nodes. I’m going to use Fedora atomic images for VMs. One useful info is that I used a 1 CPU, 2GB of RAM and 7GB disk flavor for the VMs. Below are the commands used to create the necessary environment setup. Please make sure to change IPs and different configurations to suit your environment.
# Download the cloud image
wget https://ftp-stud.hs-esslingen.de/pub/Mirrors/alt.fedoraproject.org/atomic/stable/Fedora-Atomic-25-20170512.2/CloudImages/x86_64/images/Fedora-Atomic-25-20170512.2.x86_64.qcow2
# If using HyperV, convert it to VHD format
qemu-img convert -f qcow2 -O vhdx Fedora-Atomic-25-20170512.2.x86_64.qcow2 fedora-atomic.vhdx
# Provision the cloud image, I'm using KVM so using the qcow2 image
openstack image create --public --property os_distro='fedora-atomic' --disk-format qcow2 \
--container-format bare --file /root/Fedora-Atomic-25-20170512.2.x86_64.qcow2 \
fedora-atomic.qcow2
# Create a flavor
nova flavor-create cloud.flavor auto 2048 7 1 --is-public True
# Create a key pair
openstack keypair create --public-key ~/.ssh/id_rsa.pub kolla-ubuntu
# Create Neutron networks
# Public network
neutron net-create public_net --shared --router:external --provider:physical_network \
physnet2 --provider:network_type flat
neutron subnet-create public_net 10.7.15.0/24 --name public_subnet \
--allocation-pool start=10.7.15.150,end=10.7.15.180 --disable-dhcp --gateway 10.7.15.1
# Private network
neutron net-create private_net_vlan --provider:segmentation_id 500 \
--provider:physical_network physnet1 --provider:network_type vlan
neutron subnet-create private_net_vlan 10.10.20.0/24 --name private_subnet \
--allocation-pool start=10.10.20.50,end=10.10.20.100 \
--dns-nameserver 8.8.8.8 --gateway 10.10.20.1
# Create a router
neutron router-create router1
neutron router-interface-add router1 private_subnet
neutron router-gateway-set router1 public_net
Before the Kubernetes cluster is deployed, a cluster template must be created. The nice thing about this process is that Magnum does not require long config files or definitions for this. A simple cluster template creation can look like this:
The deployment status can be checked and viewed from Horizon. There are two places where this can be done, first one in Container Infra -> Clusters tab and second in Orchestration -> Staks tab. This is because Magnum relies on Heat templates to deploy the user defined resources. I find the the Stacks option better because it allows the user to see all the resources and events involved in the process. If something goes wrong, the issue can easily be identified by a red mark.
In the end my cluster should look something like this:
SSH into the master node to check the cluster status
[root@kubemaster ~]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080
KubeUI is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-ui
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
So there it is, a fully functioning Kubernetes cluster with 1 master and 2 minion nodes.
A word on networking
Kubernetes networking is not the easiest thing to explaing but I’ll do my best to do the essentials. After an app is deployed, the user will need to access it from outside the Kubernetes Cluster. This is done with Services. To achive this, on each minion node there is a kube-proxy service running that will allow the Service to do its job. Now the service can work in multiple ways, some of them are via an VIP LoadBalancer IP provided by the cloud underneath K8S, or with port-forward on the minion node IP.
Deploy an app
Now that all is set up, an app can be deployed. I am going to install WordPress with Helm. Helm is the package manager for Kubernetes. It installs applications with charts, which are basically apps definitions written in yaml. Here are documentation on how to install Helm.
[root@kubemaster ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-release-mariadb-2689551905-56580 1/1 Running 0 10m
my-release-wordpress-3324251581-gzff5 1/1 Running 0 10m
There are multiple ways of accessing the contents of a pod. I am going to port-forward 8080 port from the master node to the 80 port of the pod.
Microsoft Azure Stack brings Azure on-premise and with that comes the opportunity to migrate your workloads from other on-prem cloud and virtualization technologies like VMware vSphere, System Center VMM, Hyper-V or OpenStack.
Coriolis provides an agentless migration and DRaaS solution to consolidate all your workloads on an Azure Stack on-premise cluster or on the Azure public cloud.
Here’s a demo showcasing how easy the whole migration workflow looks like:
Coriolis takes advantage of the latest features in Azure Stack, including Managed Disks, just announed and released this month.
You can find more information on Coriolis here and if you are attending Microsoft Ignite next week September 24th-28th in Orlando, come to visit us at booth #1045 for a live demo!
Unlock your DevOps potential in as dynamic a cloud computing start-up as it gets!
Curriculum
Our lab will consist entirely of hands-on classes where the trainee gets the chance to directly interact with the technologies that power the cloud. We will focus on walking you through the core concepts, tools and platforms which any aspiring DevOps must be aware of.
System Administration – Linux and Windows
Scripting (Bash / PowerShell)
Networking
Virtualization and Containerization
Python Development (Language, Platform and Practices)
Working with Git as a Version Control System
OpenStack (Architecture & Deployment)
Running containerized workloads with Docker and Kubernetes
Get engaged with top-notch technologies and develop your DevOps skills. By attending this lab you will:
Gain experience in multiple platforms and tools
Learn about the Open Source community
Attend highly practical working sessions
Kickstart your Cloud Engineer career
Meet your Mentors
Adrian Vladu
Senior Cloud Engineer
Claudiu Belu
Senior Cloud Engineer
Dan Ardelean
Cloud Engineer
Lucian Petruț
Senior Cloud Engineer
Nashwan Azhari
Cloud Engineer
Ready to join?
We invite you to participate in a fast-paced showdown of your technical skills in mini-CTFs designed to bring out the litter hacker in everyone. All the tasks can be easily run within your own environment with virtually zero setup so you can feel right at home as you hack away at them.
As soon as registrations for Liga AC Labs will be opened, we’ll add to our blog post the sing-up link so you can easily join our Cloud Engineer Bootcamp right away.
When the registration period ends, we’ll provide a complete guideline regarding our technical challenge. So stay tuned and check your inbox for updates.
The world is rapidly moving towards containarized deployments, serverless applications and away from traditional infrastructure based on virtual machines and bare metal servers. Mind you, virtual machines and bare metal servers are still a huge part of infrastructure, but they have become hidden away under layers upon layers of abstraction. Developers no longer need to worry that changing a version for one of the dependencies of their application, might break other applications on a running system. That is so 2015. These days we just create one container in which our application is the sole tenant. Problem solved.
Right, right…but you mentioned something about migrating VMs in the same sentence with Kubernetes?
Indeed I did.
While the world may be moving towards containarized deployments, there are still a large number of environments and legacy systems that are built directly on top of virtual machines. In cases like this you can choose to spend the time and money to re-platform your stack to function properly in the new paradigm, or you can go for something that will get you most of the way there.
We will be discussing about the latter. While long-term you will probably re-platform your stack, short term you will undoubtedly benefit from something that will de-duplicate your infrastructure footprint, while maintaining functionality. This is where projects like Kubevirt come into play. Kubevirt is a Kubernetes extension that adds virtual machine management capabilities to any standard cluster. It leverages libvirt to create KVM based virtual instances, using the same Kubernetes APIs you already leverage to create containers.
Yes, that’s right, you can now create virtual machines in Kubernetes. But why is this important? Or better yet, how is this useful?
To put it plainly, this means that you are now able to gradually start re-platforming your application stack from VM based deployments, to a fully containerized solution. You don’t have to make one giant push to re-platform everything, duplicating your infrastructure to ensure business continuity while you do it. There are no complicated networking configurations to enable connectivity between the tenant network of your VM based deployments and the pod network in your cluster.
You have one single platform that combines both containers and virtual machines. One platform that makes interoperability between containers and VMs virtually transparent: Kubernetes.
You might be thinking that while all this sounds great, you already have everything deployed on VMs. Redeploying everything on different virtual machines inside Kubernetes is a huge undertaking that will take a lot of time and resources. In some cases it might not even be feasible to do, given the nature of some legacy systems running on old machines with extended support after EoL (looking at you Microsoft).
The good news is that you don’t have to redeploy anything. You can simply move those workloads in their entirety.
Migrating to Kubevirt using Coriolis
It turns out, that Coriolis is a perfect fit for this scenario. With its plugable architecture, adding a Kubevirt import provider to Coriolis was simple and quick. Moreover, it allows Coriolis to migrate instances into your Kubevirt deployment from any of the existing public and private clouds we currently support. To name a few:
VMware
Azure
AWS
OCI-C (Oracle Cloud Infrastructure Classic)
OVM (Oracle VM server)
OpenStack
Hyper-V
You have an existing deployment of virtual machines inside a private or public cloud and you’d like to move it to your Kubevirt deployment? No problem! Coriolis can do that for you, in a fully automated way. Coriolis was designed with disaster recovery in mind. This means that you can sync your existing infrastructure from one of the supported clouds, to your Kubevirt deployments, and migrate whenever you are ready.
Coriolis strives to ensure that you never loose business continuity while your data is being synced from source to destination. Your running virtual machines will stay running while we copy everything over to their new home. This is done completely agentless. We will never require you to install anything inside your virtual machines.
So, want to test out a deployment before committing to actually shift traffic to the Kubernetes backed virtual machine deployment? That is perfectly fine. Coriolis enables you to do as many test deployments of your workload as you wish. The virtual machine data is already synced inside your Kubernetes cluster.
Deploy your migrated virtual machines inside Kubernetes, test them out, validate that everything is fine, tare them down, migrate them again. You decide when you want to shift traffic. You are in full control.
By now I hope I’ve made you curious enough to want to see a quick demo. Below you can find a video of an actual migration from VMware to Kubevirt. This demo is in real time. There are no speedups or edits. Enjoy!
ARM servers are more and more present in our day to day life, their usage varying from minimal IoT devices to huge computing clusters. So we decided to put the Windows support for ARM64 cloud images to the test, with two primary focuses:
Toolchain ecosystem – Building and running Cloudbase-Init on Windows ARM64
Virtualization – Running Windows and Linux virtual machines on Windows ARM64
Our friends from https://amperecomputing.com kindly provided the computing resources that we used to check the current state of Windows virtualization on ARM64.
The test lab consisted of 3 Ampere Computing EMAG servers (Lenovo HR330A – https://amperecomputing.com/emag), each with 32 ARM64 processors, 128 GB of RAM and 512 GB SSD.
Toolchain ecosystem on Windows ARM64: building and running Cloudbase-Init
Cloudbase-Init is a provisioning agent designed to initialize and configure guest operating systems on various platforms: OpenStack, Azure, Oracle Cloud, VMware, Kubernetes CAPI, OpenNebula, Equinix Metal (formerly: Packet), and many others.
Building and running Cloudbase-Init requires going through multiple layers of an OS ecosystem, as it needs a proper build environment, C compiler for Python and Python extensions, Win32 and WMI wrappers, a Windows service wrapper and an MSI installer.
This complexity made Cloudbase-Init the perfect candidate for checking the state of the toolchain ecosystem on Windows ARM64.
Install Windows 10 PRO ARM64 on the EMAG ARM servers
EMAG servers come with CentOS 7 preinstalled, so the first step was to have a Windows ARM64 OS installed on them.
Windows Server ARM64 images are unfortunately not publicly available, so the best option consists in using Windows Insider (https://insider.windows.com/), Windows 10 PRO ARM64 images available for download.
As there is no ISO available on the Windows Insiders website, we had to convert the VHDX to a RAW file using qemu-img.exe, boot a Linux Live ISO which had dd binary tool on it (Ubuntu is great for this) on the EMAG server and copy the RAW file content directly on the primary disk.
For the dd step, we needed a Windows machine where to download / convert the Windows 10 PRO ARM64 VHDX and two USB sticks. One USB stick for the Ubuntu Live ISO and one for the Windows 10 PRO ARM64 RAW file.
Rufus was used for creating the Ubuntu Live ISO USB and copying the RAW file to the other USB stick. Note that one USB stick must be at least 32 GB in size to cover for the ~25GB of the Windows RAW file.
After the dd process succeeds, a server reboot was required. The first boot took a while for the Windows device initialization followed by the usual “Out of the box experience”.
The following steps show how we built Cloudbase-Init for ARM64. As a side note, Windows 10 ARM64 has a builtin emulator for x86, but not for x64. Practically, we could run the x86 version of Cloudbase-Init on the system, but it would have run very slow and some features would have been limited by the emulation (starting native processes).
Gather information on the toolchain required to build Cloudbase-Init
The Cloudbase-Init ecosystem consists of these main building blocks:
Python for Windows ARM64
Python setuptools
Python pip
Python PyWin32
Cloudbase-Init
OpenStack Service Wrapper executable
Toolchain required:
Visual Studio with ARM64 support (2017 or 2019)
git
Python for Windows ARM64
Python 3.x for ARM64 can be built using Visual Studio 2017 or 2019. In our case, we used the freely available Visual Studio 2019 Community Edition, downloadable from https://visualstudio.microsoft.com/downloads/.
The required toolchain / components for Visual Studio can be installed using this vsconfig.txt. This way, we make sure that the build environment is 100% reproducible.
Python pip is required for easier management of Cloudbase-Init’s requirements installation and wheels building.
Python’s wheelpackage is required to build wheels. Wheels are the pre-built versions of Python packages. There is no need to have a compiler to install the package from source on the exact system version the wheel has been built for.
Python PyWin32 package is a wrapper for (almost) all Win32 APIs from Windows. It is a behemoth from the source code perspective, with Cloudbase-Init using a limited amount of Win32 APIs via PyWin32.
The build process takes quite a lot of time, at least half an hour, so we took a(nother) cup of coffee and enjoyed the extra time.
The patches hardcode some compiler quirks for Visual Studio 2019 and remove some unneeded extensions from the build. There is work in progress to prettify and upstream the changes.
Cloudbase-Init
Now, as all the previous steps have been completed, it is time to finally build Cloudbase-Init. Thank you for your patience.
After the installation steps were completed, the cloudbase-init.exe AMR64 executable wrapper will be available.
OpenStack Service Wrapper executable
Cloudbase-Init usually runs as a service at every boot. As cloudbase-init.exe is a normal executable, it needs a service wrapper for Windows. A service wrapper is a small program that implements the hooks for the Windows service actions, like start, stop and restart.
In this post we will focus on the state of virtualization for Windows Hyper-V on ARM64, checking if we can run Windows and Linux virtual machines.
This is the second part of the ‘State of Windows on ARM64’ series. Click here for the first part.
The test lab consists of 3 Ampere Computing EMAG servers (Lenovo HR330A – https://amperecomputing.com/emag), each with 32 ARM64 processors, 128 GB of RAM and 512 GB SSD.
Install Windows 10 PRO ARM64 on the Ampere Computing eMAG ARM servers
To begin with, please follow the instruction in the previous blog post to install Windows 10 on an ARM server. Windows Server ARM builds are not yet publicly available, so we have to stick to Windows 10 for now, but that’s enough for our purposes.
Enabling Hyper-V
In the Windows 10 build we used, trying to enable Hyper-V via the control panel GUI results in an error about virtualization not being available, but the command line installation works as expected:
As usual a reboot is required for the Hyper-V feature to be enabled.
Small detour – PowerShell on Windows 10 ARM64
By default, Windows 10 PRO ARM64 comes with PowerShell for x86 platform, which is very slow (to put it nicely).
Downloading the native PowerShell 7 for ARM64 from https://github.com/PowerShell/PowerShell/releases greatly improved the sysadmin experience on this Windows version. Now we can use the Hyper-V cmdlets straight from the native version with no extra steps required.
Windows VMs on Hyper-V
Creating a Windows VM on Hyper-V is straightforward. We used the same Windows Insiders VHDX as the host and created a VM.
For the Windows VM creation, you can either use Hyper-V Manager or PowerShell:
Note that on this hardware, only Generation 2 UEFI virtual machines are supported.
Side note: while the Hyper-V processes (VMMS.exe, VMWP.exe) are native, the Hyper-V Manager and the VM console are X86 emulated, which means again that they are very, very slow for managing your VMs. x64 emulation is in the works for the next version of Windows 10 for ARM64, I am eager to see if some of the tooling gets upgraded to x64 if not to the native versions.
Here I used an x86 32 bit qemu-img version that can run emulated on Windows ARM64, making the conversion step possible without resorting to Linux.
Next, I created the Hyper-V VM, disabled Secure boot and started the VM.
Aaand bummer!, the kernel was loaded by GRUB2 correctly, then it just hanged. That meant that the default Linux kernel version 5.4.x from Ubuntu 20.04 was not compatible with Hyper-V on ARM64 out of the box.
Luckily there are some pending Linux Kernel patches that add support for Linux running on Hyper-V ARM64:
Next, I chrooted the image, as it was the easiest way to access it, with a small twist. I needed to install the qemu-arm-static package and use chroot like this:
chroot /mnt/location-of-mounted-ubuntu-20-04 /usr/bin/qemu-aarch64-static /bin/bash
dpkg -i install linux-image-5.9.0+_5.9.0+-1_arm64.deb
# the dpkg install fails, because of the zz-update-grub hook, but the kernel is correctly installed
# manually tweak /boot/grub/grub.cfg with the new kernel / initrd names (linux-image-5.9.0-127961-geffcd948a62d)
Now I moved the newly tweaked image to the Hyper-V server, created a new VM using the image, remembered to disable ‘Secure Boot’ and all was set, we have a running ARM64 Linux VM on Hyper-V!
$vm | Set-VMFirmware -EnableSecureBoot Off
At a first glance we experienced some soft lockup issues, which possibly hint at some additional work to be done in the Hyper-V Linux Integration Services (LIS) to improve performance, but no show stopper so far.
Wrapping up
Enabling Hyper-V and creating Windows VMs on Windows 10 PRO ARM64 was pretty straightforward.
Creating Linux VMs required atweaked Linux kernel, but I am confident that those patches will land in the upstream kernel in the near future.
The built-in emulated tooling for Hyper-V is slow, but when the newer native PowerShell ARM64 version is installed, the experience improves.
Some features like nested virtualization are not yet available for Hyper-V on ARM64.
Pythonis an interpreted programming language that is used in multiple fields, ranging from education, scientific purposes to DevOps and cloud infrastructure. According to http://pypl.github.io/PYPL.html, Pythonis ranked first in Google’s search popularity.
This post is meant to snapshot the current state of the art of the Pythonecosystem on Windows ARM64, focusing on one main area: package support.
Package support is critical for Pythondevelopers in order to port their work to this new architecture, especially when relying on third party packages from Pypi or other sources as it usually happens. No one desires to spend time rewriting their application just to support another architecture or use non-upstreamed package sources, which are hard to maintain and non-trivial to test.
For the sake of this post, we are going to use Python 3.9, being the latest stable release. We are going to look at three main scenarios: multi-purpose, scientific and DevOps. We used GitHub actions to create reproducible builds leveraging a parameterized workflow: https://github.com/ader1990/cpython-builder. All packages were either install using pip with the following flags “–force-reinstall –no-binary :all:” or using “python setup.py install“.
Generally speaking, packages that are 100% Python and do not include native code, have no setup issues on ARM64 either. Even at runtime, unless they rely on some specific CPU architectural feature (e.g. checking for os.uname()) there are no reasons for them not to work. The creation of binary launcher is a different story though, as we will see in the next section. Besides that, the real issues typically arise when the package includes native code which needs to be compiled with an ARM64 C/C++ compiler on Windows as part of the setup process unless wheel files are made available on Pypi. We will get into this further down.
Out of the 25 packages, 23 were successfully built using the ARM64 Python versions on Windows 10 ARM64. numpy and cffi required patching for ARM64. setuptoolsand pip, although they could be successfully built, needed special attention in order to work correctly on ARM64:
setuptools package required patching for the Windows ARM64 launchers so that the executables resulted from the build were created as ARM64 binaries: https://github.com/ader1990/setuptools/commits/am_64. Without the ARM64 launchers, the binaries are created using the x64 launchers and cannot be run on Windows ARM64 without x64 emulation. Even if the x64 emulation is present in the most recent Windows 10 x64 version, the emulation introduces heavy performance penalties, which we will address in an upcoming post. The launchers were created using this simple launcher tree: https://github.com/ader1990/simple_launcher/tree/win_arm64.
There is work in progress to add these changes upstream so that the community can benefit from a proper frustration-free experience when using Windows on ARM64.
Python Scientific Packages
Here are some of the most popular packages in the scientific world: numpy, pandas,scikit and tensorflow. These packages are commonly used as building blocks in the ecosystem and are critical for using Python as the high level language in the AI/ML field.
Build / installation notes:
Visual Studio 2017 or 2019 (the free Community Edition is enough for open source use cases) and Windows 10 SDK are required for building these packages.
Cython 0.29.x is required to successfully build pandas.
Pandas has numpy and cython as a requirement, thus cython needs to be installed first and numpy needs to be built before pandas from the patched source.
Scikit could not be built, because it currently depends on external libraries FreeType, BLAS/LAPACK, which are not yet supported for ARM64, as the build system for Windows relies on Mingw32 (which does not have support yet for ARM64). More investigation is required on how to build those libraries for ARM64.
Tensorflow cannot be built, as it relies on Bazel build system which does not support ARM64 builds yet, we are digging more into this at the moment.
Python DevOps Packages
The DevOps world relies on Python to automate all kinds of processes: public cloud (AWS, Azure, etc), OpenStack services, instance provisioning (cloud-init, cloudbase-init). For this area, I have chosen awscli, cffi, greenlet, pywin32 and cloudbase-init.
The awscli package is the Python client for the Amazon Web Services, widely used by DevOps that want to automate workloads on AWS. The package could be easily installed without any code changes. It was not a surprise to see that the awscli package has more than half of the common multi-purpose packages above as requirements. Another observation is that the installation is slow, takes around 10 minutes to complete. We will cover more on the package installation / functionality performance in an upcoming post.
The cffi package is a recurring requirement for many packages that were initially created for *nix operating systems and were ported to Windows. As from its description, cffi is a “A Foreign Function Interface package for calling C libraries from Python”. To build this package on Windows ARM64, a NOOP patch was required: https://github.com/ader1990/cffi/tree/windows_arm64. There is work in progress to properly implement the functionality. A Visual Studio 2017 / 2019 environment is required for the build.
The greenlet package is of core importance for the Python ecosystem, as it implements “in-process sequential concurrent programming”, which is used for implementing asynchronous IO or service management. Greenlets pre-date native Python asyncio coroutines and are thus commonly used in applications that started off before Python 3.4 or required at some point Python 2.7 compatibility due to operating system constraints, like most OpenStack ecosystem projects. Building greenlet is a tad more complicated and all the building process is presented in the already upstreamed pull request: https://github.com/python-greenlet/greenlet/pull/224.
The cloudbase-init is a provisioning agent designed to initialize and configure guest operating systems on various cloud platforms like OpenStack, Azure, and AWS. The build and installation of cloudbase-init are fully covered in this extensive blog post: https://cloudbase.it/cloudbase-init-on-windows-arm64.
Wrapping up
Most common packages can be used out of the box, without any code changes, whilst the scientific and DevOps packages required some level of patching to support the new architecture, as some of those packages rely heavily on C-extensions and even inline assembly code.
There’s still work to be done in order to reach an experience level similar to the Windows x64 counterpart. To begin with, package owners need to push binary wheels to Pypi to avoid the need for a local build environment and a Continuous Integration / delivery (CI / CD) infrastructure would be very useful for such purpose, stay tuned!
This is the beginning of a short series detailing how to deploy OpenStack on ARM64, using Docker containers with Kolla and Kolla-ansible.
The objective of this first post is to create the ARM64 container images and push them to a remote registry in order to be used later on, when deploying our OpenStack cloud. We are going to use Azure Container Registry to store the images, but any other OCI compliant registry will do.
Create a container registry
Let’s start by creating the container registry and related access credentials. This can be done anywhere, e.g. from a laptop, doesn’t need to be ARM. All we need is to have the Azure CLI installed.
az login
# If you have more than one Azure subscription, choose one:
az account list --output table
az account set --subscription "Your subscription"
Next, let’s create a resource group and a container registry with a unique name. Choose also an Azure region based on your location.
RG=kolla
ACR_NAME=your_registry_name_here
LOCATION=eastus
az group create --name $RG --location $LOCATION
az acr create --resource-group $RG --name $ACR_NAME --sku Basic
We’re now creating two sets of credentials, one with push and pull access to be used when creating the images and one with pull only access to be used later on during the OpenStack deployment.
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)
SERVICE_PRINCIPAL_NAME=acr-kolla-sp-push
SP_PASSWD=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpush --query password --output tsv)
SP_APP_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)
echo "SP_APP_ID=$SP_APP_ID"
echo "SP_PASSWD=$SP_PASSWD"
SERVICE_PRINCIPAL_NAME=acr-kolla-sp-pull
SP_PASSWD_PULL_ONLY=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpull --query password --output tsv)
SP_APP_ID_PULL_ONLY=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)
echo "SP_APP_ID_PULL_ONLY=$SP_APP_ID_PULL_ONLY"
echo "SP_PASSWD_PULL_ONLY=$SP_PASSWD_PULL_ONLY"
Create and push the OpenStack Kolla container images
It’s now time to switch to an ARM server where the Kolla container images will be built. We are going to use a Lenovo server with an eMAG 32 cores Armv8 64-bit CPU provided by Ampere Computing. The host operating system is Ubuntu 20.04, but the following instructions can be easily adapted to other Linux distros.
Let’s start with installing the dependencies and add your current user to the docker group (or create a separate user).
Now we can install Kolla in a Python virtual environment and get ready to start building our container images. The OpenStack version is the recently released Victoria but a previous version can be used if needed (e.g. Ussuri).
mkdir kolla-buildcd kolla-build
python3 -m venv venvsource venv/bin/activate
pip install wheel
# Install Kolla, Victoria version
pip install "kolla>=11,<12"
The pmdk-tools Ubuntu package is not available on ARM, so we need to remove it from the nova-compute docker image build. This is done by creating a “template override” that we are going to pass to the build process.
tee template-overrides.j2 << EOT
{% extends parent_template %}
# nova-compute
{% set nova_compute_packages_remove = ['pmdk-tools'] %}
EOT
We can now build the container images and push them to the registry. This will take a while since it’s building and pushing container images for all OpenStack projects and services. Alternatively, it is possible to reduce the number of containers to a subset by creating a profile in kolla-build.conf as explained here.
In the previous blog post we created the ARM OpenStack Kolla container images and we can now proceed with deploying OpenStack. The host is a Lenovo server with an Ampere Computing eMAG 32 cores Armv8 64-bit CPU, running Ubuntu Server 20.04. For simplicity, this will be an “All-in-One” deployment, where all OpenStack components run on the same host, but it can be easily adapted to a multi-node setup.
Let’s start with installing the host package dependencies, in case those are not already there, including Docker.
Let’s check if everything is ok (nothing gets deployed yet):
ansible -i all-in-one all -m ping
kolla-genpwd is used to generate random passwords for every service, stored in /etc/kolla/passwords.yml, quite useful:
kolla-genpwd
Log in to the remote Docker registry, using the registry name and credentials created in the previous post:
ACR_NAME=# Value from ACR creation
SP_APP_ID_PULL_ONLY=# Value from ACR SP creation
SP_PASSWD_PULL_ONLY=# Value from ACR SP creation
REGISTRY=$ACR_NAME.azurecr.io
docker login $REGISTRY --username $SP_APP_ID --password $SP_PASSWD
Now, there are a few variables that we need to set, specific to the host environment. The external interface is what is used for tenant traffic.
VIP_ADDR=# An unallocated IP address in your management network
MGMT_IFACE=# Your management interface
EXT_IFACE=# Your external interface
# This must match the container images tag
OPENSTACK_TAG=11.0.0
Time to write the main configuration file in /etc/kolla/globals.yml:
The registry password goes in /etc/kolla/passwords.yml:
sed -i "s/^docker_registry_password: .*\$/docker_registry_password: $SP_PASSWD_PULL_ONLY/g" /etc/kolla/passwords.yml
Cinder, the OpenStack block storage component, supports a lot of backends. The easiest way to get started is by using NFS, but LVM would be a great choice as well if you have unused disks.
# Cinder NFS setup
CINDER_NFS_HOST=# Your local IP
# Replace with your local network CIDR if you plan to add more nodes
CINDER_NFS_ACCESS=$CINDER_NFS_HOST
sudo mkdir /kolla_nfs
echo "/kolla_nfs $CINDER_NFS_ACCESS(rw,sync,no_root_squash)" | sudo tee -a /etc/exports
echo "$CINDER_NFS_HOST:/kolla_nfs" | sudo tee -a /etc/kolla/config/nfs_shares
sudo systemctl restart nfs-kernel-server
The following settings are mostly needed for Octavia, during the next blog post in this series:
# Increase the PCIe ports to avoid this error when creating Octavia pool members:
# libvirt.libvirtError: internal error: No more available PCI slots
sudo mkdir /etc/kolla/config/nova
sudo tee /etc/kolla/config/nova/nova-compute.conf << EOT
[DEFAULT]
resume_guests_state_on_host_boot = true
[libvirt]
num_pcie_ports=28
EOT
# This is needed for Octavia
sudo mkdir /etc/kolla/config/neutron
sudo tee /etc/kolla/config/neutron/ml2_conf.ini << EOT
[ml2_type_vlan]
network_vlan_ranges = physnet1:100:200
EOT
Time to do some final checks, bootstrap the host and deploy OpenStack! The Deployment will take some time, this is a good moment for a coffee.
Congratulations, you have an ARM OpenStack cloud! Now we can get the CLI tools to access it:
pip3 install python-openstackclient python-barbicanclient python-heatclient python-octaviaclient
kolla-ansible post-deploy
# Load the vars to access the OpenStack environment
. /etc/kolla/admin-openrc.sh
The next steps are optional, but highly recommended in order to get the basic functionalities, including basic networking, standard flavors and a basic Linux image (Cirros):
# Set you external netwrork CIDR, range and gateway, matching your environment, e.g.:
export EXT_NET_CIDR='10.0.2.0/24'
export EXT_NET_RANGE='start=10.0.2.150,end=10.0.2.199'
export EXT_NET_GATEWAY='10.0.2.1'
./venv/share/kolla-ansible/init-runonce
All done! We can now create a basic VM from the command line:
# Create a demo VM
openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1
You can also head to “http://<VIP_ADDR>” and access Horizon, OpenStack’s web ui. The username is admin and the password is in /etc/kolla/passwords.yml:
In the next post we will add to the deployment Octavia, the load balancer as a service (LBaaS) component, Enjoy your ARM OpenStack cloud in the meantime!
P.S.: In case you would like to delete your whole deployment and start over:
In part 2 of this series about OpenStack on ARM64, we got to the point where our cloud is fully deployed with all the Compute (VMs), Software Defined Networking (SDN) and Software Defined Storage (SDS) up and running. One additional component that we want to add is a Load Balancer as a Service (LBaaS), which is a key requirement for pretty much any high available type of workload and a must-have feature in any cloud.
OpenStack’s current official LBaaS component is called Octavia, which replaced the older Neutron LBaaS v1 project, starting with the Liberty release. Deploying and configuring requires a few steps, which explains the need for a dedicated blog post.
Octavia’s reference implementation uses VM instances called Amphorae to perform the actual load balancing. The octavia-worker service takes care of communicating with the amphorae and to do that we need to generate a few X509 CAs and certificates used to secure the communications. The good news is that starting with the Victoria release, kolla-ansible simplifies a lot this task. Here’s how to:
# Change the following according to your organization
echo "octavia_certs_country: US" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_state: Oregon" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organization: OpenStack" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organizational_unit: Octavia" | sudo tee -a /etc/kolla/globals.yml
# This is the kolla-ansible virtual env created in the previous blog post
cd kolla
source venv/bin/activate
sudo chown $USER:$USER /etc/kolla
kolla-ansible octavia-certificates
The communication between Octavia and the Amphorae needs an isolated network, as we don’t want to share it with the tenant network for security reasons. A simple way to accomplish that is to create a provider network with a dedicated VLAN ID, which is why we enabled Neutron provider networks and OVS VLAN segmentation in the previous post. Again, starting with Victoria, this got much easier with kolla-ansible.
# This is a dedicated network, outside your management LAN address space, change as needed
OCTAVIA_MGMT_SUBNET=192.168.43.0/24
OCTAVIA_MGMT_SUBNET_START=192.168.43.10
OCTAVIA_MGMT_SUBNET_END=192.168.43.254
OCTAVIA_MGMT_HOST_IP=192.168.43.1/24
OCTAVIA_MGMT_VLAN_ID=107
sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_network:
name: lb-mgmt-net
provider_network_type: vlan
provider_segmentation_id: $OCTAVIA_MGMT_VLAN_ID
provider_physical_network: physnet1
external: false
shared: false
subnet:
name: lb-mgmt-subnet
cidr: "$OCTAVIA_MGMT_SUBNET"
allocation_pool_start: "$OCTAVIA_MGMT_SUBNET_START"
allocation_pool_end: "$OCTAVIA_MGMT_SUBNET_END"
gateway_ip: "$OCTAVIA_MGMT_HOST_IP"
enable_dhcp: yes
EOT
Unless there is a dedicated network adapter, a virtual ethernet one can be used. This needs to be configured at boot and added to the OVS br-ex switch.
# This sets up the VLAN veth interface
# Netplan doesn't have support for veth interfaces yet
sudo tee /usr/local/bin/veth-lbaas.sh << EOT
#!/bin/bash
sudo ip link add v-lbaas-vlan type veth peer name v-lbaas
sudo ip addr add $OCTAVIA_MGMT_HOST_IP dev v-lbaas
sudo ip link set v-lbaas-vlan up
sudo ip link set v-lbaas up
EOT
sudo chmod 744 /usr/local/bin/veth-lbaas.sh
sudo tee /etc/systemd/system/veth-lbaas.service << EOT
[Unit]
After=network.service
[Service]
ExecStart=/usr/local/bin/veth-lbaas.sh
[Install]
WantedBy=default.target
EOT
sudo chmod 644 /etc/systemd/system/veth-lbaas.service
sudo systemctl daemon-reload
sudo systemctl enable veth-lbaas.service
sudo systemctl start veth-lbaas.service
docker exec openvswitch_vswitchd ovs-vsctl add-port \
br-ex v-lbaas-vlan tag=$OCTAVIA_MGMT_VLAN_ID
A few more Octavia kolla-ansible configurations…
echo "enable_octavia: \"yes\"" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_network_interface: v-lbaas" | sudo tee -a /etc/kolla/globals.yml
# Flavor used when booting an amphora, change as needed
sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_flavor:
name: "amphora"
is_public: no
vcpus: 1
ram: 1024
disk: 5
EOT
sudo mkdir /etc/kolla/config/octavia
# Use a config drive in the Amphorae for cloud-init
sudo tee /etc/kolla/config/octavia/octavia-worker.conf << EOT
[controller_worker]
user_data_config_drive = true
EOT
…and we can finally tell kolla-ansible to deploy Octavia:
Octavia uses a special VM image for the Amphorae, which needs to be built for ARM64. We prepared Dockerfiles for building either an Ubuntu or a CentOS image, you can choose either one in the following snippets. We use containers to perform the build in order to isolate the requirements and be independent from the host OS.
ARM64 needs a trivial patch in the diskimage-create.sh build script (we also submitted it upstream):
git clone https://opendev.org/openstack/octavia -b stable/victoria
# Use latest branch Octavia to create Ubuntu image
cd octavia
# diskimage-create.sh includes armhf but not arm64
git apply ../0001-Add-arm64-in-diskimage-create.sh.patch
cd ..
Build the image (this will take a bit):
# Again, choose either Ubuntu or CentOS (not both!)
# Note the mount of /mnt and /proc in the docker container
# BEWARE!!!!! Without mounting /proc, the diskimage-builder fails to find mount points and deletes the host's /dev,
# making the host unusable
docker run --privileged -v /dev:/dev -v /proc:/proc -v /mnt:/mnt \
-v $(pwd)/octavia/:/octavia -ti amphora-image-build-arm64-ubuntu
# Create CentOS 8 Amphora image
docker run --privileged -v /dev:/dev -v $(pwd)/octavia/:/octavia \
-ti amphora-image-build-arm64-centos
Add the image to Glance, using the octavia user in the service project. The amphora tag is used by Octavia to find the image.
. /etc/kolla/admin-openrc.sh
# Switch to the octavia user and service project
export OS_USERNAME=octavia
export OS_PASSWORD=$(grep octavia_keystone_password /etc/kolla/passwords.yml | awk '{ print $2}')
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
openstack image create amphora-x64-haproxy.qcow2 \
--container-format bare \
--disk-format qcow2 \
--private \
--tag amphora \
--file octavia/diskimage-create/amphora-x64-haproxy.qcow2
# We can now delete the image file
rm -f octavia/diskimage-create/amphora-x64-haproxy.qcow2
Currently, we need a small patch in Octavia to properly render the userdata for the Amphorae:
# Patch the user_data_config_drive_template
cd octavia
git apply ../0001-Fix-userdata-template.patch
# For now just update the octavia-worker container, no need to restart it
docker cp octavia/common/jinja/templates/user_data_config_drive.template \
octavia_worker:/usr/lib/python3/dist-packages/octavia/common/jinja/templates/user_data_config_drive.template
Finally, let’s create a load balancer to make sure everything works fine:
# To create the loadbalancer
. /etc/kolla/admin-openrc.sh
openstack loadbalancer create --name loadbalancer1 --vip-subnet-id public1-subnet
# Check the status until it's marked as ONLINE
openstack loadbalancer list
Congratulations! You have a working LBaaS in your private cloud!!
Troubleshooting
In case something goes wrong, finding the root cause might be tricky. Here are a few suggestions to ease up the process.
# Check for errors
sudo tail -f /var/log/kolla/octavia/octavia-worker.log
# SSH into amphora
# Get amphora VM IP either from the octavia-worker.log or from:
openstack server list --all-projects
ssh ubuntu@<amphora_ip> -i octavia_ssh_key #ubuntu
ssh cloud-user@<amphora_ip> -i octavia_ssh_key #centos
# Instances stuck in pending create cannot be deleted
# Password: grep octavia_database_password /etc/kolla/passwords.yml
docker exec -ti mariadb mysql -u octavia -p octavia
update load_balancer set provisioning_status = 'ERROR' where provisioning_status = 'PENDING_CREATE';
exit;
The wait is over, Ceph 16 (Pacific) provides Windows native support as a result of the Cloudbase Solutions and Suse partnership.
Ceph is probably the most commonly used software defined storage solution out there. For example, according to surveys, more than 70% of the OpenStack deployments are powered by Ceph. That’s no wonder, considering that it can run on commodity hardware, yet manage to scale up to hundreds of storage nodes and deliver impressive performance.
Using Ceph on Windows had been a pain point, requiring proxies such as iSCSI gateways or re-exporting CephFS using Samba. Those approaches deliver suboptimal performance and overcomplicate the deployment architecture. All that hassle is now gone as RBD and CephFS can be used on Windows natively.
For best performance and features, Windows Server 2019 is recommended. Windows Server 2016 is supported as well but having a few known limitations. Older Windows Server versions as well as client versions such as Windows 10 might work as well but are not currently supported.
Installing
This MSI installer is the recommended way of installing Ceph on Windows. Along with the Ceph binaries, it also bundles the WNBD driver, which is used for mapping RBD images.
Please refer to those guides if you prefer manually building and installing Ceph and WNBD.
Configuring
Minimal configuration is needed in order to use Ceph on Windows. The default config file location is C:\ProgramData\ceph\ceph.conf.
Here’s a config sample. Don’t forget to fill in the right Ceph Monitor addresses and to provide a Ceph keyring file at the specified location. For the time being, slashes “/” must be used as path separators instead of backslashes “\”.
[global]
log to stderr = true
; Uncomment the following to use Windows Event Log
; log to syslog = true
run dir = C:/ProgramData/ceph/out
crash dir = C:/ProgramData/ceph/out
; Use the following to change the cephfs client log level
; debug client = 2
[client]
keyring = C:/ProgramData/ceph/keyring
; log file = C:/ProgramData/ceph/out/$name.$pid.log
admin socket = C:/ProgramData/ceph/out/$name.$pid.asok
; client_permissions = true
; client_mount_uid = 1000
; client_mount_gid = 1000
[global]
mon host = <ceph_monitor_addresses>
RBD
Rados Block Device (RBD) has been the primary focus of this effort. The same CLI that you’re probably already familiar with can be used to create RBD images and attach them to the host as well as Hyper-V virtual machines.
The following PowerShell sample creates an RBD image, attaches it to the host and adds an NTFS partition on top.
rbd create blank_image --size=1G
rbd device map blank_image
$mappingJson = rbd-wnbd show blank_image --format=json
$mappingJson = $mappingJson | ConvertFrom-Json
$diskNumber = $mappingJson.disk_number
# The disk must be online before creating or accessing partitions.
Set-Disk -Number $diskNumber -IsOffline $false
# Initialize the disk, partition it and create a fileystem.
Get-Disk -Number $diskNumber | `
Initialize-Disk -PassThru | `
New-Partition -AssignDriveLetter -UseMaximumSize | `
Format-Volume -Force -Confirm:$false
By default, all RBD mappings are persistent. The “ceph-rbd” service, which can be deployed using the above mentioned MSI installer, takes care of reattaching the RBD images after host reboots. This also allows adjusting the Windows service start order so that RBD images can be mapped before starting services that may depend on it.
The following screenshot shows an RBD image attached to a Hyper-V VM along with benchmark results. We will do a deep dive on the benchmarks in a future post (disclaimer: they look great!).
WNBD
RBD images are exposed as SCSI disks by using the WNBD Storport Miniport driver, written as part of this porting effort.
One interesting feature provided by the WNBD driver is the NBD client capability, allowing it to be used for attaching arbitrary NBD exports. This has been the default way of attaching RBD images before a more efficient mechanism was implemented. Due to its usefulness, this feature has been left in place, although it’s likely to be moved outside the driver at some point, leveraging the same API as rbd-wnbd does.
To mount a standalone NBD image, use the following:
CephFS support on Windows was our second main goal and is currently experimental. We’re using Dokany, which resembles FUSE, and a revamped version of the seemingly abandoned ceph-dokan project.
The following simple command mounts CephFS using the “X:” drive letter:
ceph-dokan.exe -l x
Current limitations
While the current state of the porting covers already the majority of the yes cases, there are a few limitations that you should be aware of. Those are missing features that are likely to be implemented in a subsequent version.
RBD images cannot be used yet for backing Cluster Shared Volumes (CSV) in Windows Server Failover Clustering (WSFC), which requires SCSI Persistent Reservations support
WNBD disks cannot be live resized
The Python bindings are unavailable
The ceph CLI tool cannot be used natively yet. However, it may be used through Windows Subsystem for Linux to contact running services.
Coming next
The next blog post will be focusing on RBD performance. Stick around for some more benchmark results and a performance tuning guide.
Make sure to check out the previous blog post introducing Ceph on Windows, in case you’ve missed it. We are now going to look at how the performance looks like.
Before this Ceph Windows porting, the only way to access Ceph storage from Windows was by using the Ceph iSCSI gateway, which can easily become a performance bottleneck. Our goal was to outperform the iSCSI gateway and to get as close to the native Linux RBD throughput as possible.
Spoiler alert: we managed to surpass both!
Test environment
Before showing some actual results, let’s talk a bit about the test environment. We used 4 identical baremetal servers, having the following specs:
CPU
Intel(R) Xeon(R) E5-2650 @ 2.00GHz
2 sockets
8 cores per socket
32 vcpus
memory
128GB
1333Mhz
network adapters
Chelsio T420-C
2 x 10Gb/s
LACP bond
9000 MTU
You may have noticed that we are not mentioning storage disks, that’s because the Ceph OSDs were configured to use memory backed pools. Keep in mind that we’re putting the emphasis on Ceph client performance, not the storage iops on the Linux OSD side.
We used an all-in-one Ceph 16 cluster running on top of Ubuntu 20.04. On the client side, we covered Windows Server 2016, Windows Server 2019 as well as Ubuntu 20.04.
The benchmarks have been performed using the fio tool. It’s highly configurable, commonly used for Ceph benchmarks and most importantly, it’s cross platform.
Below is a sample FIO command line invocation for anyone interested in repeating the tests. We are using direct IO, 2MB blocks, 64 concurrent asynchronous IO operations, testing various IO operations over 8GB chunks. When testing block devices, we are using the raw disk device as opposed to having an NTFS or ReFS partition. Note that this may require fio>=3.20 on Windows.
Windows power plan – few people expect this to be a concern for severs, but by default the “high performance” power plan is not enabled by default, which can lead to CPU throttling
Adding the Ceph and FIO binaries to the Windows Defender whitelist
Using Jumbo frames – we’ve noticed a 15% performance improvement
Enabling the CUBIC TCP congestion algorithm on Windows Server 2016
Test results
Baremetal RBD and CephFS IO
Let’s get straight to the RBD and CephFS benchmark results. Note that we’re using MB/s for measuring speed (higher is better).
It is worth mentioning that RBD caching has been disabled. As we can see, Windows Server 2019 manages to deliver impressive IO throughput. Windows Server 2016 isn’t as fast but it still manages to outperform the Linux RBD clients, including krbd. We are seeing the same pattern on the CephFS side.
We’ve tested the iSCSI gateway with an all-in-one Ceph cluster. The performance bottleneck is likely to become more severe with larger Ceph clusters, considering that the iSCSI gateway doesn’t scale very well.
Virtual Machines
Providing virtual machine block storage is also one of the main Ceph use cases. Here are the test results for Hyper-V Server 2019 and KVM on Ubuntu 20.04, both running Ubuntu 20.04 and Windows Server 2019 VMs booted from RBD images.
The WS 2019 Hyper-V VM managed to get almost native IO speed. What’s interesting is that it managed to fare better than the Ubuntu guest even on KVM, which is probably worth investigating.
WNBD
As stated in the previous post, our initial approach was to attach RBD images using the NBD protocol. That didn’t deliver the performance that we were hoping for, mostly due to the Winsock Kernel (WSK) framework, which is why we implemented from scratch a more efficient IO channel. For convenience, you can still use WNBD as a standalone NBD client for other purposes, in which case you may be interested to knowing how well it performs. It manages to deliver 933MB/s on WS 2019 and 280MB/s on WS 2016 in this test environment.
At the moment, rbd-wnbd uses DeviceIoControl to retrieve IO requests and send IO replies back to the WNBD driver, which is also known as inverted calls. Unlike the RBD NBD server, libwnbd allows adjusting the number of IO dispatch workers. The following table shows how the number of workers impacts performance. Keep in mind that in this specific case, we are benchmarking the driver connection, so no data gets transmitted from / to the Ceph cluster. This gives us a glimpse of the maximum theoretical bandwidth that WNBD could provide, fully saturating the available CPUs:
Apart from IO performance, we were also interested in making sure that a large amount of disks can be managed at the same time. For this reason, we wrote a simple Python script that creates a temporary image, attaches it to the host, performs various IO operations and then cleans it up. Here are the test results for 1000 iterations, 50 at a time. This test was essential in improving RBD performance and stability.
One might argue what is the point in running a cloud infrastructure software like OpenStack on top of another one, namely the Azure public cloud as in this blog post. The main use cases are typically testing and API compatibility, but as Azure nested virtualization and pass-through features came a long way recently in terms of performance, other more advanced use cases are viable, especially in areas where OpenStack has a strong user base (e.g. Telcos).
There are many ways to deploy OpenStack, in this post we will use Kolla Ansible for a containerized OpenStack with Ubuntu 20.04 Server as the host OS.
Preparing the infrastructure
In our scenario, we need at least one beefy virtual machine, that supports nested virtualization and can handle all the CPU/RAM/storage requirements for a full fledged All-In-One OpenStack. For this purpose, we chose a Standard_D8s_v3 size for the OpenStack controller virtual machine (8 vCPU, 32 GB RAM) and 512 GB of storage. For a multinode deployment, subject of a future post, more virtual machines can be added, depending on how many virtual machines are to be supported by the deployment.
# connect to Azure
az login
# create an ssh key for authentication
ssh-keygen
# create the OpenStack controller VM
az vm create `
--name openstack-controller `
--resource-group "openstack-rg" `
--subscription "openstack-subscription" `
--image Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest `
--location westeurope `
--admin-username openstackuser `
--ssh-key-values ~/.ssh/id_rsa.pub `
--nsg-rule SSH `
--os-disk-size-gb 512 `
--size Standard_D8s_v3
# az vm create will output the public IP of the instance
$openStackControllerIP = "<IP of the VM>"
# create the static private IP used by Kolla as VIP
az network nic ip-config create --name MyIpConfig `
--nic-name openstack-controllerVMNic `
--private-ip-address 10.0.0.10 `
--resource-group "openstack-rg" `
--subscription "openstack-subscription"
# connect via SSH to the VM
ssh openstackuser@$openStackControllerIP
# fix the fqdn
# Kolla/Ansible does not work with *.cloudapp FQDNs, so we need to fix it
sudo tee /etc/hosts << EOT
$(hostname -i) $(hostname)
EOT
# create a dummy interface that will be used by OpenVswitch as the external bridge port
# Azure Public Cloud does not allow spoofed traffic, so we need to rely on NAT for VMs to
# have internal connectivity.
sudo ip tuntap add mode tap br_ex_port
sudo ip link set dev br_ex_port up
OpenStack deployment
For the deployment, we will use the Kolla Ansible containerized approach.
Firstly, installation of the base packages for Ansible/Kolla/Cinder is required.
# from the Azure OpenStack Controller VM
# install ansible/kolla requirements
sudo apt install -y python3-dev libffi-dev gcc libssl-dev python3-venv net-tools
# install Cinder NFS backend requirements
sudo apt install -y nfs-kernel-server
# Cinder NFS setup
CINDER_NFS_HOST=$openStackControllerIP
# Replace with your local network CIDR if you plan to add more nodes
CINDER_NFS_ACCESS=$CINDER_NFS_HOST
sudo mkdir /kolla_nfs
echo "/kolla_nfs $CINDER_NFS_ACCESS(rw,sync,no_root_squash)" | sudo tee -a /etc/exports
echo "$CINDER_NFS_HOST:/kolla_nfs" | sudo tee -a /etc/kolla/config/nfs_shares
sudo systemctl restart nfs-kernel-server
Afterwards, let’s install Ansible/Kolla in a Python virtualenv.
As a next step, we need to configure the OpenStack settings
# This is the static IP we created initially
VIP_ADDR=10.0.0.10
# Azure VM interface is eth0
MGMT_IFACE=eth0
# This is the dummy interface used for OpenVswitch
EXT_IFACE=br_ex_port
# OpenStack Train version
OPENSTACK_TAG=11.0.0
# now use the information above to write it to Kolla configuration file
sudo tee -a /etc/kolla/globals.yml << EOT
kolla_base_distro: "ubuntu"
openstack_tag: "$OPENSTACK_TAG"
kolla_internal_vip_address: "$VIP_ADDR"
network_interface: "$MGMT_IFACE"
neutron_external_interface: "$EXT_IFACE"
enable_cinder: "yes"
enable_cinder_backend_nfs: "yes"
enable_neutron_provider_networks: "yes"
EOT
After the deployment, we need to create the admin environment variable script.
pip3 install python-openstackclient python-barbicanclient python-heatclient python-octaviaclient
kolla-ansible post-deploy
# Load the vars to access the OpenStack environment
. /etc/kolla/admin-openrc.sh
Let’s make the finishing touches and create an OpenStack instance.
# Set you external network CIDR, range and gateway, matching your environment, e.g.:
export EXT_NET_CIDR='10.0.2.0/24'
export EXT_NET_RANGE='start=10.0.2.150,end=10.0.2.199'
export EXT_NET_GATEWAY='10.0.2.1'
./venv/share/kolla-ansible/init-runonce
# Enable NAT so that VMs can have Internet access and be able to
# reach their floating IP from the controller node.
sudo ifconfig br-ex $EXT_NET_GATEWAY netmask 255.255.255.0 up
sudo iptables -t nat -A POSTROUTING -s $EXT_NET_CIDR -o eth0 -j MASQUERADE
# Create a demo VM
openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1
Conclusions
Deploying OpenStack on Azure is fairly straightforward, with the caveat that the OpenStack instances cannot be accessed from the Internet without further changes (this affects only inbound traffic, the OpenStack instances can access the Internet). Here are the main changes that we introduced to be able to perform the deployment in this scenario:
Add a static IP on the first interface that will be used as the OpenStack API IP
Set the OpenStack Controller FQDN to be the same as the hostname
Create a dummy interface which will be used as the br-ex external port (there is no need for a secondary NIC, as Azure drops any spoofed packets)
Add iptables NAT rules to allow OpenStack VM outbound (Internet) connectivity
When GitHub Actions was introduced, it gave private repositories the ability to easily create a CI/CD pipeline by simply creating a yaml file within their repository. No special software was needed, no external access had to be granted to third party CI/CD systems. It just worked. One year later, this feature was made available, for free, to public repositories as well. Now, any project hosted on GitHub can enable its own CI/CD pipeline, by creating a workflow. By default, the jobs run on a GitHub hosted runner, which is a virtual machine spun up in Azure using the Standard_DS2_v2 size. Also, you have a choice of images for various operating systems and versions, which are bundled with a lot of the common libraries and SDKs that are used in projects throughout GitHub.
If you want to test your project, you have a fleet of GitHub hosted runners primed and ready to execute a request from your repository to run a workflow. The virtual machines the tests run on are replaced after each run. This means you always get a clean machine whenever you want to run a test. Which is great, and in most cases is more than enough to run a series of unit tests or even integration tests. The workflows have a maximum duration of 6 hours, after which the job is automatically canceled and the runner is cleaned up.
But what happens if you want to run your tests on an operating system that is not in the list of supported images? Or what if you need more disk space? More CPU/Memory? What if you’re testing a huge project like Flatcar which needs to build many packages as part of their pipeline? What if you need access to a GPU, or some other specialized hardware?
Well, in that case GitHub recommends you set up your own self-hosted runners.
But can you do this easily? Does it require a huge learning curve? Complicated setups? I mean, I want my own runners, but not if I have to go through piles of howtos to get them.
The answer is: yes, it can be done easily. We’ll get to that soon. But first, we need to understand the problem that’s being solved.
About self hosted runners
Self hosted runners are compute resources (virtual machines, containers or bare metal servers), on which you install and run the GitHub Actions Runner. This runner will then connect to GitHub, and become available within your repository, organization or enterprise. You can then target that particular runner in your workflow, using labels.
There are two ways a runner can be registered:
Persistent
Ephemeral
Persistent runners are set up and manually maintained by you. You install them, add them to your repository and use them however many times you wish. Persistent runners are capable of running as many jobs as you throw at them. However, it falls onto you, to make sure that the machine is cleaned up and in working order after each job run. Otherwise, new jobs that get scheduled to it will most likely fail, or at the very best they will give you unreliable results.
Ephemeral runners accept only one job, after which they are automatically removed by GitHub from the list of available runners. This ensures that you always get a fresh machine to run your tests, but in this case, you need to implement some sort of auto-scaling that will tear down the runner that completed a job, and replace it with a new one. These runners give you the best experience, as they are fresh and untouched by previous tests.
We’ll be focusing on ephemeral runners in this article, and a way to automatically scale and maintain a pool of those.
The challenges of auto scaling
Auto-scaling of runners is done using GitHub web hooks. Whenever a new workflow job is triggered, GitHub will push an event via web hooks that will let you know that a job has be queued and a new worker is needed. If a worker is already online and idle, that worker is selected, and another web hook is triggered that lets you know a job is now in_progress. Finally, when a job finishes, a final web hook is triggered with a completed message. As part of the queued web hook, we also get a list of labels that the job is targeting.
We can use this information to implement our auto-scaling solution. Now here comes the tricky part. We need some sort of automation that will spin up runners that match the requested label. A label describes the needs of the workflow job. So we need a way to model that request into an operation that will set up exactly the kind of runner that is suited for that job. If your workflow requests a label called gpu, you need to spin up a runner with access to a GPU. If your workflow requests a label called hpc you may need to set up a runner with access to lots of CPU and memory. The idea is to be able to define multiple types of runners and make them available to your workflows. After all, this is the reason you might decide to use self-hosted runners instead of the default ones provided by GitHub.
You may have your own specialized hardware that you want to make available to a workflow, or you may have some spare hardware gathering dust and want to give it new life. Or you may have access to multiple cloud accounts that you could leverage to spin up compute resources of various types.
Garm is a self-hosted, automated system that maintains pools of GitHub runners on potentially any IaaS which has an API that will allow us to create compute resources. Garm is a single binary written in Go that you can run on any machine, within your private network. It requires no central management system, it doesn’t need to call home and is fully open source under the Apache 2.0 license.
Garm is meant to be easy to set up, easy to configure and hopefully, something you can forget about once it’s up and running. There are no complicated concepts to understand, no lengthy setup guide, no administrator guide that could rival the New York phone book in thickness. Garm is a simple app that aims to stay out of your way.
The only API endpoint that needs to be public, is the web hook endpoint, which GitHub calls into. It’s how GitHub lets garm know that a new runner is needed and that old runners need to be cleaned up.
Everything else can be hidden away behind a reverse proxy.
Where can garm create compute resources?
Right now garm has native support for LXD and an external provider for OpenStack and Azure. The current external OpenStack and Azure Providers are just a sample at the moment, but it can be used for testing and as an example for creating new external providers that can enable garm to leverage other clouds. External providers are executables that garm calls into, to manage the lifecycle of instances that end up running the GitHub Action Runner. They are similar to what containerd does when it comes co CNIs. As long as those binaries adhere to the required interface, garm can use them.
Sounds like garm spins up virtual machines?
In short: yes, but it doesn’t have to use VMs exclusively. I’ll explain.
We focused on virtual machines for the initial release because of their isolation from the host. Running workflows for public repositories is not without risks, so we need to be mindful of where we run jobs. The isolation offered by a virtual machine is desirable in favor of that of a container. That being said, there is no reason why a provider can’t be written for any system that can spin up compute resources, including containers.
In fact, writing a provider is easy, and you already have two examples of how to do it. With a little over 400 lines of bash, you could write a provider for virtually anything that has an API. And it doesn’t have to be bash. You could use anything you prefer, as long as the API for external providers is implemented by your executable.
In any case, I think it’s time to have a look at what garm can do.
Defining repositories/organizations
This article won’t go into details about how to set up garm. Those details are laid out in the project home page on GitHub. Instead, I’ll show you how to use it to manage your runners.
Garm has three layers:
Repositories or organizations
Pools of runners
The runners
Repositories and organizations can have multiple pools. Each pool can have different settings, can each use a different provider and will spin up multiple runners of the same type. When defining a new repository or organization, we need a Personal Access Token (PAT) to be configured in garm. Repositories use PATs to request registration tokens for runners, list existing runners and potentially forcefully remove them if the compute instance becomes defunct (on the roadmap). You can define multiple PATs and configure each repository or organization to use a different one.
Here is an example of defining a repository:
Creating pools
A pool of runners will create a number of runners of the same type inside a single provider. You can have multiple pools defined for your repository and each pool may have different settings with access to different providers. You can create one pool on LXD, another pool on OpenStack, each maintaining runners for different operating systems and with different sizes.
Let’s define a pool for the previously created repository:
We created a pool on LXD using default as a flavor and ubuntu:20.04 as an image. For LXD, garm maps flavors to profiles. The image names are the same images you would use to spin up virtual machines using the lxc command. So this pool will spin up an Ubuntu 20.04 image from the usual ubuntu: remote and will apply the default profile.
You can create new LXD profiles with whatever resources your runners need. Need multiple disks, more CPU or access to a number of different networks? Add them to the profile. The VMs that will be created with that profile will automatically have the desired specifications.
Let’s enable the pool and have it spin up the runners:
By default, when you create a new pool, the maximum number of runners will be set to 5 and the minimum idle runners will be set to 1 (configurable during create). This means that this particular pool will create a maximum number of 5 runners. The minimum idle runner option, attempts to maintain at least 1 runner in idle state, ready to be used by a GitHub workflow.
If you want more total runners or more idle runners, you can update the pool:
Now let’s add a new pool for the same repository, but this time we’ll add it on the external OpenStack provider:
On OpenStack flavor maps to the OpenStack flavor and image maps to the glance image. The flavor in OpenStack, aside from the basic resources it can configure, has the ability to target specific hosts via host aggregates and grant access to specialized hardware like GPUs, FPGAs, etc. If you need runners with access to special hardware, have a look at host aggregates.
Now that we have our pools and a few runners up and running, let’s try them out in a workflow:
We can see that we have five runners in total. Three on the LXD pool and another 2 on the OpenStack pool. As we trigger workflows in github, garm spins up new runners to replace the ones that are currently being used, maintaining that minimum idle runner count.
That’s all. If you want to try it out, head over to the project home page on GitHub and take it for a spin. Fair warning, this is an initial release, so if you run into any trouble, drop us a line.
AmpereComputing were kind enough to send us at our Cloudbase Solutions office the new version of their top-tier server product, which incorporates their latest Ampere ALTRA ARM64 Processor. The server version has a beefy dual-socket AMPERE ALTRA setup, with 24 NVME slots and up to 8 TB installable RAM, spread over 8 channels.
Let the unboxing begin!
Unboxing Started
First, we can see the beautiful dual-socket setup, each ALTRA CPU being nicely placed between the multiple fans in a row and the back end. The server height format is 2U, which was required for the dual-socket and multi-NVME placement.
The AMPERE ALTRA CPU in the box has a whopping 80 cores which boast 3.0 Ghz clock speed in this variant (Altra AC-108021002P), built on 7nm architecture. This server setup can take advantage of two such CPUs, with 160 cores spread over 2 NUMA nodes. Also, the maximum amount of RAM can go to 8TB (4TB / socket), if using 32 DIMMs of 256GB each. In our setup, we have 2 x 32GB ECC in dual-channel setup for each NUMA node, totaling 128GB.
From the networking side perspective, we have a BMC network port, 2 x 1Gb Intel i350 and an Open Compute Rack port (OCP 3.0) for one PCIe Gen4 network card, in our case a Broadcom BCM957504-N425G, that supports 100Gbps over the 4 ports (25Gbps/each). Furthermore, if you are a Tensorflow or AI workload aficionado, there is plenty of room for 3x Double Width GPU or 8x Single Width GPU (example: 3x Nvidia A100 or 8x Nvidia T4).
Intel 350 Dual Port + IPMI portOCP 3 Broadcom Network Card with 4x25Gpbs ports
The storage setup comes with 24 NVME PCIe ports, where we have one 1TB SSD, plus another M2 NVME SSD that has been placed inside the server, southwards to the first socket.
24 NVME slots, one from the left being active
After the back cover has been put back, the server placed in the rack and connected to a redundant power source (the two power sources have 2000 Watts each), let’s go and power on the system!
Before the OS installation, let’s make a small detour and check out the IPMI web interface, which is based on the MEGARAC SP-X WEB UI, a very fast UI. The board is an Ast2600, firmware version 0.40.1. The H5Viewer console is snappy, and a complete power cycle takes approximately 2 minutes (from a powered off system state to the Operating System bootloader start).
This version of Windows 11 has been installed as a trial only for demo purposes. At this moment, Windows 11 ARM64 build 25201 was available for download. The installation procedure is not yet so straightforward, as there is no ISO currently provided by Microsoft. The following steps were required in order to install the above Windows version on the Altra Server:
After the server has been rebooted and completing the Out Of The Box Experience (OOBE) steps, Windows 11 ARM64 is ready to be used.
As you can see, Windows properly recognises the CPU and NUMA architecture, with 2 NUMA nodes with 80 cores each. We tried to see if a simple tool like cpuconsume will properly work while running on a certain NUMA Node, using the command “start /node <numa node> /affinity” and we observed that the CPU load was also spread on the other node, so we decided to investigate this initially weird behaviour.
In Windows, there are two concepts that apply from the perspective of NUMA topology: Processor Groups and NUMA Support, both being implemented without any caveats for processors with a maximum of 64 cores. Here comes the interesting part, as in our case, one processor has 80 cores, which is by default split in 3 Processor Groups in the default Windows installation. The first group aggregates 64 cores from NUMA Node 0, group 3 aggregates 64 cores from NUMA Node 1, while Group 2 aggregates the remaining 16 + 16 cores from NUMA Node 0 and NUMA Node 1, for a total of 32 cores. If cpuconsume.exe is started with this command: “cmd /c start /node 0 /affinity 0xffffffffff cpuconsume.exe -cpu-time 10000”, there is 50% chance that it is started on Processor Group 0 or Processor Group 1. Also, changing the affinity from Task Manager, while it was working to change the affinity in the same group, it did not work to change it to processors from multiple Processor Groups.
Fortunately, AMPERE ALTRA’s firmware configuration offers a great trade-off solution, which makes running Windows application simpler on NUMA Nodes – it has a configurable topology: Monolithic, Hemisphere and Quadrant.
The Monolithic configuration is the default one, where there is 1:1 NUMA Node to Socket (CPU). In this case, there are 3 Processor Groups on Windows, the ones explained above. Setting the configuration to “Quadrant” did not seem to produce any changes on how the Windows Kernel sees it vs the Monolithic one. The remaining and best option was “Hemisphere”, where the cores are split into 4 NUMA Nodes, and Windows Kernel maps those NUMA Nodes to 4 Processor Groups, with 40 cores each. In this case, Windows applications can have the affinity correctly set using the “start /node <node number> /affinity <cpu mask>” command.
CPU Affinity
In the above screenshot, you can see that consume.exe has been instantiated on each NUMA Node and configured to use only 39 cores (mask 0xfffffffffe).
On the Windows Driver support, the Broadcom OCP3 card is not currently supported on this Windows version. There is no support either for the Intel I350 network cards. Fortunately, all the Mellanox Series are supported and we could add one such card to the system from the Ampere EMAG series we had received before (see https://cloudbase.it/cloudbase-init-on-windows-arm64/). For the management network, we used a classic approach – an USB 2.0 to 100Mb Ethernet adapter, that suited perfectly our installation use case.
From the virtualization perspective, we could easily turn on Hyper-V and try out booting a Windows and a Linux Hyper-V Virtual Machines. Windows works perfectly out of the box as a Virtual Machine, while Linux VMs require kernel >=5.15.rc1, which includes this patch that adds Hyper-V ARM64 boot support. For example, Ubuntu 22.04 or higher will boot on Hyper-V. Some issues were noticed on the virtualization part, nested virtualization is currently not supported (failure to start the Virtual Machine if processor virtualization extensions are enabled) and the Linux Kernel needs to have the kernel stall issues fixed, as seen in the screenshot below.
Windows Hyper-V
The performance of the hardware is well managed by the latest Windows version, the experience was flawless during the short period. We will come back with an in-depth review of performance and stability, AmpereComputing did a great job with their latest server products, kudos to them!
This is the first blog post in a series about running mixed x86 and ARM64 Kubernetes clusters, starting with a general architectural overview and then moving to detailed step by step instructions.
Kubernetes doesn’t need any introduction at this point, as it became the de facto standard container orchestration system. If you, dear reader, developed or deployed any workload in the last few years, there’s a very high probability that you had to interact with a Kubernetes cluster.
Most users deploy Kubernetes via services provided by their hyperscaler of choice, typically employing virtual machines as the underlying isolation mechanism. This is a well proven solution, at the expense of an inefficient resource usage, causing many organizations to look for alternatives in order to reduce costs.
One solution consists in deploying Kubernetes on top of an existing on-premise infrastructure running a full scale IaaS solution like OpenStack, or a traditional virtualization solution like VMware, using VMs underneath. This is similar to what happens on public clouds, with the advantage of allowing users to mix legacy virtualized workloads with modern container based applications on top of the same infrastructure. It is a very popular option, as we see a rise in OpenStack deployments for this specific purpose.
But, as more and more companies are interested in dedicated infrastructure for their Kubernetes clusters, especially for Edge use cases, there’s no need for an underlying IaaS or virtualization technology that adds unnecessary complexity and performance limitations.
This is where deploying Kubernetes on bare metal servers really shines, as the clusters can take full advantage of the whole infrastructure, often with significant TCO benefits. Running on bare metal allows us to freely choose between the x64 and ARM64 architectures, or a combination of both, taking advantage of the lower energy footprint provided by ARM servers with the compatibility offered by the more common x64 architecture.
A Kubernetes infrastructure comes with non-trivial complexity, which requires a fully automated solution for deployment, management, upgrades, observability and monitoring. Here’s a brief list the key components in the solution that we are about to present.
Host operating system
When it comes to Linux there’s definitely no lack of options. We needed a Linux distro aimed at lean container infrastructure workloads, with a large deployment base on many different physical servers, avoiding a full fledged traditional Linux server footprint. We decided to use Flatcar, for a series of reasons:
Longstanding (in cloud years) proven success, being the continuation of CoreOS
CNCF incubating project
Active community, expert in container scenarios, including Cloudbase engineers
Support for both ARM64 and x64
Commercial support, provided by Cloudbase, as a result of the partnership with Microsoft / Kinvolk
The host OS options are not limited to Flatcar, we tested successfully many other alternatives, including Mariner and Ubuntu. This is not trivial, as packaging and optimizing images for this sort of infrastructure requires significant domain expertize.
Bare metal host provisioning
This component allows to boot every host via IPMI (or other API provided by the BMC), install the operating system via PXE and in general configure every aspect of the host OS and Kubernetes in an automated way. Over the years we worked with many open source host provisioning solutions (MaaS, Ironic, Crowbar), but we opted for Thinkerbell in this case due to its integration in the Kubernetes ecosystem and support for Cluster API (CAPI).
Distributed Storage
Although the solution presented here can support traditional SAN storage, the storage model in our scenario will be distributed and hyperconverged, with every server providing compute, storage and networking roles. We chose Ceph (deployed via Rook), being the leading open source distributed storage and given our involvement in the community. When properly configured, it can deliver outstanding performance even on small clusters.
Networking and load balancing
While traditionally Kubernetes clusters employ Flannel or Calico for networking, a bare metal scenario can take advantage of a more modern technology like Cilium. Additionally, Cilium can provide load balancing via BGP out of the box, without the need for additional components like MetalLB.
High availability
All components in the deployment are designed with high availability in mind, including storage, networking, compute nodes and API.
Declarative GitOps
Argo CD offers a way to manage in a declarative way the whole CI/CD deployment pipeline. Other open source alternatives like Tekton or FluxCD can also be employed.
Observability and Monitoring
Last but not least, observability is a key area beyond simple logs, metrics and traces, to ensure that the whole infrastructure performs as expected, for which we employ Prometheus and Grafana. For monitoring, and ensuring that prompt actions can be taken in case of issues, we use Sentry.
Coming next
The next blog posts in this series will explain in detail how to deploy the whole architecture presented here, thanks for reading!
