KubeOpsVM

• 1: Release Notes
• 1.1: KubeOps VM 2.2.0_Beta0
• 1.2: KubeOps VM 2.2.0_Alpha3
• 1.3: KubeOps VM 2.1.0
• 1.4: KubeOps VM 2.0.4
• 1.5: KubeOps VM 2.0.3
• 1.6: KubeOps VM 2.0.2
• 1.7: KubeOps VM 2.0.1
• 1.8: KubeOps VM 1.7.4
• 1.9: KubeOps VM 1.7.3
• 1.10: KubeOps VM 1.7.2
• 1.11: KubeOps VM 1.7.1
• 1.12: KubeOps VM 1.7.0
• 1.13: KubeOps VM 1.7.0_Beta1
• 1.14: KubeOps VM 1.0.0_Beta0
• 1.15: KubeOps VM 1.0.0_Alpha2
• 1.16: KubeOps VM 1.0.0_Alpha1
• 1.17: KubeOps VM 1.0.0_Alpha0
• 2: Getting Started
• 2.1: About KubeOpsVM
• 2.2: About KubeOpsVM Licences
• 2.3: General Concepts of KubeOpsVM
• 2.4: Install KubeOps virtualization on a Dedicated Machine
• 3: How To Guides
• 3.1: How to install KubeOps Virtualization (kubevirt)
• 3.2: How to create and deploy a Custom Virtual Machine using KubeOps
• 3.3: How to access a virtual machine in a KubeOps Cluster
• 3.4: How to create a ContainerDisk from a local image
• 3.5: How to import a VM from vSphere and deploy it in a Cluster using KubeVirt
• 3.6: How to Monitor KubeVirt in Grafana
• 3.7: How to create L3 networking for VMs
• 3.8: How to back up VMs and vm snapshots with Velero
• 4: Reference
• 4.1: Detailed Explanation for Configuration File for Custom VMs
• 4.2: Troubleshooting
• 4.3: Commandline Operations

1 - Release Notes

1.1 - KubeOps VM 2.2.0_Beta0

KubeOpsVM 2.2.0_Beta0 - Release Date 23.07.2026

Changelog kubevirt 2.2.0_Beta0

Whats new?

+ Added Kubevirt-Manager support

1.2 - KubeOps VM 2.2.0_Alpha3

KubeOpsVM 2.2.0_Alpha3 - Release Date 06.07.2026

Changelog kubevirt 2.2.0_Alpha3

Whats new?

• Added Kubevirt-Manager support

1.3 - KubeOps VM 2.1.0

KubeOpsVM 2.1.0 - Release Date 16.06.2026

Changelog kubevirt 2.1.0

Whats new?

+ Fixed a bug where the update logic did nothing in airgap

1.4 - KubeOps VM 2.0.4

KubeOpsVM 2.0.4 - Release Date 02.03.2026

Changelog kubevirt 2.0.4

Fixes

+ Fixed a bug where imagePullSecrets were not created

1.5 - KubeOps VM 2.0.3

KubeOpsVM 2.0.3 - Release Date 10.02.2026

Changelog kubevirt 2.0.3

Fixes

+ Fixed a bug where imagePullSecrets were not created

1.6 - KubeOps VM 2.0.2

KubeOpsVM 2.0.2 - Release Date 28.01.2026

Changelog kubevirt 2.0.2

Fixes

+ Fixed a bug where imagePullSecrets were not created

1.7 - KubeOps VM 2.0.1

KubeOpsVM 2.0.1 - Release Date 22.12.2025

Changelog kubevirt 2.0.1

Fixes

+ Updated kubeopsctl versions

1.8 - KubeOps VM 1.7.4

KubeOpsVM 1.7.4 - Release Date 30.09.2025

Changelog RHEL8.10-KubeOps-VM-1.0.4

Fixes

+ Updated kubeopsctl versions

1.9 - KubeOps VM 1.7.3

KubeOpsVM 1.7.3 - Release Date 15.09.2025

Changelog RHEL8.10-KubeOps-VM-1.0.3

Fixes

+ Updated kubeopsctl versions

1.10 - KubeOps VM 1.7.2

KubeOpsVM 1.7.2 - Release Date 05.09.2025

Changelog RHEL8.10-KubeOps-VM-1.0.2

Fixes

+ Updated KOSI and kubeopsctl versions

1.11 - KubeOps VM 1.7.1

KubeOpsVM 1.7.1 - Release Date 30.07.2025

Changelog RHEL8.10-KubeOps-VM-1.0.1

Fixes

+ Updated KOSI and kubeopsctl versions

1.12 - KubeOps VM 1.7.0

KubeOpsVM 1.7.0 - Release Date 30.06.2025

Changelog RHEL8.10-KubeOps-VM-1.0.0

Whats new?

+ Updated KUBEOPSCTL version

1.13 - KubeOps VM 1.7.0_Beta1

KubeOpsVM 1.7.0_Beta1 RC0 - Release Date 09.05.2025

Changelog RHEL8.10-KubeOps-VM-1.0.0_Beta0

Fixes

- Updated KUBEOPSCTL version

1.14 - KubeOps VM 1.0.0_Beta0

KubeOpsVM 1.0.0_Beta0 - Release Date 30.04.2025

Changelog RHEL8.10-KubeOps-VM-1.0.0_Beta0

Whats new?

+ Added ISO for creating a Cluster

Fixes

+ Updated KOSI and kubeopsctl versions
+ Updated looping for KOSI login, IP addresses, and subscription manager
+ Updated KOSI and KUBEOPSCTL versions

Changelog RHEL8.10-Compliant-Base-1.0.0_Beta0

Whats new?

+ Added ISO for adding nodes to a Cluster

Changelog kubevirt-1.0.0_Beta0

Whats new?

+ Added new kosi package for creating vms in a kubernetes cluster

1.15 - KubeOps VM 1.0.0_Alpha2

KubeOpsVM 1.0.0_Alpha2 - Release Date 11.04.2025

Changelog RHEL8.10-KubeOps-VM-1.0.0_Alpha2

Updates

• Updated KOSI and KUBEOPSCTL versions

1.16 - KubeOps VM 1.0.0_Alpha1

KubeOpsVM 1.0.0_Alpha1 - Release Date 08.04.2025

Changelog RHEL8.10-KubeOps-VM-1.0.0_Alpha1

Updates

• Updated KOSI and kubeopsctl versions
• Updated looping for KOSI login, IP addresses, and subscription manager

1.17 - KubeOps VM 1.0.0_Alpha0

KubeOpsVM 1.0.0_Alpha0 - Release Date 01.04.2025

Changelog RHEL8.10-KubeOps-VM-1.0.0_Alpha0

New

• Added ISO for creating a Cluster

Changelog RHEL8.10-Compliant-Base-1.0.0_Alpha0

New

• Added ISO for adding nodes to a Cluster

Changelog kubevirt-1.0.0_Alpha0

New

• Added new kosi package for creating vms in a kubernetes cluster

2 - Getting Started

2.1 - About KubeOpsVM

This article provides an overview of KubeOpsVM, a virtualization solution designed for Kubernetes-based environments. It explains how KubeOpsVM enables the operation and management of virtual machines alongside containerized workloads within a shared orchestration platform.

What Is KubeOpsVM?

KubeOpsVM is a virtualization platform that extends Kubernetes-based infrastructure to support virtual machines. It allows organizations to manage VM-based and container-based workloads within the same environment, using Kubernetes as the orchestration layer.

By consolidating these workload types on a single platform, KubeOpsVM reduces the need for separate virtualization and container management systems. This approach simplifies infrastructure operations, improves consistency across environments, and supports more efficient resource utilization.

Why KubeOpsVM Matters

Many organizations adopting cloud-native technologies must continue to operate existing applications that depend on traditional virtual machines. Managing these workloads separately from containerized services often increases operational complexity and creates fragmentation across tools, processes, and infrastructure.

KubeOpsVM addresses this challenge by enabling virtual machines to run within Kubernetes environments. This makes it possible to manage legacy and modern workloads through a common control plane, helping organizations streamline administration and support gradual modernization strategies.

Key Features Driving Innovation

KubeOpsVM distinguishes itself by delivering a robust virtualization experience through Kubernetes integration. Its primary features include:

• Unified Workload Management - Manage virtual machines and containerized workloads through a shared Kubernetes-based platform and operational model.
• Scalability - Scale virtual machines in response to changing demand by using Kubernetes scheduling and resource management capabilities.
• High Availability - Support highly available virtualized workloads in multi-node environments through Kubernetes self-healing, scheduling, and failover mechanisms.
• Resource Efficiency - Run virtualized and containerized workloads on the same infrastructure to improve resource utilization and operational efficiency.

A Vision for the Future

KubeOpsVM is designed for virtualization in cloud-native environments. It addresses the growing need to operate virtual machines alongside container-based workloads in Kubernetes, helping organizations connect existing systems with modern platform architectures. This supports a more practical and gradual path toward modernization.

Whether the goal is to maintain legacy applications, improve deployment consistency, or increase workload portability, KubeOpsVM offers a unified approach to running VMs within Kubernetes-based platforms. As cloud-native adoption continues to grow, KubeOpsVM is positioned to support future infrastructure models where virtualized and containerized workloads operate together more seamlessly.

2.2 - About KubeOpsVM Licences

KubeOpsVM functionality is available with the Compliance + VM licence.

When you create an account, you receive a free 30 day trial version of our licences.

2.3 - General Concepts of KubeOpsVM

This article explores the core concepts of KubeOpsVM, a virtualization solution that integrates with Kubernetes to manage virtual machines alongside containerized workloads. It highlights how KubeOpsVM bridges traditional VMs and cloud-native systems, offering scalability and streamlined orchestration for modern infrastructure.

What is the purpose of a Virtual Machine?

A virtual machine (VM) is a software-based environment that simulates a physical computer, enabling it to run an operating system and applications independently from the host machine. Its primary purpose is to optimize hardware usage by allowing multiple operating systems to run on a single device. This setup is crucial for tasks like software development, where developers can test applications in various environments without needing separate hardware.

Security and isolation are other key advantages, as VMs keep applications separate from the host system, reducing the risk of system-wide damage from potential threats. They also simplify disaster recovery, since entire VM states can be backed up, cloned, or transferred easily. Additionally, VMs provide flexibility by enabling cross-platform compatibility, allowing users to run different operating systems simultaneously. This feature supports legacy software that might not be compatible with modern systems, ensuring continued access to older applications.

Virtual machines are integral to cloud computing, IT infrastructure, and personal computing due to their scalability, flexibility, and efficiency.

How does KubeOpsVM work?

KubeOpsVM is like a “computer inside a computer.” It allows you to run an operating system (OS) and applications in a sandboxed environment, separate from your actual physical machine.

In the example shown, any VM can be hosted on any agent (worker or master) within the cluster.

A virtual machine does not necessarily have to run on a specific machine. Depending on the current requirements, the VM is deployed on the appropriate machine in the cluster. kubevirt takes over the complete administration - no further intervention is necessary.

One of the main advantages of KubeOps virtualization is the asynchronous use and access of a VM and the deployment and execution of the VM depending on the available resources or machines. This ensures that each VM can always work independently of workloads.

2.4 - Install KubeOps virtualization on a Dedicated Machine

This guide describes how to set up a Kubernetes-based virtualization environment by using kubevirt automated installation routines. The setup is performed on a physical or virtual machine and runs largely automatically, requiring only minimal user input.

As part of the installation, KubeVirt and additional KubeOps components, such as the KOSI package manager, are installed automatically on the target node.

KubeVirt provides the virtualization layer within Kubernetes and enables virtual machines to run as Kubernetes-managed resources.

System architecture after installation

After the installation is complete, the environment consists of:

-a single-node Kubernetes cluster

• one running node functioning as both the control-plane and worker node
• one virtual machine ubuntu vm running in the namespace kubevirt
• one preconfigured ubuntu image running on that virtual machine

The ubuntu image includes predefined login credentials.

Requirements

Before you begin the installation, ensure that you have the credentials for the following accounts available:

• your KubeOps.net account
• your Red Hat Subscription Manager account

For more information about Red Hat Subscription Manager, see the Red Hat online documentation.

System Requirements

The ISO can be installed on either a physical machine or a virtual machine. Ensure that the target system meets the following minimum requirements.

The KubeOps-VM ISO is based on Red Hat Enterprise Linux (RHEL). Only RHEL 9.6 and 9.7 are supported.

Step 1 - Download the KubeOps-VM ISO File

Download the ISO file from the following location:
https://kubeops.net/users-only/welcome.

Step 2 - Install the KubeOps-VM ISO

During installation, you will be prompted to provide configuration values. Follow the on-screen instructions and enter the required information.

1. Boot your machine from the ISO.
2. Select Install Red Hat Enterprise to start the installation.

3. Select the hard disk on which the system will be installed.

4. Configure the network settings:

• Select your preferred networking device.
• Select whether the IP configuration is dynamic or static.
• Enter the IP address.
• Enter the subnet mask address.
• Enter the gateway address.
• Enter the DNS server address.
• Enter your preferred hostname. This will be the name of the machine. ( Do not use special characters in the hostname.)

5. Select your preferred activation method for the Red Hat Subscription Manager.
6. Select your preferred security method for the GRUB bootloader.
   We recommend the option “random password” - this will set up the security automatically. For more information about the GRUB bootloader and its security, refer to the respective documentation.
7. Enter your KubeOps.net account credentials.
   Notice The setup process may take several minutes. Do not shut down the machine while the installation is in progress.

8. Follow the remaining on-screen instructions to complete the installation.

Step 3 - Login to the Control-plane Node on the Cluster

During setup using an ISO file, the control-plane node was automatically preconfigured with a user (root) and its password.

1. Log in to the control-plane node remotely.
   For example, using ssh. The first time you log in, you must use the preconfigured password (see above). Use the hostname or the IP address that you assigned in a previous step during the installation.

ssh ko-admin@[HOSTNAME]

or

ssh ko-admin@[IP ADDRESS]

2. At first login, you will be prompted to change the default password. ollow the on-screen instructions. After the password change is complete, you will be automatically logged out.

   Notice During the first boot of the machine, additional software and parameters are configured in the background, such as the cluster setup and configuration.

3. Login to the machine again using your new credentials.

4. Most kubeops software require root privileges. Switch to root user.

sudo -i

5. To check the state of background configuration tasks, run the journalctl command.

journalctl -u clustersetup.service

Keep this view open while the installation finishes.

Check your kubeops virtualization by using the kubectl get pods command.

kubectl get pods -n kubevirt

The installation is complete when the status of all pods is set to “Running”. For example:

Note: Pods in the kubevirt namespace may initially remain in the Pending state. If this happens, you can either add worker nodes as described in How to join Nodes to a KubeOps Cluster or or remove the control-plane taint from the single node with kubectl taint nodes <control-plane-node-name> node-role.kubernetes.io/control-plane:NoSchedule-

6. To change the password of the root user, run command:

passwd

Step 4 - Install the example VM

After the platform installation is complete, you can deploy the example virtual machine.

1. Login to the machine using your updated credentials.
2. Switch to root user as most kubeops software needs root privileges.

sudo -i

3. Create a values file for the virtual machines:

Example:

ubuntuvmValues:
  name: ubuntu-vm
  namespace: kubevirt-vm 
  sshkeys:
    - key: <public ssh key, p.e. id_rsa.pub file>
  dataVolume:
    storageClassName: rook-cephfs
    source:
      secret: kubeops-auth

In this example,these values are saved in the vmvalues.yaml file.

4. Ensure that you are still logged in to KOSI using your kosi username. KOSI sessions remain valid for 24 hours, after that time you need to login again.

kosi login -u [USERNAME]

5. Use the KOSI package manager to search for available software packages. In this case, we restrict the search to KubeOps virtualization using the hub kubeops-vm and filter for ubuntu.

kosi search --hub kubeops-vm --ps ubuntu

6. Install the package using the kosi install command. The example VM configuration is named kubevirt-ubuntu-vm

kosi install --hub kubeops-vm kubeops/kubevirt-ubuntu-vm:1.0.0 -f vmvalues.yaml --dname ubuntu

Create the required secret

Create a custom secret for image access:

kubectl create secret generic kubeops-auth --from-literal=accessKeyId=<kubeopsaccount> --from-literal=secretKey="<password-token for imagepullregistry>" --namespace=<namespace of vm>

Note: The image pull token is not the same as the account password. If the registry uses OIDC integration, for example with Keycloak, use the appropriate token instead of the user password.

Additional requirements for air-gapped environments

For an airgap installation,

• Use a domain name for Harbor.
• Do not use an IP address with a port
• Do not manually copy images to the nodes or load them with containerd
• Ensure that all required images are available through the configured registry

You may also need to create an image pull secret named kubeops-vm. This secret uses the same registry credentials as the kubeops-auth secret created by the KubeOps platform tool, like rook-ceph.

You can create it with the following command:

kubectl -n kubevirt create secret docker-registry kubeops-vm --docker-server=<registry for image pulling> --docker-username=<username of registry> --docker-password=<password-token for imagepullregistry>

Step 5 - Access the example virtual machine

After deployment, you can access the example virtual machine remotely from the control-plane node. For example, using ssh.

1. Login to the machine using your updated credentials.
2. Retrieve the IP address of your example VM with following command:

kubectl get vmi -A

This command lists all virtual machine instances. Since we only have one VM installed, the output looks something like this.

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY

kubevirt    ubuntu-vm   22h   Running   192.168.16.48   stackedadmin   True

3. Log in to the example VM remotely (e.g. using ssh). Use the IP address from the previous step. Use the credentials of the running ubuntu image from the table below.

ssh ubuntu@192.168.16.48

Next Steps

For additional commands and operational tasks for KubeOps virtual machines, see Commandline Operations.

3 - How To Guides

In the following sections, you will find everything from initial setup and configuration, to advanced tips and tricks that will help you get the most out of the software. Our aim is to assist you in becoming proficient with KubeOpsVM, enhancing both your productivity and your user experience.

Lets get started on your journey to mastering KubeOpsVM!

3.1 - How to install KubeOps Virtualization (kubevirt)

KubeVirt provides a powerful way to run virtual machines on top of Kubernetes, creating a unified platform for both containerized and traditional workloads. This guide walks through the essential steps for getting KubeVirt up and running using the KOSI package manager - from pulling and preparing the KubeVirt package, to creating a custom configuration file with your credentials, and finally verifying a successful installation in the kubevirt-namespace. Once these steps are complete, you’ll be ready to deploy and manage virtual machines alongside your existing container infrastructure - all within a single Kubernetes cluster.

KubeVirt is a specialized package that extends the capabilities of the KubeOps platform to support machine virtualization.

Requirements

To follow the next steps, make sure you have your credentials ready from your KubeOps.net account.

Make sure you have prepared the following points:

• A KubeOps cluster with one master node.
• Access to the master node with root privileges.

Step 1 - Download KubeOpsVM

1. Log in to the master node remotely. For example, using ssh.
2. If necessary, switch to your cluster user. If you forgot your cluster user, check your cluster-values.yaml.

su <your linux username>

3. Log in to KOSI using your KOSI username.

kosi login -u [USERNAME]

4. Search for available software packages by using the KOSI package manager. In this case, we restrict the search to KubeOps virtualization using kubeops-vm.

kosi search --hub kubeops-vm

5. Log in to the local container registry on the master node using podman. Replace <master-ip> with the IP address of your master node.

podman login <master-ip>:30002 -u admin -p <harbor admin password> --tls-verify=false

6. The package is named kubeops/kubevirt:2.2.0_Beta0 and lies on the hub kubeops-vm. Download the package using the kosi pull command and save it with the desired file name. In this example, the file is saved as kubevirt-2.2.0_Beta0.tgz. The -r flag sets the source registry the container images are pulled from, and -t sets the local target registry the images are pushed to.

kosi pull --hub kubeops-vm kubeops/kubevirt:2.2.0_Beta0 -o kubevirt-2.2.0_Beta0.tgz -r <master-ip>:30002/kubeops -t 127.0.0.1:30002/kubeops

Step 2 - Remove the Taint of the Master Node (only necessary on single node cluster)

This step is only required for a single-node cluster.

If your cluster consists of only one node, you must remove the taint from the master node so that workloads can be scheduled on it.

Replace [NODENAME] with the name of you master node.

kubectl taint nodes [NODENAME] node-role.kubernetes.io/control-plane:NoSchedule-

Step 3 - Create Kubevirt Configuration File

Before installing the package, create a YAML configuration file. Save the following content in the same directory as the package file and name the file kubevirt-values.yaml.

Replace [your username] and [your harbor cli secret] with your KOSI username and your Harbor CLI secret.

The harbor-values must be adapted to your harbor instance.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1

deleteNs: false
localRegistry: true

packages:
  - name: kubevirt-manager
    enabled: true
    values:
      standard:
        namespace: kubevirt
        hostname: kubeops-vms.local
        nodeport: 30733
        ingressName: nginx
      advanced: {}

  - name: kubevirt
    enabled: true
    hub: kubeops-vm
    values:
      standard:
        namespace: kubevirt

        imagePullSecretCred:
          registry: 127.0.0.1:30002/kubeops
          user: [your username]
          password: [your harbor cli secret] # must be CLI Token from harbor

        kubevirtobject:
          spec:
            configuration:
              developerConfiguration:
                logVerbosity:
                  virtLauncher: 2
                  virtHandler: 3
                  virtController: 4
                  virtAPI: 5
                  virtOperator: 6

                featureGates:
                  - ExpandDisks
                  - CPUManager
                  - ExperimentalIgnitionSupport
                  - HypervStrictCheck
                  - Sidecar
                  - HostDevices
                  - Snapshot
                  - VMExport
                  - HotplugVolumes
                  - HostDisk
                  - ExperimentalVirtiofsSupport
                  - DownwardMetricsFeatureGate
                  - Root
                  - ClusterProfiler
                  - WorkloadEncryptionSEV
                  - VSOCK
                  - DisableCustomSELinuxPolicy
                  - KubevirtSeccompProfile
                  - DisableMDEVConfiguration
                  - PersistentReservation
                  - VMPersistentState
                  - MultiArchitecture
                  - AutoResourceLimitsGate
                  - AlignCPUs
                  - VolumesUpdateStrategy
                  - VolumeMigration
                  - InstancetypeReferencePolicy

                useEmulation: false

        cdiobject:
          spec:
            config:
              insecureRegistries:
                - registry.preprod.kubeops.net
              scratchSpaceStorageClass: rook-cephfs # Optional; Default: rook-cephfs
  - name: harbor
    enabled: false
    values:
      standard:
        namespace: harbor # optional, default is harbor
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        harborpass: "password" # required: set password for harbor access
        databasePassword: "Postgres_Password" # required: set password for database access
        redisPassword: "Redis_Password" # required: set password for redis access
        externalURL: http://10.2.10.11:30002 # required, the ip address and port, from which harbor is accessable outside of the cluster
        nodePort: 30002 # required
        hostname: harbor.local # required
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi # optional, default is 40Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            jobservice:
              jobLog:
                size: 1Gi # optional, default is 1Gi
                storageClass: "rook-cephfs" #optional, default is rook-cephfs
            database:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            redis:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            trivy: 
              size: 5Gi # optional, default is 5Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
      advanced: #  All values from https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration are overwritable

Step 4 - Install KubeOps-VM

1. Install the KubeVirt package using the prepared configuration file you created.

kosi install -p kubevirt-2.2.0_Beta0.tgz -f kubevirt-values.yaml

2. Verify the installation by checking the pods in the kubevirt namespace.

kubectl get pods -n kubevirt

The installation is complete when the status of all pods show the status “Running”. For example:

the following example, the kubevirt namespace is used. If KubeVirt is deployed in a different namespace, replace the value passed to the -n parameter.

kubectl -n kubevirt create secret docker-registry kubeops-vm --docker-server=127.0.0.1:30002 --docker-username=admin --docker-password=<admin password - see harborpass value in kubeopsctl.yaml>

Step 5 - Pull and Install KubeVirt Manager Package

1. Pull the KubeVirt Manager package.

kosi pull --hub kubeops kubeops/kubevirt-manager:2.2.0_Beta0 -o kubevirt-manager-2.2.0_Beta0.tgz -r master-ip:30002/kubeops -t 127.0.0.1:30002/kubeops

2. Install the KubeVirt Manager package.

kosi install -p kubevirt-manager-2.2.0_Beta0.tgz -f kubevirt-values.yaml

Next Steps

After the installation is complete, you can start creating and deploying a custom virtual machine. For more information, see How to create and deploy a Custom Virtual Machine using KubeOps.

3.2 - How to create and deploy a Custom Virtual Machine using KubeOps

Creating custom virtual machines on top of Kubernetes is a powerful way to unify containerized and traditional workloads. In this introductory guide, you’ll learn how to set up your KubeOps cluster to deploy a Cirros-based VM using KubeVirt, from checking off a few key prerequisites to preparing and applying a concise YAML configuration file.

Whether you’re a seasoned Kubernetes user looking to expand your virtualization capabilities or just getting started with hybrid infrastructure, these steps will walk you through everything from generating the custom VM file to verifying its running status and accessing it over SSH. By following this process, you’ll have a seamlessly integrated virtual machine environment in no time - ready to support your application needs.

Requirements

Make sure you have prepared the following points:

• A KubeOps cluster with at least one master node and root privileges.
• A running kubeVirt instance. (see How to install KubeOps Virtualisation (KubeVirt)).
• Enough resources to support a virtual machine

For example:

To follow this guide, use the following values to configure the custom virtual machine. If you wish, you can adjust the values yourself. Make sure that the adjusted values are applied accordingly everywhere.

Additional Information

For users to understand the most basic functionality of KubeVirt, one must know about the following fundemental concepts:

KubeVirt offers different methods to engage with the virtual machines through specialized custom resources (CR). On the lowest level is the VirtualMachineInstance (VMI) which represents the stateless instance of a virtual machine. These VMIs can be managed by higher level resources, such as VirtualMachine (VM), which realizes a stateful VMI that can be stopped and started while maintaining its data and state.

The manifests for these resources contain a multitude of parameters that surpass the needs of most users. Consequently this guide will not explain all parameters.

How to create and deploy a Custom Virtual Machine

The following steps will show you, how to manually create and access a virtual machine, using Kubernetes with a running KubeVirt instance.

Step 1 - Create a Configuration File for the Virtual Machine

To set up a virtual machine you need to create a configuration file. The configuration file contains information, such as name of the vm as well as its namespace, used resources (e.g. RAM) and states (e.g. running, halted). Use the file contents below and make desired changes.

Use the following content to create a basic vm configuration file custom-vm.yaml.

This file needs to be adapted at spec.spec.volumes[0].containerDisk.image. You need to insert your local harbor deployments’ port.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: example-vm-cirros
  name: example-vm
  namespace: kubevirt
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: example-vm-cirros
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 64M
      terminationGracePeriodSeconds: 0
      volumes:
      - name: containerdisk
        containerDisk:
          image: "127.0.0.1:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0"
          imagePullSecret: kubeops-vm
          imagePullPolicy: Always
      - cloudInitNoCloud:
          userDataBase64: IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK
        name: cloudinitdisk

The image, from which the virtual machine is created, can be found in the first section of spec.templates.spec.volumes of the manifest. It is used in the form of a ContainerDisk, meaning the image is part of a container image, within a remote repository.

This manifest will use a preconfigured image from the repository kubevirt/cirros-container-disk-demo:latest with the following credentials:

KubeVirt supports the ability to assign a startup script to a virtual machine. The script is executed automatically when the VM initializes. There are multiple datasources supported, which will inject those scripts through the use of ephemeral disks.

One of those datasources is called cloudInitNoCloud, which is mentioned here as the second volume in spec.templates.spec.volumes. VMs with the cloud-init package installed will detect the ephemeral disk and execute custom userdata scripts at boot.

Step 2 - Transfer the VM Image to your Local Registry

• Pull the image from the kubeops registry

 podman pull registry.kubeops.net/kubeops/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0

• Tag the image for the local registry

In the following command you need to insert your master1 ip address into the placeholder

podman tag registry.kubeops.net/kubeops/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0 [your master1 ip]:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0

• Push the image to the local registry

In the following command you need to insert your master1 ip address into the placeholder

podman push [your master1 ip]:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0 --tls-verify=false

Step 3 - Deploy the Virtual Machine

Deploy the vm using the kubectl command and the custom-vm.yaml file.

kubectl apply -f custom-vm.yaml

Check the state of the vm.

kubectl get vmi -A

This command lists all virtual machine instances. Since we only have one VM installed, the output looks something like this.

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY
kubevirt    customvm   22h   Running   192.168.16.48   stackedadmin   True

Step 4 - Access the Custom VM

To connect to your newly created VM follow the documentation on how to access a Virtual Machine in a KubeOps Cluster

3.3 - How to access a virtual machine in a KubeOps Cluster

Requirements

Before proceeding, ensure that the following prerequisites are met:

• A KubeOps cluster with at least one master node.
• A running virtual machine in said cluster.
• The credentials of the running image within the virtual machine (username and password).

For example:

• From where to access the vm:
• Nodes inside the cluster
• Outside the cluster

sudo -i

kubectl get vmi -A

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY

kubevirt    [CUSTOMVM-1]   22h   Running   [192.168.16.XXX]   stackedadmin   True
kubevirt    [CUSTOMVM-2]   22h   Running   [192.168.16.YYY]   stackedadmin   True

ssh [USERNAME]@[IP-ADDRESS]

apiVersion: v1
kind: Service
metadata:
  name: vm-ssh-forwarding
  namespace: kubevirt
spec:
  type: NodePort
  ports:
    - port: 22
      nodePort: 30226
      targetPort: 22
      protocol: TCP
  selector:
    kubevirt.io/vm: customvm # Binds directly to the underlying VM pod

 kubectl apply -f serviceforwarding.yaml

NAME                ENDPOINTS          AGE
vm-ssh-forwarding   10.244.138.61:22   5d23h

kubectl get vmi -A

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY

kubevirt    example-vm   22h   Running   192.168.16.48   stackedadmin   True

ssh [vm user]@[your master1 ip] -p 30226 # this port must match the one defined in your service 

3.4 - How to create a ContainerDisk from a local image

KubeVirt supports multiple ways of using images to create virtual machines. One of the most convenient methods is using the ContainerDisk feature, which allows VM disks to be stored and distributed as container images through a registry.

To create a ContainerDisk, users can simply inject the image into a container in qcow2 or raw format. The container should be based on scratch and the images must be placed into the /disk directory. No other content is required.

The following Dockerfile builds a ContainerDisk, which you can store in a OCI registry such as harbor or the Docker Hub:

FROM kubevirt/container-disk-v1alpha
ADD <your Image>.qcow2 /disk

You can the build the OCI arteiact with the command:

podman build . -t

Note: The ContainerDisk must be readable by the user with the UID 107 (qemu).

If you want to access the virtual machine, it is recommended to use a SSH key in the userdata startup section of your YAML file:

      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            ssh_authorized_keys:
              - ssh-rsa AAAAB3NzaK8L93bWxnyp test@test.com            

3.5 - How to import a VM from vSphere and deploy it in a Cluster using KubeVirt

How to Import a VM from vSphere and Deploy it in a Cluster using KubeVirt

This guide outlines the steps to export a virtual machine from vSphere, convert it to the qcow2 format, and deploy it inside a Kubernetes cluster using KubeVirt.

Requirements

To follow these steps, ensure you have:

A VM in the vSphere environment.

A Kubernetes cluster with KubeVirt installed.

Access to a container registry to store the VM image.

Step 1 - Export OVF Template from vSphere

In vSphere, locate the VM you want to export.

Ensure the VM is powered off before exporting.

Right-click the VM, navigate to Template, and select Export OVF Template.

Provide a name for the exported template and click OK.

The export process will generate three files: *.vmdk, *.nvram, and *.ovf.

Only the *.vmdk file is required for further processing.

Step 2 - Convert the VMDK File to QCOW2 Format

KubeVirt requires disk images in qcow2 format. Use the qemu-img tool to convert the .vmdk file:

qemu-img convert -f vmdk -O qcow2 [input-file].vmdk [output-file].qcow2

Replace [input-file].vmdk with the name of your downloaded VMDK file and [output-file].qcow2 with your desired output name.

Step 3 - Create a Container Image from the QCOW2 File

To use the converted disk inside KubeVirt, create a ContainerDisk by building a container image with the qcow2 file.

Create a Dockerfile

Create a Dockerfile with the following content:

FROM scratch 

ADD --chown=107:107 [filename].qcow2 /disk/

Replace [filename].qcow2 with the actual filename of your converted disk image.

Build and Push the Image

Execute the following commands to build the image and push it to a container registry:

podman build -t [registry]/[namespace]/[vm-image-name]:[tag] -t localhost/[namespace]/[vm-image-name]:[tag] .

podman push [registry]/[namespace]/[vm-image-name]:[tag]

Replace the placeholders as follows:

[registry]: The URL of your container registry

[namespace]: Your project name

[vm-image-name]: A name for the VM image

[tag]: The version tag (e.g., 1.0.0)

Once the image is pushed, it can be used as a ContainerDisk in KubeVirt to deploy a virtual machine.

Step 4 - Deploy the VM using KubeVirt

Create a secret for authentication:

kubectl create secret generic kubeops-vm-sec --from-literal=accessKeyId=kubeops --from-literal=secretKey=registry-token --namespace kubevirt-vms-import

kubectl -n kubevirt-vms-import create secret docker-registry kubeops-vm --docker-server=registry.kubeops.net --docker-username="<username>" --docker-password="<registry-token>"

 kubectl patch serviceaccount default -n kubevirt-vms-import -p '{"imagePullSecrets": [{"name": "kubeops-vm"}]}'

Create a manifest file for the virtual machine:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: sk2-master01
  name: sk2-master01-vm
  namespace: kubevirt-vms-import
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: sk2-master01
    spec:
      domain:
        devices:
          disks:
          - name: datavolumedisk
            disk:
              bus: virtio
          - name: cloudinitvolume
            disk:
              bus: virtio
        resources:
          requests:
            memory: 4092M
      volumes:
      - name: datavolumedisk
        dataVolume:
          name: sk2-master01-dv
      - name: cloudinitvolume
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            chpasswd:
              list: |
              expire: False
            ssh_pwauth: True
            disable_root: false            
  dataVolumeTemplates:
  - metadata:
      name: sk2-master01-dv
    spec:
      storage:
        resources:
          requests:
            storage: 68Gi
        storageClassName: rook-cephfs
        accessModes:
          - ReadWriteMany
      source:
        registry:
          url: "docker://registry.kubeops.net/kubeops-vm/kubeops/kubevirt/sk2_master01_export:1.0.0" # (change the url to the image you want to deploy)
          secretRef: kubeops-vm-sec

kubectl apply -f filename.yaml

Step 5 - Verify Deployment

Run the following commands to check if the VM is deployed successfully:

kubectl get dv -n kubevirt-vms-import

kubectl get vmi -n kubevirt-vms-import

Ensure that the DataVolume is successfully downloaded and the VirtualMachineInstance is running.

3.6 - How to Monitor KubeVirt in Grafana

Monitor KubeVirt VMIs with Grafana and Prometheus

This guide explains how to monitor KubeVirt Virtual Machine Instances (VMIs) using Grafana and Prometheus. By the end of this guide, you will have a Grafana dashboard that displays metrics related to KubeVirt VMIs.

Prerequisites

• A running multi-node Kubernetes cluster with KubeVirt installed

  ⚠️ Note: Single-node clusters are not supported due to component scheduling and storage limitations.

• A functional storage provider (e.g., Rook-Ceph or any other CSI-compatible solution) must be installed and available

  ⚠️ Required for deploying both Prometheus and Grafana, as they rely on PersistentVolumeClaims (PVCs)

• Prometheus deployed in the cluster

• Grafana installed and configured to connect to Prometheus

Step 1: Ensure Prometheus is Collecting KubeVirt Metrics

KubeVirt exposes various metrics that Prometheus can scrape. These metrics are documented here: KubeVirt Metrics.

To check whether Prometheus is collecting KubeVirt metrics:

1. Open the Prometheus Dashboard

2. Search for KubeVirt-related metrics, for example kubevirt_vmi_memory_available_bytes.

3. If no metrics appear, ensure the KubeVirt servicemonitor has been added to Prometheus:

    kubectl get servicemonitor -A | grep kubevirt
   

4. Verify that KubeVirt was installed correctly and that monitorNamespace and monitorAccount are configured correctly. How to install KubeOps Virtualization (kubevirt)

5. Reload Prometheus operator.

6. Reload KubeVirt operator.

Step 2: Import the KubeVirt Grafana Dashboard

KubeVirt provides a ready-to-use Grafana dashboard JSON file: KubeVirt Grafana Dashboard.

To import it into Grafana:

1. Open the Grafana Dashboard.
2. Navigate to Dashboards > New > Import.
3. Paste the JSON content from the KubeVirt dashboard file.
4. Select Prometheus as the data source.
5. Click Load.

Step 3: Customize the Dashboard (Optional)

To add specific VMI-related metrics to the dashboard:

1. Click Edit on a panel.

2. Use PromQL queries to fetch desired VMI metrics, e.g.,

   • CPU Usage: rate(kubevirt_vmi_cpu_usage_seconds_total[5m])
   • Memory Usage: kubevirt_vmi_memory_available_bytes
   • Network Traffic: rate(kubevirt_vmi_network_receive_bytes_total[5m])

3. Save the dashboard.

Note: You can view query output in the Prometheus dashboard.

Conclusion

You have successfully set up Grafana to monitor KubeVirt VMIs using Prometheus. You can now track VMI performance and resource usage effectively.

3.7 - How to create L3 networking for VMs

This guide explains how to configure L3 networking for KubeVirt virtual machines (VMs) using Multus. With this setup, VMs can communicate across nodes through dedicated secondary networks, which improves network segmentation and deployment flexibility in Kubernetes environments.

Prerequisites

Before you begin, ensure that the following requirements are met:

• A running Kubernetes cluster
• kubectl command-line tool configured to connect to your cluster
• Calico and Multus installed and configured
• KOSI is installed and authenticated successfully

Configuring Node Networking using Network Manager State (NMState)

Network Manager State (nmstate) is a declarative network configuration tool for Linux. It provides an API and CLI to manage network settings in a structured, state-driven manner.

In Kubernetes, nmstate is often used with NodeNetworkConfigurationPolicy (NNCP) to configure node networking dynamically, allowing for changes such as setting up VLANs, bridges, bonds, and other networking features.

1. Deploy the nmstate Operator

First, create the nmstate-values.yaml:

# nmstate-values.yaml
nmstateValues:
  namespace: nmstate

Deploy the nmstate operator and its CRDs using this file:

kosi install --hub kubeops kubeops/nmstate:2.0.1 -f nmstate-values.yaml

2. Configure Node Networking with NNCP

Create an nncp-values.yaml file to define network configurations dynamically, including bridges and routes:

# nncp-values.yaml
namespace: "nmstate"
nodes:
  - nodeName: cluster1master1
    nodeIP: 10.2.10.11
    bridgeIP: 10.10.1.1 # bridge IP for cluster1master1.
    bridgeSubnet: 10.10.1.0 # `/24` subnet for bridge
    nodeInterface: ens192 # use `ip a` or `ifconfig` to identify interface

  - nodeName: cluster1master2
    nodeIP: 10.2.10.12
    bridgeIP: 10.10.2.1
    bridgeSubnet: 10.10.2.0
    nodeInterface: ens192

  - nodeName: cluster1worker1
    nodeIP: 10.2.10.14
    bridgeIP: 10.10.3.1
    bridgeSubnet: 10.10.3.0
    nodeInterface: ens192

  - nodeName: cluster1worker2
    nodeIP: 10.2.10.15
    bridgeIP: 10.10.4.1
    bridgeSubnet: 10.10.4.0
    nodeInterface: ens192

Note:

If you’re adding a new node, assign it a unique bridgeIP and bridgeSubnet, that follow the existing addressing pattern.

For example, if your last node uses:

bridgeIP: 10.10.4.1

bridgeSubnet: 10.10.4.0/24

add the next node like this:

  - nodeName: <new_node>
    nodeIP: <node_ip>
    bridgeIP: 10.10.5.1
    bridgeSubnet: 10.10.5.0
    nodeInterface: ens192

Use this values file to deploy the node network configuration policies on all nodes:

kosi install --hub kubeops kubeops/nncp:2.0.1 -f nncp-values.yaml

3. Allow Forwarded Traffic on the Bridge

Run the following commands on the relevant nodes to allow traffic through the secondary bridge:

iptables -A FORWARD -i br-secondary -j ACCEPT
iptables -A FORWARD -o br-secondary -j ACCEPT

4. Verify Bridge Configuration and Node Connectivity

• Verify the Bridge on Each Node

To confirm that the bridge exists and has the expected IP address:

ip a show br-secondary

• Verify Attached Interfaces

List the bridge interfaces and verify that the expected interfaces are attached:

ip link show type bridge

• Verify Routes and Connectivity

Check if the expected routes are in place:

ip route show

• Test Connectivity Between Nodes

Test connectivity to another node’s bridge IP:

ping -c 4 <another_node_bridgeIP>

Install KubeVirt

For installation instructions, please refer to the KubeVirt Installation Guide

Networks and Interfaces

L3 Networking with Multus

To set up L3 networking with Multus, each worker node requires a separate NetworkAttachmentDefinition (NAD) with a unique IP address and subnet matching the node’s network configuration. This allows isolated network environments for each pod, enabling multi-interface support on pods and communication between nodes in the cluster.

Plan the Network

Before creating the NADs, define the IP ranges and subnets for each worker node to avoid conflicts and routing conflicts.

Example IP range mapping for worker nodes:

Example NAD Configuration for Worker Node 1

This example shows how to configure a NetworkAttachmentDefinition (NAD) for worker node 1 (cluster1worker1), with a static IP range assigned to its pods:

# nad_cluster1worker1.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: secondary-network-node1 # Assign unique names to each node's NAD
  annotations:
    k8s.v1.cni.cncf.io/nodeSelector: '{"kubernetes.io/hostname": "cluster1worker1"}'
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "secondary-network",
      "type": "bridge",
      "bridge": "br-secondary",
      "ipam": {
        "type": "static",
        "addresses": [
          {
            "address": "10.10.3.10/24", # use correct worker node's subnet and mention the appropriate IP for pod's network
            "gateway": "10.10.3.1"
          }
        ],
        "routes": [
          {
            "dst": "10.10.4.0/24",
            "gw": "10.10.3.1"
          }
        ]
      }
    }

Apply the K8s Manifest Definition files

To deploy the NetworkAttachmentDefinition (NAD) objects for your nodes, apply the following Kubernetes manifest files:

kubectl apply -f nad_cluster1worker1.yaml
kubectl apply -f nad_cluster1worker2.yaml

Verify the created Network Attachment Definition Objects

To ensure that the NetworkAttachmentDefinition (NAD) objects have been successfully created, run the following command:

kubectl get net-attach-def -A

Deploy Virtual Machines (VMs) and Attach the Secondary Networks

Once the NADs are created, you can deploy Virtual Machines (VMs) and attach them to the secondary networks. The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached.

Network Selection from NADs

Each VM must be connected to the correct network defined by the NetworkAttachmentDefinition (NAD) for its corresponding node. For example, if you have created a NAD for cluster1worker1 named secondary-network-node1, ensure that the VM manifest refers to this network in the annotations section and correctly links to the networkName under the multus configuration.

Ensure that the network name in the manifest matches the NAD definition.

Create Image Pull Secret

Please create a secret for pulling image from registry.kubeops.net.

kubectl -n "l3-networking" create secret docker-registry kubeops-vm \
  --docker-server=registry.kubeops.net \
  --docker-username=<kubeops-user> \
  --docker-password=<harbor_token>

Example VM Manifest for Secondary Network

The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached to a specific NAD:

# vm1.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: "ubuntu-vm"
  namespace: "l3-networking"
  annotations:
    k8s.v1.cni.cncf.io/networks: secondary-network-node1 # Ensure this matches the NAD network name
spec:
  nodeSelector:
    kubernetes.io/hostname: cluster1worker1 # edit for scheduling node on other node
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/size: small
    spec:
      domain:
        cpu:
          cores: 1
        devices:
          disks:
            - name: containervolume
              disk:
                bus: virtio
            - name: cloudinitvolume
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: secondary-network
            bridge: {}
        resources:
          requests:
            memory: 2048M
      networks:
      - name: default
        pod: {}
      - name: secondary-network
        multus:
          networkName: secondary-network-node1 # Ensure this matches the NAD network name
      volumes:
        - name: containervolume
          containerDisk:
            image: registry.kubeops.net/kubeops-vm/kubeops/ubuntu-cloudimg-focal:2.0.1
            imagePullSecret: kubeops-vm
            imagePullPolicy: Always
        - name: cloudinitvolume
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              chpasswd:
                list: |
                  ubuntu:ubuntu
                  root:toor
                expire: False
              ssh_pwauth: True
              disable_root: false              

Apply the VM Manifest

Once the NetworkAttachmentDefinition (NAD) objects have been created, you can deploy the Virtual Machines (VMs) and apply the manifests. The following commands apply the VM manifests to your Kubernetes cluster.

kubectl apply -f vm1.yaml
kubectl apply -f vm2.yaml

Verify the Networking

After the VMs start successfully, verify that the secondary network interfaces and routes are configured correctly.

On each VM

1. Check the IP Address

First, check the IP address assigned to the pod (VM). This is crucial to ensure that the secondary network interface has been correctly assigned an IP.

kubectl exec -n <namespace> <pod-1> -- ip a

2. Assign IP Address if Not Automatically Assigned (Optional)

kubectl exec -n <namespace> <pod-1> -- sudo dhclient enp2s0

3. Check the Routes

Verify the routing configuration on the VM to ensure that traffic can flow through the secondary network. Use the following command:

kubectl exec -n <namespace> <pod-1> -- ip r

This will display the routing table for the VM and confirm if the secondary network is correctly set up.

Verify VM Communication

After ensuring the network settings are correct, test the communication between VMs by pinging the secondary network IP of another pod.

Test Connectivity Between VMs

Use the ping command to check if the VMs can communicate over the secondary network. Replace <pod2_secondary_network_IP> with the actual IP address of the secondary network interface on the second VM.

kubectl exec -n <namespace> <pod-1> -- ping -c 2 <pod2_secondary_network_IP>

If the ping is successful, it confirms that the VMs are able to communicate over the secondary network.

Additional Notes:

• Ensure that you use the correct namespace and pod name while executing the commands.

• Make sure the secondary network interface is correctly configured on each VM, and the IP address and routes are properly assigned.

• If communication fails, check for issues in the NetworkAttachmentDefinition (NAD) configuration or verify that the correct CNI plugins are running.

3.8 - How to back up VMs and vm snapshots with Velero

Overview

This guide describes how to configure Velero to back up and restore KubeVirt virtual machines in a Kubernetes cluster.

Velero stores backup data in an S3-compatible object storage backend. In this example, MinIO is used as the backup storage target.

Prerequisites

Before you begin, ensure that the following prerequisites are met:

• A running Kubernetes cluster is available.
• Rook-Ceph is configured as the storage provider.
• An S3-compatible object storage service is available.( in this case MinIO)
• A KubeVirt virtual machine is running in the cluster

In following example MinIO is used.

Step 1 - Configure Velero

Define the Velero configuration in your values file:

veleroValues:
 namespace: "velero"
 accessKeyId: "minio"
 secretAccessKey: "minio123"
 backupLocationConfig:
   s3Url: "http://10.2.10.110:30090" # nodeport of minio

Enable Velero in the kubeopsctl.yaml file :

velero: true 

Apply the configuration:

kubeopsctl apply -f kubeopsctl.yaml

Step 2 - Verify S3 Bucket Access

The bucket for Velero ( default Velero ) in the s3 storage should be accessible for Velero, so ensure that the bucket exists and that Velero has permission to write to it.

In this example, the bucket is configured so that Velero has write access:

Step 3 - Create a Backup

To create a backup of the namespace containing the KubeVirt virtual machine, run:

velero backup get
velero backup create kubeops-vm-backup --include-namespaces <Namespace of kubevirt vm>

The name of the backup must be unique.

Step 4 - Verify the Backup

To review the backup status and details, run:

velero backup get
velero backup describe kubeops-vm-backup
velero backup logs kubeops-vm-backup

Confirm that the backup completed successfully before proceeding.

Step 5 - Restore the Backup

To restore the backup, run:

velero restore get
velero restore create kubeops-vm-restore --from-backup kubeops-vm-backup

The restore name must be unique.
After the restore is complete, verify that the virtual machine resources have been recreated successfully in the target namespace.

velero restore get
velero restore describe kubeops-vm-restore
velero restore logs kubeops-vm-restore

4 - Reference

4.1 - Detailed Explanation for Configuration File for Custom VMs

To set up a virtual machine you need to create a configuration file. The configuration file contains information, such as name of the VM, used resources (e.g. RAM) and states (e.g. running, halted).

Heed the comments (#) in the file below.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: example-vm-cirros
  name: example-vm
  namespace: {{ values.vmValues.namespace | object.default "kubevirt" }}
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: example-vm-cirros
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 64M
      terminationGracePeriodSeconds: 0
      volumes:
      - name: containerdisk
        containerDisk:
          image: {{ package.includes.containers.example.registry }}/{{ package.includes.containers.example.image }}:{{ package.includes.containers.example.tag }}
          imagePullSecret: kubeops-vm
          imagePullPolicy: Always
      - cloudInitNoCloud:
          userDataBase64: IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK
        name: cloudinitdisk

4.2 - Troubleshooting

Troubleshooting

virt-handler pods only 2/3 ready and status is CrashLoopBackOff

{"component":"virt-handler","level":"fatal","msg":"Failed to create an inotify watcher","pos":"cert-manager.go:105","reason":"too many open files","timestamp":"2026-07-14T07:55:17.060525Z"}              │
│ panic: Failed to create an inotify watcher

sudo sysctl -w fs.inotify.max_user_watches=1048576

sudo sysctl -w fs.inotify.max_user_instances=512

4.3 - Commandline Operations

Learn the most common command-line operations for managing KubeOps virtual machines - including creating a VM with a YAML configuration file, starting or stopping the VM by patching its runStrategy, viewing the VM or instance status with kubectl describe, restarting the VM (which deletes the instance and causes data loss), and deleting the VM either in a cascading manner (removing the VM and its instance) or by leaving orphaned instances.

Create a new virtual machine

The command kubectl with the paramater create uses the VM configuration file vm-config.yaml. For more detailed explaination on VM configuration files, see [Detailed Explanation for Configuration File for Custom VMs](Detailed Explanation for Configuration File for Custom VMs).

kubectl create -f vm-config.yaml

Start the virtual machine

Start the virtual machine VMNAME. Replace VMNAME with the name of your virtual machine,

kubectl patch virtualmachine VMNAME --type merge -p \
'{"spec":{"runStrategy":"Always"}}'

Status of a virtual machine

Print the status of the virtual machine VMNAME. Replace VMNAME with the name of your virtual machine,

kubectl describe virtualmachine VMNAME

Status of a virtual machine instance

kubectl describe virtualmachineinstance VMNAME

Stop the virtual machine instance

kubectl patch virtualmachine VMNAME --type merge -p \
'{"spec":{"runStrategy":"Halted"}}'

Restart the virtual machine

{{ hazard_notice }} Restarting the virtual machine will delete the virtual machine instance. This action cannot be undone. {{ /hazard_notice }}

kubectl delete virtualmachineinstance vm

Cascade delete (implicit)

Kubectl first deletes the virtual machine, followed by deleting the corresponding virtual machine instance.

Delete the virtual machine VMNAME. Replace VMNAME with the name of your virtual machine,

{{ hazard_notice }} This action cannot be undone. {{ /hazard_notice }}

kubectl delete virtualmachine VMNAME

Cascade delete (explicit)

Kubectl first deletes the virtual machine, followed by deleting the corresponding virtual machine instance.

Delete the virtual machine VMNAME. Replace VMNAME with the name of your virtual machine,

{{ hazard_notice }} This action cannot be undone. {{ /hazard_notice }}

kubectl delete virtualmachine VMNAME --cascade=true

Delete orphans

The running virtual machine is only detached, but not deleted.

Delete orphans of the virtual machine VMNAME. Replace VMNAME with the name of your virtual machine,

kubectl delete virtualmachine VMNAME --cascade=false