This is the multi-page printable view of this section. Click here to print.
In the following sections, you will find everything from initial setup and configuration, to advanced tips and tricks that will help you get the most out of the software. Our aim is to assist you in becoming proficient with KubeOpsVM, enhancing both your productivity and your user experience.
Lets get started on your journey to mastering KubeOpsVM!
KubeVirt provides a powerful way to run virtual machines on top of Kubernetes, creating a unified platform for both containerized and traditional workloads. This guide walks through the essential steps for getting KubeVirt up and running using the KOSI package manager - from pulling and preparing the KubeVirt package, to creating a custom configuration file with your credentials, and finally verifying a successful installation in the kubevirt-namespace. Once these steps are complete, you’ll be ready to deploy and manage virtual machines alongside your existing container infrastructure - all within a single Kubernetes cluster.
KubeVirt is a specialized package that extends the capabilities of the KubeOps platform to support machine virtualization.
To follow the next steps, make sure you have your credentials ready from your KubeOps.net account.
Make sure you have prepared the following points:
ssh.cluster-values.yaml.su <your linux username>
kosi login -u [USERNAME]
kubeops-vm.kosi search --hub kubeops-vm
podman. Replace <master-ip> with the IP address of your master node.podman login <master-ip>:30002 -u admin -p <harbor admin password> --tls-verify=false
kubeops/kubevirt:2.2.0_Beta0 and lies on the hub kubeops-vm. Download the package using the kosi pull command and save it with the desired file name. In this example, the file is saved as kubevirt-2.2.0_Beta0.tgz. The -r flag sets the source registry the container images are pulled from, and -t sets the local target registry the images are pushed to.kosi pull --hub kubeops-vm kubeops/kubevirt:2.2.0_Beta0 -o kubevirt-2.2.0_Beta0.tgz -r <master-ip>:30002/kubeops -t 127.0.0.1:30002/kubeops
This step is only required for a single-node cluster.
If your cluster consists of only one node, you must remove the taint from the master node so that workloads can be scheduled on it.
Replace [NODENAME] with the name of you master node.
kubectl taint nodes [NODENAME] node-role.kubernetes.io/control-plane:NoSchedule-
Before installing the package, create a YAML configuration file. Save the following content in the same directory as the package file and name the file kubevirt-values.yaml.
Replace
[your username]and[your harbor cli secret]with your KOSI username and your Harbor CLI secret.
The harbor-values must be adapted to your harbor instance.
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true
packages:
- name: kubevirt-manager
enabled: true
values:
standard:
namespace: kubevirt
hostname: kubeops-vms.local
nodeport: 30733
ingressName: nginx
advanced: {}
- name: kubevirt
enabled: true
hub: kubeops-vm
values:
standard:
namespace: kubevirt
imagePullSecretCred:
registry: 127.0.0.1:30002/kubeops
user: [your username]
password: [your harbor cli secret] # must be CLI Token from harbor
kubevirtobject:
spec:
configuration:
developerConfiguration:
logVerbosity:
virtLauncher: 2
virtHandler: 3
virtController: 4
virtAPI: 5
virtOperator: 6
featureGates:
- ExpandDisks
- CPUManager
- ExperimentalIgnitionSupport
- HypervStrictCheck
- Sidecar
- HostDevices
- Snapshot
- VMExport
- HotplugVolumes
- HostDisk
- ExperimentalVirtiofsSupport
- DownwardMetricsFeatureGate
- Root
- ClusterProfiler
- WorkloadEncryptionSEV
- VSOCK
- DisableCustomSELinuxPolicy
- KubevirtSeccompProfile
- DisableMDEVConfiguration
- PersistentReservation
- VMPersistentState
- MultiArchitecture
- AutoResourceLimitsGate
- AlignCPUs
- VolumesUpdateStrategy
- VolumeMigration
- InstancetypeReferencePolicy
useEmulation: false
cdiobject:
spec:
config:
insecureRegistries:
- registry.preprod.kubeops.net
scratchSpaceStorageClass: rook-cephfs # Optional; Default: rook-cephfs
- name: harbor
enabled: false
values:
standard:
namespace: harbor # optional, default is harbor
ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
harborpass: "password" # required: set password for harbor access
databasePassword: "Postgres_Password" # required: set password for database access
redisPassword: "Redis_Password" # required: set password for redis access
externalURL: http://10.2.10.11:30002 # required, the ip address and port, from which harbor is accessable outside of the cluster
nodePort: 30002 # required
hostname: harbor.local # required
harborPersistence:
persistentVolumeClaim:
registry:
size: 40Gi # optional, default is 40Gi
storageClass: "rook-cephfs" #optional, default is rook-cephfs
jobservice:
jobLog:
size: 1Gi # optional, default is 1Gi
storageClass: "rook-cephfs" #optional, default is rook-cephfs
database:
size: 1Gi # optional, default is 1Gi
storageClass: "rook-cephfs" #optional, default is rook-cephfs
redis:
size: 1Gi # optional, default is 1Gi
storageClass: "rook-cephfs" #optional, default is rook-cephfs
trivy:
size: 5Gi # optional, default is 5Gi
storageClass: "rook-cephfs" #optional, default is rook-cephfs
advanced: # All values from https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration are overwritable
kosi install -p kubevirt-2.2.0_Beta0.tgz -f kubevirt-values.yaml
kubevirt namespace.kubectl get pods -n kubevirt
The installation is complete when the status of all pods show the status “Running”. For example:
ImagePullBackOff error, recreate the image pull secret
the following example, the kubevirt namespace is used. If KubeVirt is deployed in a different namespace, replace the value passed to the -n parameter.
kubectl -n kubevirt create secret docker-registry kubeops-vm --docker-server=127.0.0.1:30002 --docker-username=admin --docker-password=<admin password - see harborpass value in kubeopsctl.yaml>
kosi pull --hub kubeops kubeops/kubevirt-manager:2.2.0_Beta0 -o kubevirt-manager-2.2.0_Beta0.tgz -r master-ip:30002/kubeops -t 127.0.0.1:30002/kubeops
kosi install -p kubevirt-manager-2.2.0_Beta0.tgz -f kubevirt-values.yaml
After the installation is complete, you can start creating and deploying a custom virtual machine. For more information, see How to create and deploy a Custom Virtual Machine using KubeOps.
Creating custom virtual machines on top of Kubernetes is a powerful way to unify containerized and traditional workloads. In this introductory guide, you’ll learn how to set up your KubeOps cluster to deploy a Cirros-based VM using KubeVirt, from checking off a few key prerequisites to preparing and applying a concise YAML configuration file.
Whether you’re a seasoned Kubernetes user looking to expand your virtualization capabilities or just getting started with hybrid infrastructure, these steps will walk you through everything from generating the custom VM file to verifying its running status and accessing it over SSH. By following this process, you’ll have a seamlessly integrated virtual machine environment in no time - ready to support your application needs.
Make sure you have prepared the following points:
For example:
To follow this guide, use the following values to configure the custom virtual machine. If you wish, you can adjust the values yourself. Make sure that the adjusted values are applied accordingly everywhere.
| Filename (VM configuration file) | custom-vm.yaml |
| Virtual Machine Label | kubevirt.io/vm: customvm |
| Virtual Machine Name | customvm |
| Virtual Machine Namespace | kubevirt |
| Virtual Machine Memory | 64M |
For users to understand the most basic functionality of KubeVirt, one must know about the following fundemental concepts:
KubeVirt offers different methods to engage with the virtual machines through specialized custom resources (CR). On the lowest level is the VirtualMachineInstance (VMI) which represents the stateless instance of a virtual machine. These VMIs can be managed by higher level resources, such as VirtualMachine (VM), which realizes a stateful VMI that can be stopped and started while maintaining its data and state.
The manifests for these resources contain a multitude of parameters that surpass the needs of most users. Consequently this guide will not explain all parameters.
The following steps will show you, how to manually create and access a virtual machine, using Kubernetes with a running KubeVirt instance.
To set up a virtual machine you need to create a configuration file. The configuration file contains information, such as name of the vm as well as its namespace, used resources (e.g. RAM) and states (e.g. running, halted). Use the file contents below and make desired changes.
Use the following content to create a basic vm configuration file custom-vm.yaml.
This file needs to be adapted at
spec.spec.volumes[0].containerDisk.image. You need to insert your local harbor deployments’ port.
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
kubevirt.io/vm: example-vm-cirros
name: example-vm
namespace: kubevirt
spec:
runStrategy: Always
template:
metadata:
labels:
kubevirt.io/vm: example-vm-cirros
spec:
domain:
devices:
disks:
- disk:
bus: virtio
name: containerdisk
- disk:
bus: virtio
name: cloudinitdisk
machine:
type: ""
resources:
requests:
memory: 64M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: "127.0.0.1:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0"
imagePullSecret: kubeops-vm
imagePullPolicy: Always
- cloudInitNoCloud:
userDataBase64: IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK
name: cloudinitdisk
The image, from which the virtual machine is created, can be found in the first section of spec.templates.spec.volumes of the manifest. It is used in the form of a ContainerDisk, meaning the image is part of a container image, within a remote repository.
This manifest will use a preconfigured image from the repository kubevirt/cirros-container-disk-demo:latest with the following credentials:
| User (cirros image) | cirros |
| Password (cirros image) | gocubsgo |
KubeVirt supports the ability to assign a startup script to a virtual machine. The script is executed automatically when the VM initializes. There are multiple datasources supported, which will inject those scripts through the use of ephemeral disks.
One of those datasources is called cloudInitNoCloud, which is mentioned here as the second volume in spec.templates.spec.volumes. VMs with the cloud-init package installed will detect the ephemeral disk and execute custom userdata scripts at boot.
podman pull registry.kubeops.net/kubeops/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0
In the following command you need to insert your master1 ip address into the placeholder
podman tag registry.kubeops.net/kubeops/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0 [your master1 ip]:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0
In the following command you need to insert your master1 ip address into the placeholder
podman push [your master1 ip]:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0 --tls-verify=false
Deploy the vm using the kubectl command and the custom-vm.yaml file.
kubectl apply -f custom-vm.yaml
Check the state of the vm.
kubectl get vmi -A
This command lists all virtual machine instances. Since we only have one VM installed, the output looks something like this.
NAMESPACE NAME AGE PHASE IP NODENAME READY
kubevirt customvm 22h Running 192.168.16.48 stackedadmin True
To connect to your newly created VM follow the documentation on how to access a Virtual Machine in a KubeOps Cluster
To know how to create a ContainerDisk from a local image, see Create a ContainerDisk from a local image guide
For more detailed explanation for virtual machine configuration files , see Detailed Explanation for Configuration File for Custom VMs.
Before proceeding, ensure that the following prerequisites are met:
For example:
In the example shown, the custom VM is accessed via the master node**, Worker Node 1**, or Worker Node 2.
A KubeOps virtual machine is not exposed by default and therefore cannot be accessed remotely by the admin node.
sudo -i
kubectl get vmi command.kubectl get vmi -A
This command lists all virtual machine instances. Choose the IP of the desired virtual machine.
NAMESPACE NAME AGE PHASE IP NODENAME READY
kubevirt [CUSTOMVM-1] 22h Running [192.168.16.XXX] stackedadmin True
kubevirt [CUSTOMVM-2] 22h Running [192.168.16.YYY] stackedadmin True
ssh). Use the IP address from the previous step. Use the credentials of the image.
Replace [USERNAME] and [IP-ADDRESS] accordingly.ssh [USERNAME]@[IP-ADDRESS]
The following YAML defines a Kubernetes Service that forwards the SSH port of your VM to the spec.ports.nodePort value.
The Service connects to your VM by using the selector, so make sure that the given label matches the one of your VM. Store it in a local YAML file serviceforwarding.yaml
apiVersion: v1
kind: Service
metadata:
name: vm-ssh-forwarding
namespace: kubevirt
spec:
type: NodePort
ports:
- port: 22
nodePort: 30226
targetPort: 22
protocol: TCP
selector:
kubevirt.io/vm: customvm # Binds directly to the underlying VM pod
Apply the Service with the following command:
kubectl apply -f serviceforwarding.yaml
To verify that the Service is connected to your VM, check the Endpoints resource in your Kubernetes cluster. The output should look similar to this:
NAME ENDPOINTS AGE
vm-ssh-forwarding 10.244.138.61:22 5d23h
You can access the custom VM remotely from any host that has network access to the cluster machines, for example by using ssh.
kubectl get vmi command.kubectl get vmi -A
This command lists all virtual machine instances. Since we only have one VM installed, the output looks similar to this:
NAMESPACE NAME AGE PHASE IP NODENAME READY
kubevirt example-vm 22h Running 192.168.16.48 stackedadmin True
To connect to the example VM remotely using ssh, use the IP address of a cluster node because the Service type is NodePort. In this example, the master node IP address is used. Also use the credentials of the running cirros image.
ssh [vm user]@[your master1 ip] -p 30226 # this port must match the one defined in your service
KubeVirt supports multiple ways of using images to create virtual machines. One of the most convenient methods is using the ContainerDisk feature, which allows VM disks to be stored and distributed as container images through a registry.
To create a ContainerDisk, users can simply inject the image into a container in qcow2 or raw format. The container should be based on scratch and the images must be placed into the /disk directory. No other content is required.
The following Dockerfile builds a ContainerDisk, which you can store in a OCI registry such as harbor or the Docker Hub:
FROM kubevirt/container-disk-v1alpha
ADD <your Image>.qcow2 /disk
You can the build the OCI arteiact with the command:
podman build . -t
Note: The ContainerDisk must be readable by the user with the UID 107 (qemu).
If you want to access the virtual machine, it is recommended to use a SSH key in the userdata startup section of your YAML file:
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#cloud-config
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaK8L93bWxnyp test@test.com
How to Import a VM from vSphere and Deploy it in a Cluster using KubeVirt
This guide outlines the steps to export a virtual machine from vSphere, convert it to the qcow2 format, and deploy it inside a Kubernetes cluster using KubeVirt.
To follow these steps, ensure you have:
A VM in the vSphere environment.
A Kubernetes cluster with KubeVirt installed.
Access to a container registry to store the VM image.
In vSphere, locate the VM you want to export.
Ensure the VM is powered off before exporting.
Right-click the VM, navigate to Template, and select Export OVF Template.
Provide a name for the exported template and click OK.
The export process will generate three files: *.vmdk, *.nvram, and *.ovf.
Only the *.vmdk file is required for further processing.
KubeVirt requires disk images in qcow2 format. Use the qemu-img tool to convert the .vmdk file:
qemu-img convert -f vmdk -O qcow2 [input-file].vmdk [output-file].qcow2
Replace [input-file].vmdk with the name of your downloaded VMDK file and [output-file].qcow2 with your desired output name.
To use the converted disk inside KubeVirt, create a ContainerDisk by building a container image with the qcow2 file.
Create a Dockerfile
Create a Dockerfile with the following content:
FROM scratch
ADD --chown=107:107 [filename].qcow2 /disk/
Replace [filename].qcow2 with the actual filename of your converted disk image.
Build and Push the Image
Execute the following commands to build the image and push it to a container registry:
podman build -t [registry]/[namespace]/[vm-image-name]:[tag] -t localhost/[namespace]/[vm-image-name]:[tag] .
podman push [registry]/[namespace]/[vm-image-name]:[tag]
Replace the placeholders as follows:
[registry]: The URL of your container registry
[namespace]: Your project name
[vm-image-name]: A name for the VM image
[tag]: The version tag (e.g., 1.0.0)
Once the image is pushed, it can be used as a ContainerDisk in KubeVirt to deploy a virtual machine.
Create a secret for authentication:
kubectl create secret generic kubeops-vm-sec --from-literal=accessKeyId=kubeops --from-literal=secretKey=registry-token --namespace kubevirt-vms-import
kubectl -n kubevirt-vms-import create secret docker-registry kubeops-vm --docker-server=registry.kubeops.net --docker-username="<username>" --docker-password="<registry-token>"
kubectl patch serviceaccount default -n kubevirt-vms-import -p '{"imagePullSecrets": [{"name": "kubeops-vm"}]}'
Create a manifest file for the virtual machine:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
kubevirt.io/vm: sk2-master01
name: sk2-master01-vm
namespace: kubevirt-vms-import
spec:
runStrategy: Always
template:
metadata:
labels:
kubevirt.io/vm: sk2-master01
spec:
domain:
devices:
disks:
- name: datavolumedisk
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
resources:
requests:
memory: 4092M
volumes:
- name: datavolumedisk
dataVolume:
name: sk2-master01-dv
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
expire: False
ssh_pwauth: True
disable_root: false
dataVolumeTemplates:
- metadata:
name: sk2-master01-dv
spec:
storage:
resources:
requests:
storage: 68Gi
storageClassName: rook-cephfs
accessModes:
- ReadWriteMany
source:
registry:
url: "docker://registry.kubeops.net/kubeops-vm/kubeops/kubevirt/sk2_master01_export:1.0.0" # (change the url to the image you want to deploy)
secretRef: kubeops-vm-sec
kubectl apply -f filename.yaml
Run the following commands to check if the VM is deployed successfully:
kubectl get dv -n kubevirt-vms-import
kubectl get vmi -n kubevirt-vms-import
Ensure that the DataVolume is successfully downloaded and the VirtualMachineInstance is running.
This guide explains how to monitor KubeVirt Virtual Machine Instances (VMIs) using Grafana and Prometheus. By the end of this guide, you will have a Grafana dashboard that displays metrics related to KubeVirt VMIs.
A running multi-node Kubernetes cluster with KubeVirt installed
⚠️ Note: Single-node clusters are not supported due to component scheduling and storage limitations.
A functional storage provider (e.g., Rook-Ceph or any other CSI-compatible solution) must be installed and available
⚠️ Required for deploying both Prometheus and Grafana, as they rely on PersistentVolumeClaims (PVCs)
Prometheus deployed in the cluster
Grafana installed and configured to connect to Prometheus
KubeVirt exposes various metrics that Prometheus can scrape. These metrics are documented here: KubeVirt Metrics.
To check whether Prometheus is collecting KubeVirt metrics:
Open the Prometheus Dashboard
Search for KubeVirt-related metrics, for example kubevirt_vmi_memory_available_bytes.
If no metrics appear, ensure the KubeVirt servicemonitor has been added to Prometheus:
kubectl get servicemonitor -A | grep kubevirt
Verify that KubeVirt was installed correctly and that monitorNamespace and monitorAccount are configured correctly. How to install KubeOps Virtualization (kubevirt)
Reload Prometheus operator.
Reload KubeVirt operator.
KubeVirt provides a ready-to-use Grafana dashboard JSON file: KubeVirt Grafana Dashboard.
To import it into Grafana:
Dashboards > New > Import.Load.To add specific VMI-related metrics to the dashboard:
Click Edit on a panel.
Use PromQL queries to fetch desired VMI metrics, e.g.,
rate(kubevirt_vmi_cpu_usage_seconds_total[5m])kubevirt_vmi_memory_available_bytesrate(kubevirt_vmi_network_receive_bytes_total[5m])Save the dashboard.
Note: You can view query output in the Prometheus dashboard.
You have successfully set up Grafana to monitor KubeVirt VMIs using Prometheus. You can now track VMI performance and resource usage effectively.
This guide explains how to configure L3 networking for KubeVirt virtual machines (VMs) using Multus. With this setup, VMs can communicate across nodes through dedicated secondary networks, which improves network segmentation and deployment flexibility in Kubernetes environments.
Before you begin, ensure that the following requirements are met:
kubectl command-line tool configured to connect to your clusterNetwork Manager State (nmstate) is a declarative network configuration tool for Linux. It provides an API and CLI to manage network settings in a structured, state-driven manner.
In Kubernetes, nmstate is often used with NodeNetworkConfigurationPolicy (NNCP) to configure node networking dynamically, allowing for changes such as setting up VLANs, bridges, bonds, and other networking features.
First, create the nmstate-values.yaml:
# nmstate-values.yaml
nmstateValues:
namespace: nmstate
Deploy the nmstate operator and its CRDs using this file:
kosi install --hub kubeops kubeops/nmstate:2.0.1 -f nmstate-values.yaml
Create an nncp-values.yaml file to define network configurations dynamically, including bridges and routes:
# nncp-values.yaml
namespace: "nmstate"
nodes:
- nodeName: cluster1master1
nodeIP: 10.2.10.11
bridgeIP: 10.10.1.1 # bridge IP for cluster1master1.
bridgeSubnet: 10.10.1.0 # `/24` subnet for bridge
nodeInterface: ens192 # use `ip a` or `ifconfig` to identify interface
- nodeName: cluster1master2
nodeIP: 10.2.10.12
bridgeIP: 10.10.2.1
bridgeSubnet: 10.10.2.0
nodeInterface: ens192
- nodeName: cluster1worker1
nodeIP: 10.2.10.14
bridgeIP: 10.10.3.1
bridgeSubnet: 10.10.3.0
nodeInterface: ens192
- nodeName: cluster1worker2
nodeIP: 10.2.10.15
bridgeIP: 10.10.4.1
bridgeSubnet: 10.10.4.0
nodeInterface: ens192
Note:
If you’re adding a new node, assign it a unique
bridgeIPandbridgeSubnet, that follow the existing addressing pattern.For example, if your last node uses:
bridgeIP: 10.10.4.1
bridgeSubnet: 10.10.4.0/24
add the next node like this:
- nodeName: <new_node> nodeIP: <node_ip> bridgeIP: 10.10.5.1 bridgeSubnet: 10.10.5.0 nodeInterface: ens192
Use this values file to deploy the node network configuration policies on all nodes:
kosi install --hub kubeops kubeops/nncp:2.0.1 -f nncp-values.yaml
Run the following commands on the relevant nodes to allow traffic through the secondary bridge:
iptables -A FORWARD -i br-secondary -j ACCEPT
iptables -A FORWARD -o br-secondary -j ACCEPT
To confirm that the bridge exists and has the expected IP address:
ip a show br-secondary
List the bridge interfaces and verify that the expected interfaces are attached:
ip link show type bridge
Check if the expected routes are in place:
ip route show
Test connectivity to another node’s bridge IP:
ping -c 4 <another_node_bridgeIP>
For installation instructions, please refer to the KubeVirt Installation Guide
To set up L3 networking with Multus, each worker node requires a separate NetworkAttachmentDefinition (NAD) with a unique IP address and subnet matching the node’s network configuration. This allows isolated network environments for each pod, enabling multi-interface support on pods and communication between nodes in the cluster.
Before creating the NADs, define the IP ranges and subnets for each worker node to avoid conflicts and routing conflicts.
Example IP range mapping for worker nodes:
| Node | Subnet | Example IP for Pod |
|---|---|---|
| cluster1worker1 | 10.10.3.0/24 | 10.10.3.10 |
| cluster1worker2 | 10.10.4.0/24 | 10.10.4.10 |
This example shows how to configure a NetworkAttachmentDefinition (NAD) for worker node 1 (cluster1worker1), with a static IP range assigned to its pods:
# nad_cluster1worker1.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: secondary-network-node1 # Assign unique names to each node's NAD
annotations:
k8s.v1.cni.cncf.io/nodeSelector: '{"kubernetes.io/hostname": "cluster1worker1"}'
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "secondary-network",
"type": "bridge",
"bridge": "br-secondary",
"ipam": {
"type": "static",
"addresses": [
{
"address": "10.10.3.10/24", # use correct worker node's subnet and mention the appropriate IP for pod's network
"gateway": "10.10.3.1"
}
],
"routes": [
{
"dst": "10.10.4.0/24",
"gw": "10.10.3.1"
}
]
}
}
To deploy the NetworkAttachmentDefinition (NAD) objects for your nodes, apply the following Kubernetes manifest files:
kubectl apply -f nad_cluster1worker1.yaml
kubectl apply -f nad_cluster1worker2.yaml
To ensure that the NetworkAttachmentDefinition (NAD) objects have been successfully created, run the following command:
kubectl get net-attach-def -A
Once the NADs are created, you can deploy Virtual Machines (VMs) and attach them to the secondary networks. The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached.
Each VM must be connected to the correct network defined by the NetworkAttachmentDefinition (NAD) for its corresponding node. For example, if you have created a NAD for cluster1worker1 named secondary-network-node1, ensure that the VM manifest refers to this network in the annotations section and correctly links to the networkName under the multus configuration.
Ensure that the network name in the manifest matches the NAD definition.
Please create a secret for pulling image from registry.kubeops.net.
kubectl -n "l3-networking" create secret docker-registry kubeops-vm \
--docker-server=registry.kubeops.net \
--docker-username=<kubeops-user> \
--docker-password=<harbor_token>
The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached to a specific NAD:
# vm1.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: "ubuntu-vm"
namespace: "l3-networking"
annotations:
k8s.v1.cni.cncf.io/networks: secondary-network-node1 # Ensure this matches the NAD network name
spec:
nodeSelector:
kubernetes.io/hostname: cluster1worker1 # edit for scheduling node on other node
runStrategy: Always
template:
metadata:
labels:
kubevirt.io/size: small
spec:
domain:
cpu:
cores: 1
devices:
disks:
- name: containervolume
disk:
bus: virtio
- name: cloudinitvolume
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
- name: secondary-network
bridge: {}
resources:
requests:
memory: 2048M
networks:
- name: default
pod: {}
- name: secondary-network
multus:
networkName: secondary-network-node1 # Ensure this matches the NAD network name
volumes:
- name: containervolume
containerDisk:
image: registry.kubeops.net/kubeops-vm/kubeops/ubuntu-cloudimg-focal:2.0.1
imagePullSecret: kubeops-vm
imagePullPolicy: Always
- name: cloudinitvolume
cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
list: |
ubuntu:ubuntu
root:toor
expire: False
ssh_pwauth: True
disable_root: false
Once the NetworkAttachmentDefinition (NAD) objects have been created, you can deploy the Virtual Machines (VMs) and apply the manifests. The following commands apply the VM manifests to your Kubernetes cluster.
kubectl apply -f vm1.yaml
kubectl apply -f vm2.yaml
After the VMs start successfully, verify that the secondary network interfaces and routes are configured correctly.
First, check the IP address assigned to the pod (VM). This is crucial to ensure that the secondary network interface has been correctly assigned an IP.
kubectl exec -n <namespace> <pod-1> -- ip a
kubectl exec -n <namespace> <pod-1> -- sudo dhclient enp2s0
Verify the routing configuration on the VM to ensure that traffic can flow through the secondary network. Use the following command:
kubectl exec -n <namespace> <pod-1> -- ip r
This will display the routing table for the VM and confirm if the secondary network is correctly set up.
After ensuring the network settings are correct, test the communication between VMs by pinging the secondary network IP of another pod.
Use the ping command to check if the VMs can communicate over the secondary network. Replace <pod2_secondary_network_IP> with the actual IP address of the secondary network interface on the second VM.
kubectl exec -n <namespace> <pod-1> -- ping -c 2 <pod2_secondary_network_IP>
If the ping is successful, it confirms that the VMs are able to communicate over the secondary network.
Additional Notes:
Ensure that you use the correct namespace and pod name while executing the commands.
Make sure the secondary network interface is correctly configured on each VM, and the IP address and routes are properly assigned.
If communication fails, check for issues in the NetworkAttachmentDefinition (NAD) configuration or verify that the correct CNI plugins are running.
This guide describes how to configure Velero to back up and restore KubeVirt virtual machines in a Kubernetes cluster.
Velero stores backup data in an S3-compatible object storage backend. In this example, MinIO is used as the backup storage target.
Before you begin, ensure that the following prerequisites are met:
In following example MinIO is used.
Define the Velero configuration in your values file:
veleroValues:
namespace: "velero"
accessKeyId: "minio"
secretAccessKey: "minio123"
backupLocationConfig:
s3Url: "http://10.2.10.110:30090" # nodeport of minio
Enable Velero in the kubeopsctl.yaml file :
velero: true
Apply the configuration:
kubeopsctl apply -f kubeopsctl.yaml
The bucket for Velero ( default Velero ) in the s3 storage should be accessible for Velero, so ensure that the bucket exists and that Velero has permission to write to it.
In this example, the bucket is configured so that Velero has write access:
To create a backup of the namespace containing the KubeVirt virtual machine, run:
velero backup get
velero backup create kubeops-vm-backup --include-namespaces <Namespace of kubevirt vm>
The name of the backup must be unique.
To review the backup status and details, run:
velero backup get
velero backup describe kubeops-vm-backup
velero backup logs kubeops-vm-backup
Confirm that the backup completed successfully before proceeding.
To restore the backup, run:
velero restore get
velero restore create kubeops-vm-restore --from-backup kubeops-vm-backup
The restore name must be unique.
After the restore is complete, verify that the virtual machine resources have been recreated successfully in the target namespace.
velero restore get
velero restore describe kubeops-vm-restore
velero restore logs kubeops-vm-restore