KubeOpsDoku – license-required

Kubeopsctl-2.0: Joining a Node in a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

To increase performance or add additional resource capacity to your cluster, adding a node to the cluster is the correct process. This process with kubeopsctl is very easy.
You can use the following steps to join control-plane nodes or worker-nodes to a Kubernetes cluster.

Join Node Process:

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

ETCD Backup Recommendation

Before performing changes on the control planes, it is recommended to create an ETCD backup. Refer to the official Kubernetes documentation for details here

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull a specific Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-controlplaneXX
    iPAddress: 10.2.10.XXX
    type: controlplane
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone1
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone1.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6       # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true             # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-controlplaneXX   # -> has to be changed
    iPAddress: 10.2.10.XXX      # -> has to be changed
    type: controlplane
    kubeVersion: 1.31.6         # -> check with actual version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. If you are ready just start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

Example 1: Joining a Worker Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion x.xx.x you can pull other Kubernetes versions.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-workerXX
    iPAddress: 10.2.10.XX
    type: worker
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone2
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone2.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6        # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true              # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
  - name: demo-workerXX          # -> has to be changed
    iPAddress: 10.2.10.XX        # -> has to be changed
    type: worker
    kubeVersion: 1.31.6          # -> check with actual version     
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. Before joining the node, note the following:

Note

It is recommended if you join worker nodes to a cluster and rook-ceph is installed, to adjust your rook-ceph CRUSH map. Refer to the official rook-ceph documentation for details here. for updating the crush map, you should set in your enterprise-values.yaml:

...
packages:
- name: rook-ceph
  enabled: true
...

If you are ready, start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

If you need to apply the CRUSH map adjustment from above, use the following command instead:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Deleting a Node from a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Deleting a Node from a Kubernetes cluster

Note This guide requires a Compliance enterprise license.

In rare cases, it may be necessary to remove nodes from a Kubernetes cluster. This how-to guide explains the prerequisites and the key considerations to keep in mind before starting the node removal process.

You can use the following steps to delete nodes from a Kubernetes cluster.

Prerequisites

In order to run rook-ceph stable for a longer period your cluster needs at least 3 zones with each zone containing at least 1 worker-node
To check which mon and osd is running on the node you want to delete you can use the command kubectl get pods -n rook-ceph -o wide | grep <node-name> | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'. As an output you get the mon and the osd running on that node. If you don’t get an output, you don’t have to delete the ressource and can skip to the “delete the node”-section

Worker

Important: Due to rook-ceph, a worker node must not be removed without following the steps below. In this example, worker01 (zone1) is removed from the cluster. Worker01 contains osd.0 and mon-c.

Scale down the rook-ceph-operator deployment to 0

This prevents new MONs or OSDs from being created.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=0

Check which hosts and OSDs belong to each zone

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

ID   CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
 -1         0.21478  root default
 -9         0.04880      zone zone1
 -7         0.04880          host worker01                              # worker01 is being removed
  0    ssd  0.04880              osd.0          up   1.00000  1.00000   # osd.0 is being removed
-15         0.04880          host worker04
  3    ssd  0.04880              osd.3          up   1.00000  1.00000
-11         0.10739      zone zone2
 -3         0.05859          host worker02
  1    ssd  0.05859              osd.1          up   0.95001  1.00000
-13         0.05859      zone zone3
 -5         0.05859          host worker03
  2    ssd  0.05859              osd.2          up   0.95001  1.00000

From this output you can see that osd.0 is part of worker01.

Scale down the OSD deployment

kubectl scale deploy -n rook-ceph rook-ceph-osd-<x> --replicas=0
# Example: kubectl scale deploy -n rook-ceph rook-ceph-osd-0 --replicas=0

Remove the OSD via ceph-tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- bash
# show OSD tree
ceph osd tree
# mark OSD out
ceph osd out <x>
# Example: ceph osd out 0
ceph osd purge <x> --yes-i-really-mean-it
# Example: ceph osd purge 0 --yes-i-really-mean-it
ceph auth del osd.<x>
# adjust CRUSH map
ceph osd crush remove <nodename>
# exit from ceph-tools
exit
# show OSD tree (now without the deleted node)
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

Delete OSD and MON deployments

After checking the OSD and MON running on the node, delete the corresponding deployments.

kubectl delete deploy -n rook-ceph rook-ceph-osd-<x> rook-ceph-mon-<y>

Example

kubectl delete deploy -n rook-ceph rook-ceph-osd-0 rook-ceph-mon-c

Remove the deleted mon from the ceph tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm <y>
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

Example

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm c
# verfify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

This is the dump before executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b
2: [v2:192.168.196.110:3300/0,v1:192.168.196.110:6789/0] mon.c

This is the dump after executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b

Delete the node from the kubernetes cluster

Remove the node you want to delete from the cluster-values.yaml file.
After updating the file, apply the deletion with the following command:

kubeopsctl apply --delete -f cluster-values.yaml

Example

The cluster-values.yaml without node1 but with node4

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6      
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker04
    iPAddress: 10.2.10.214
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6  
  - name: worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

After updating the cluster-values.yaml, execute the delete command:

kubeopsctl apply --delete -f cluster-values.yaml

Scale the rook-ceph-operator deployment back to 1

This allows a new MON to be created automatically in zone2.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=1

Timing and health checks

The total duration depends on cluster size and node performance. Before proceeding, verify Ceph health and placement groups are clean.

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph pg stat

Typical duration ranges from 15 to 120 minutes.

If you want to rejoin the same node, reset it to a time prior to joining the cluster. Only this way you can be sure, that no leftovers from the deletion process remain!

Kubeopsctl-2.0: Upgrading a Kubernetes cluster

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

Upgrading a Kubernetes cluster is essential to maintain security, stability, and compatibility.Like Kubernetes itself, we adhere to the version skew policy and only allow upgrades between releases that differ by a single minor version. This ensures compatibility between components, reduces the risk of instability, and keeps the cluster in a supported and secure state.

For more information about the Version Skew Policy, see the official Kubernetes documentation:
Click here

Use the following steps to upgrade the Kubernetes version of a cluster.

Kubernetes Version Upgrade Process:

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull --kubernetesVersion <x.xx.x>

2. Change your target version inside the cluster-values

Note The actual version is the current Kubernetes version in your cluster. The target version in the nodes, is version we want to do the upgrade (e.g., see this yaml file)

Note To upgrade your cluster, the parameter changeCluster must be set to true (e.g., see this yaml file)

3. Start the upgrade

kubeopsctl apply -f cluster-values.yaml

Important It is important to start the upgrade with the initial controlplane node. It is the first node with which the cluster was created.

Example 1 - Upgrade all nodes in the cluster to a specific version

In this example, a cluster is upgraded from Kubernetes version v1.33.5 to v1.34.1.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.34.1 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.34.1

2. Update the target version in `cluster-values.yaml`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version in the kubeVersion field of each node.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.33.5     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true           # -> important! Needs to be set for an upgrade
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.34.1       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.34.1       # -> target version

2. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully.
When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

Example 2 - Tranche upgrade of zones to a specific version

In this example, the cluster is upgraded in tranches. The order is:

zone1 (because it contains the initial control plane node),
then zone3,
and finally zone2.

Important It is important to start your tranche upgrade with zone1, because zone1 contains the initial control plane node.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.33.5 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.33.5

2. Adjust your cluster-values in `zone1`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version only for the nodes in zone1. In the snippet below, only zone1 is configured with the target version.

Note Pay close attention to the format and syntax in cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

3. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

4. Adjust your `cluster-values.yaml` in `zone2`

Next, change the target version of zone2:

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

5. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

6. Adjust your `cluster-values.yaml` `in zone3`

Finally, change the target version of zone 3.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.33.5       # -> target version

7. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Kubeopsctl-2.0: Installing KubeOps Compliance applications

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be individually configured as needed.

By separating the cluster values from the application values, the application values can be modified independently and installed at a later stage, providing greater flexibility and maintainability.

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

⚠ Warning To ensure proper DNS resolution, all components of the logging stack — Filebeat, Logstash, OpenSearch, and OpenSearch Dashboards—must be deployed within the same Kubernetes namespace (or network domain).

Example 1: Installing Applications in a non-airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. For more information about available packages as well as parameters for each package check here.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2      
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and install the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the installation process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Installing Applications in an airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                        # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2      
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsctl-2.0: Updating KubeOps Compliance applications

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be configured individually as needed.

By separating cluster values from application values, application values can be modified independently and installed later, providing greater flexibility and maintainability.

kubeopsctl automatically detects whether an application is already deployed and updates it accordingly. de]

Important If you have made changes to your deployments, reapply them afterward or secure them using the advancedValues configuration.

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

Updated KubeOpsctl

If you have an older kubeopsctl version installed, update it before starting with updating Compliance appliactions.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

Example 1: Updating Applications in a non-airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application update process
Important for only update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and update the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the update process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Updating Applications in an airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Update kubeopsctl

If you have an older kubeopsctl version installed, update it using the following commands.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application update process
Important for only the update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                          # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Kubeopsvm: How to install KubeOps Virtualization (kubevirt)

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance Enterprise license.

KubeVirt provides a powerful way to run virtual machines on top of Kubernetes, creating a unified platform for both containerized and traditional workloads. This guide walks through the essential steps for getting KubeVirt up and running using the KOSI package manager - from pulling and preparing the KubeVirt package, to creating a custom configuration file with your credentials, and finally verifying a successful installation in the kubevirt-namespace. Once these steps are complete, you’ll be ready to deploy and manage virtual machines alongside your existing container infrastructure - all within a single Kubernetes cluster.

KubeVirt is a specialized package that extends the capabilities of the KubeOps platform to support machine virtualization.

Requirements

To follow the next steps, make sure you have your credentials ready from your KubeOps.net account.

Make sure you have prepared the following points:

A KubeOps cluster with one master node.
Access to the master node with root privileges.

Step 1 - Download KubeOpsVM

Log in to the master node remotely. For example, using ssh.
If necessary, switch to your cluster user. If you forgot your cluster user, check your cluster-values.yaml.

su <your linux username>

kosi login -u [USERNAME]

Search for available software packages by using the KOSI package manager. In this case, we restrict the search to KubeOps virtualization using kubeops-vm.

kosi search --hub kubeops-vm

Log in to the local container registry on the master node using podman. Replace <master-ip> with the IP address of your master node.

podman login <master-ip>:30002 -u admin -p <harbor admin password> --tls-verify=false

The package is named kubeops/kubevirt:2.2.0_Beta0 and lies on the hub kubeops-vm. Download the package using the kosi pull command and save it with the desired file name. In this example, the file is saved as kubevirt-2.2.0_Beta0.tgz. The -r flag sets the source registry the container images are pulled from, and -t sets the local target registry the images are pushed to.

kosi pull --hub kubeops-vm kubeops/kubevirt:2.2.0_Beta0 -o kubevirt-2.2.0_Beta0.tgz -r <master-ip>:30002/kubeops -t 127.0.0.1:30002/kubeops

Step 2 - Remove the Taint of the Master Node (only necessary on single node cluster)

This step is only required for a single-node cluster.

If your cluster consists of only one node, you must remove the taint from the master node so that workloads can be scheduled on it.

Replace [NODENAME] with the name of you master node.

kubectl taint nodes [NODENAME] node-role.kubernetes.io/control-plane:NoSchedule-

Step 3 - Create Kubevirt Configuration File

Before installing the package, create a YAML configuration file. Save the following content in the same directory as the package file and name the file kubevirt-values.yaml.

Replace [your username] and [your harbor cli secret] with your KOSI username and your Harbor CLI secret.

The harbor-values must be adapted to your harbor instance.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1

deleteNs: false
localRegistry: true

packages:
  - name: kubevirt-manager
    enabled: true
    values:
      standard:
        namespace: kubevirt
        hostname: kubeops-vms.local
        nodeport: 30733
        ingressName: nginx
      advanced: {}

  - name: kubevirt
    enabled: true
    hub: kubeops-vm
    values:
      standard:
        namespace: kubevirt

        imagePullSecretCred:
          registry: 127.0.0.1:30002/kubeops
          user: [your username]
          password: [your harbor cli secret] # must be CLI Token from harbor

        kubevirtobject:
          spec:
            configuration:
              developerConfiguration:
                logVerbosity:
                  virtLauncher: 2
                  virtHandler: 3
                  virtController: 4
                  virtAPI: 5
                  virtOperator: 6

                featureGates:
                  - ExpandDisks
                  - CPUManager
                  - ExperimentalIgnitionSupport
                  - HypervStrictCheck
                  - Sidecar
                  - HostDevices
                  - Snapshot
                  - VMExport
                  - HotplugVolumes
                  - HostDisk
                  - ExperimentalVirtiofsSupport
                  - DownwardMetricsFeatureGate
                  - Root
                  - ClusterProfiler
                  - WorkloadEncryptionSEV
                  - VSOCK
                  - DisableCustomSELinuxPolicy
                  - KubevirtSeccompProfile
                  - DisableMDEVConfiguration
                  - PersistentReservation
                  - VMPersistentState
                  - MultiArchitecture
                  - AutoResourceLimitsGate
                  - AlignCPUs
                  - VolumesUpdateStrategy
                  - VolumeMigration
                  - InstancetypeReferencePolicy

                useEmulation: false

        cdiobject:
          spec:
            config:
              insecureRegistries:
                - registry.preprod.kubeops.net
              scratchSpaceStorageClass: rook-cephfs # Optional; Default: rook-cephfs
  - name: harbor
    enabled: false
    values:
      standard:
        namespace: harbor # optional, default is harbor
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        harborpass: "password" # required: set password for harbor access
        databasePassword: "Postgres_Password" # required: set password for database access
        redisPassword: "Redis_Password" # required: set password for redis access
        externalURL: http://10.2.10.11:30002 # required, the ip address and port, from which harbor is accessable outside of the cluster
        nodePort: 30002 # required
        hostname: harbor.local # required
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi # optional, default is 40Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            jobservice:
              jobLog:
                size: 1Gi # optional, default is 1Gi
                storageClass: "rook-cephfs" #optional, default is rook-cephfs
            database:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            redis:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            trivy: 
              size: 5Gi # optional, default is 5Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
      advanced: #  All values from https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration are overwritable

Step 4 - Install KubeOps-VM

Install the KubeVirt package using the prepared configuration file you created.

kosi install -p kubevirt-2.2.0_Beta0.tgz -f kubevirt-values.yaml

Verify the installation by checking the pods in the kubevirt namespace.

kubectl get pods -n kubevirt

The installation is complete when the status of all pods show the status “Running”. For example:

Troubleshoot If you receive an ImagePullBackOff error, recreate the image pull secret

the following example, the kubevirt namespace is used. If KubeVirt is deployed in a different namespace, replace the value passed to the -n parameter.

kubectl -n kubevirt create secret docker-registry kubeops-vm --docker-server=127.0.0.1:30002 --docker-username=admin --docker-password=<admin password - see harborpass value in kubeopsctl.yaml>

Step 5 - Pull and Install KubeVirt Manager Package

Pull the KubeVirt Manager package.

kosi pull --hub kubeops kubeops/kubevirt-manager:2.2.0_Beta0 -o kubevirt-manager-2.2.0_Beta0.tgz -r master-ip:30002/kubeops -t 127.0.0.1:30002/kubeops

Install the KubeVirt Manager package.

kosi install -p kubevirt-manager-2.2.0_Beta0.tgz -f kubevirt-values.yaml

Next Steps

After the installation is complete, you can start creating and deploying a custom virtual machine. For more information, see How to create and deploy a Custom Virtual Machine using KubeOps.

Kubeopsvm: How to create and deploy a Custom Virtual Machine using KubeOps

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

Creating custom virtual machines on top of Kubernetes is a powerful way to unify containerized and traditional workloads. In this introductory guide, you’ll learn how to set up your KubeOps cluster to deploy a Cirros-based VM using KubeVirt, from checking off a few key prerequisites to preparing and applying a concise YAML configuration file.

Whether you’re a seasoned Kubernetes user looking to expand your virtualization capabilities or just getting started with hybrid infrastructure, these steps will walk you through everything from generating the custom VM file to verifying its running status and accessing it over SSH. By following this process, you’ll have a seamlessly integrated virtual machine environment in no time - ready to support your application needs.

Requirements

Make sure you have prepared the following points:

A KubeOps cluster with at least one master node and root privileges.
A running kubeVirt instance. (see How to install KubeOps Virtualisation (KubeVirt)).
Enough resources to support a virtual machine

For example:

To follow this guide, use the following values to configure the custom virtual machine. If you wish, you can adjust the values yourself. Make sure that the adjusted values are applied accordingly everywhere.


Filename (VM configuration file)	`custom-vm.yaml`
Virtual Machine Label	`kubevirt.io/vm: customvm`
Virtual Machine Name	`customvm`
Virtual Machine Namespace	`kubevirt`
Virtual Machine Memory	`64M`

Additional Information

For users to understand the most basic functionality of KubeVirt, one must know about the following fundemental concepts:

KubeVirt offers different methods to engage with the virtual machines through specialized custom resources (CR). On the lowest level is the VirtualMachineInstance (VMI) which represents the stateless instance of a virtual machine. These VMIs can be managed by higher level resources, such as VirtualMachine (VM), which realizes a stateful VMI that can be stopped and started while maintaining its data and state.

The manifests for these resources contain a multitude of parameters that surpass the needs of most users. Consequently this guide will not explain all parameters.

How to create and deploy a Custom Virtual Machine

The following steps will show you, how to manually create and access a virtual machine, using Kubernetes with a running KubeVirt instance.

Step 1 - Create a Configuration File for the Virtual Machine

To set up a virtual machine you need to create a configuration file. The configuration file contains information, such as name of the vm as well as its namespace, used resources (e.g. RAM) and states (e.g. running, halted). Use the file contents below and make desired changes.

Use the following content to create a basic vm configuration file custom-vm.yaml.

This file needs to be adapted at spec.spec.volumes[0].containerDisk.image. You need to insert your local harbor deployments’ port.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: example-vm-cirros
  name: example-vm
  namespace: kubevirt
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: example-vm-cirros
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 64M
      terminationGracePeriodSeconds: 0
      volumes:
      - name: containerdisk
        containerDisk:
          image: "127.0.0.1:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0"
          imagePullSecret: kubeops-vm
          imagePullPolicy: Always
      - cloudInitNoCloud:
          userDataBase64: IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK
        name: cloudinitdisk

The image, from which the virtual machine is created, can be found in the first section of spec.templates.spec.volumes of the manifest. It is used in the form of a ContainerDisk, meaning the image is part of a container image, within a remote repository.

This manifest will use a preconfigured image from the repository kubevirt/cirros-container-disk-demo:latest with the following credentials:


User (cirros image)	`cirros`
Password (cirros image)	`gocubsgo`

KubeVirt supports the ability to assign a startup script to a virtual machine. The script is executed automatically when the VM initializes. There are multiple datasources supported, which will inject those scripts through the use of ephemeral disks.

One of those datasources is called cloudInitNoCloud, which is mentioned here as the second volume in spec.templates.spec.volumes. VMs with the cloud-init package installed will detect the ephemeral disk and execute custom userdata scripts at boot.

Step 2 - Transfer the VM Image to your Local Registry

Pull the image from the kubeops registry

 podman pull registry.kubeops.net/kubeops/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0

Tag the image for the local registry

In the following command you need to insert your master1 ip address into the placeholder

podman tag registry.kubeops.net/kubeops/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0 [your master1 ip]:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0

Push the image to the local registry

In the following command you need to insert your master1 ip address into the placeholder

podman push [your master1 ip]:5001/kubeops/kubevirt/cirros-container-disk-demo:v1.4.0-patch0 --tls-verify=false

Step 3 - Deploy the Virtual Machine

Deploy the vm using the kubectl command and the custom-vm.yaml file.

kubectl apply -f custom-vm.yaml

Check the state of the vm.

kubectl get vmi -A

This command lists all virtual machine instances. Since we only have one VM installed, the output looks something like this.

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY
kubevirt    customvm   22h   Running   192.168.16.48   stackedadmin   True

Notice We highly recommend documenting the basic settings of your virtual machines (especially the configurations). This includes in particular: the used configuration file (yaml), the label of the vm, the **name of the vm **and the namespace of the vm. You can also document the image used. Documentation makes it easier to maintain and expand the virtual machines.

Step 4 - Access the Custom VM

To connect to your newly created VM follow the documentation on how to access a Virtual Machine in a KubeOps Cluster

References

To know how to create a ContainerDisk from a local image, see Create a ContainerDisk from a local image guide
For more detailed explanation for virtual machine configuration files , see Detailed Explanation for Configuration File for Custom VMs.

Kubeopsvm: How to access a virtual machine in a KubeOps Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Note This guide requires a Compliance enterprise license.

Before proceeding, ensure that the following prerequisites are met:

A KubeOps cluster with at least one master node.
A running virtual machine in said cluster.
The credentials of the running image within the virtual machine (username and password).

For example:

In the example shown, the custom VM is accessed via the master node**, Worker Node 1**, or Worker Node 2.

A KubeOps virtual machine is not exposed by default and therefore cannot be accessed remotely by the admin node.

Step 1 - Identify your Virtual Machine

Log in to the master or worker machine.
Most kubeops software requires root privileges. Switch to the root user.

sudo -i

Identify your virtual machine by running the kubectl get vmi command.

kubectl get vmi -A

This command lists all virtual machine instances. Choose the IP of the desired virtual machine.

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY

kubevirt    [CUSTOMVM-1]   22h   Running   [192.168.16.XXX]   stackedadmin   True
kubevirt    [CUSTOMVM-2]   22h   Running   [192.168.16.YYY]   stackedadmin   True

Step 2 - Access the Virtual Machine

Log in to the example VM remotely (e.g. using ssh). Use the IP address from the previous step. Use the credentials of the image. Replace [USERNAME] and [IP-ADDRESS] accordingly.

ssh [USERNAME]@[IP-ADDRESS]

Step 1 - Create a Service forwarding the SSH port of the VM

The following YAML defines a Kubernetes Service that forwards the SSH port of your VM to the spec.ports.nodePort value. The Service connects to your VM by using the selector, so make sure that the given label matches the one of your VM. Store it in a local YAML file serviceforwarding.yaml

apiVersion: v1
kind: Service
metadata:
  name: vm-ssh-forwarding
  namespace: kubevirt
spec:
  type: NodePort
  ports:
    - port: 22
      nodePort: 30226
      targetPort: 22
      protocol: TCP
  selector:
    kubevirt.io/vm: customvm # Binds directly to the underlying VM pod

Apply the Service with the following command:

 kubectl apply -f serviceforwarding.yaml

To verify that the Service is connected to your VM, check the Endpoints resource in your Kubernetes cluster. The output should look similar to this:

NAME                ENDPOINTS          AGE
vm-ssh-forwarding   10.244.138.61:22   5d23h

Step 2 - Access the Custom VM

You can access the custom VM remotely from any host that has network access to the cluster machines, for example by using ssh.

Log in to the machine.
Identify the IP address of your custom VM by using the kubectl get vmi command.

kubectl get vmi -A

This command lists all virtual machine instances. Since we only have one VM installed, the output looks similar to this:

NAMESPACE   NAME         AGE   PHASE     IP              NODENAME       READY

kubevirt    example-vm   22h   Running   192.168.16.48   stackedadmin   True

To connect to the example VM remotely using ssh, use the IP address of a cluster node because the Service type is NodePort. In this example, the master node IP address is used. Also use the credentials of the running cirros image.

ssh [vm user]@[your master1 ip] -p 30226 # this port must match the one defined in your service

Kubeopsvm: How to create a ContainerDisk from a local image

Mon, 01 Jan 0001 00:00:00 +0000

Note This feature requires a Compliance enterprise license.

KubeVirt supports multiple ways of using images to create virtual machines. One of the most convenient methods is using the ContainerDisk feature, which allows VM disks to be stored and distributed as container images through a registry.

To create a ContainerDisk, users can simply inject the image into a container in qcow2 or raw format. The container should be based on scratch and the images must be placed into the /disk directory. No other content is required.

The following Dockerfile builds a ContainerDisk, which you can store in a OCI registry such as harbor or the Docker Hub:

FROM kubevirt/container-disk-v1alpha
ADD <your Image>.qcow2 /disk

You can the build the OCI arteiact with the command:

podman build . -t

Note: The ContainerDisk must be readable by the user with the UID 107 (qemu).

If you want to access the virtual machine, it is recommended to use a SSH key in the userdata startup section of your YAML file:

      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            ssh_authorized_keys:
              - ssh-rsa AAAAB3NzaK8L93bWxnyp test@test.com

Kubeopsvm: How to import a VM from vSphere and deploy it in a Cluster using KubeVirt

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

How to Import a VM from vSphere and Deploy it in a Cluster using KubeVirt

This guide outlines the steps to export a virtual machine from vSphere, convert it to the qcow2 format, and deploy it inside a Kubernetes cluster using KubeVirt.

Requirements

To follow these steps, ensure you have:

A VM in the vSphere environment.

A Kubernetes cluster with KubeVirt installed.

Access to a container registry to store the VM image.

Step 1 - Export OVF Template from vSphere

In vSphere, locate the VM you want to export.

Ensure the VM is powered off before exporting.

Right-click the VM, navigate to Template, and select Export OVF Template.

Provide a name for the exported template and click OK.

The export process will generate three files: *.vmdk, *.nvram, and *.ovf.

Only the *.vmdk file is required for further processing.

Step 2 - Convert the VMDK File to QCOW2 Format

KubeVirt requires disk images in qcow2 format. Use the qemu-img tool to convert the .vmdk file:

qemu-img convert -f vmdk -O qcow2 [input-file].vmdk [output-file].qcow2

Replace [input-file].vmdk with the name of your downloaded VMDK file and [output-file].qcow2 with your desired output name.

Step 3 - Create a Container Image from the QCOW2 File

To use the converted disk inside KubeVirt, create a ContainerDisk by building a container image with the qcow2 file.

Create a Dockerfile

Create a Dockerfile with the following content:

FROM scratch 

ADD --chown=107:107 [filename].qcow2 /disk/

Replace [filename].qcow2 with the actual filename of your converted disk image.

Build and Push the Image

Execute the following commands to build the image and push it to a container registry:

podman build -t [registry]/[namespace]/[vm-image-name]:[tag] -t localhost/[namespace]/[vm-image-name]:[tag] .

podman push [registry]/[namespace]/[vm-image-name]:[tag]

Replace the placeholders as follows:

[registry]: The URL of your container registry

[namespace]: Your project name

[vm-image-name]: A name for the VM image

[tag]: The version tag (e.g., 1.0.0)

Once the image is pushed, it can be used as a ContainerDisk in KubeVirt to deploy a virtual machine.

Step 4 - Deploy the VM using KubeVirt

Create a secret for authentication:


kubectl create secret generic kubeops-vm-sec --from-literal=accessKeyId=kubeops --from-literal=secretKey=registry-token --namespace kubevirt-vms-import

kubectl -n kubevirt-vms-import create secret docker-registry kubeops-vm --docker-server=registry.kubeops.net --docker-username="<username>" --docker-password="<registry-token>"

 kubectl patch serviceaccount default -n kubevirt-vms-import -p '{"imagePullSecrets": [{"name": "kubeops-vm"}]}'

Create a manifest file for the virtual machine:


apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: sk2-master01
  name: sk2-master01-vm
  namespace: kubevirt-vms-import
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: sk2-master01
    spec:
      domain:
        devices:
          disks:
          - name: datavolumedisk
            disk:
              bus: virtio
          - name: cloudinitvolume
            disk:
              bus: virtio
        resources:
          requests:
            memory: 4092M
      volumes:
      - name: datavolumedisk
        dataVolume:
          name: sk2-master01-dv
      - name: cloudinitvolume
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            chpasswd:
              list: |
              expire: False
            ssh_pwauth: True
            disable_root: false            
  dataVolumeTemplates:
  - metadata:
      name: sk2-master01-dv
    spec:
      storage:
        resources:
          requests:
            storage: 68Gi
        storageClassName: rook-cephfs
        accessModes:
          - ReadWriteMany
      source:
        registry:
          url: "docker://registry.kubeops.net/kubeops-vm/kubeops/kubevirt/sk2_master01_export:1.0.0" # (change the url to the image you want to deploy)
          secretRef: kubeops-vm-sec

kubectl apply -f filename.yaml

Step 5 - Verify Deployment

Run the following commands to check if the VM is deployed successfully:

kubectl get dv -n kubevirt-vms-import

kubectl get vmi -n kubevirt-vms-import

Ensure that the DataVolume is successfully downloaded and the VirtualMachineInstance is running.

Kubeopsvm: How to Monitor KubeVirt in Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

Monitor KubeVirt VMIs with Grafana and Prometheus

This guide explains how to monitor KubeVirt Virtual Machine Instances (VMIs) using Grafana and Prometheus. By the end of this guide, you will have a Grafana dashboard that displays metrics related to KubeVirt VMIs.

Prerequisites

A running multi-node Kubernetes cluster with KubeVirt installed

⚠️ Note: Single-node clusters are not supported due to component scheduling and storage limitations.
A functional storage provider (e.g., Rook-Ceph or any other CSI-compatible solution) must be installed and available

⚠️ Required for deploying both Prometheus and Grafana, as they rely on PersistentVolumeClaims (PVCs)
Prometheus deployed in the cluster
Grafana installed and configured to connect to Prometheus

Step 1: Ensure Prometheus is Collecting KubeVirt Metrics

KubeVirt exposes various metrics that Prometheus can scrape. These metrics are documented here: KubeVirt Metrics.

To check whether Prometheus is collecting KubeVirt metrics:

Open the Prometheus Dashboard
Search for KubeVirt-related metrics, for example kubevirt_vmi_memory_available_bytes.
If no metrics appear, ensure the KubeVirt servicemonitor has been added to Prometheus:
```
 kubectl get servicemonitor -A | grep kubevirt
```
Verify that KubeVirt was installed correctly and that monitorNamespace and monitorAccount are configured correctly. How to install KubeOps Virtualization (kubevirt)
Reload Prometheus operator.
Reload KubeVirt operator.

Step 2: Import the KubeVirt Grafana Dashboard

KubeVirt provides a ready-to-use Grafana dashboard JSON file: KubeVirt Grafana Dashboard.

To import it into Grafana:

Open the Grafana Dashboard.
Navigate to Dashboards > New > Import.
Paste the JSON content from the KubeVirt dashboard file.
Select Prometheus as the data source.
Click Load.

Step 3: Customize the Dashboard (Optional)

To add specific VMI-related metrics to the dashboard:

Click Edit on a panel.
Use PromQL queries to fetch desired VMI metrics, e.g.,
- CPU Usage: rate(kubevirt_vmi_cpu_usage_seconds_total[5m])
- Memory Usage: kubevirt_vmi_memory_available_bytes
- Network Traffic: rate(kubevirt_vmi_network_receive_bytes_total[5m])
Save the dashboard.

Note: You can view query output in the Prometheus dashboard.

Conclusion

You have successfully set up Grafana to monitor KubeVirt VMIs using Prometheus. You can now track VMI performance and resource usage effectively.

Kubeopsvm: How to create L3 networking for VMs

Mon, 01 Jan 0001 00:00:00 +0000

Note This guide requires a Compliance enterprise license.

This guide explains how to configure L3 networking for KubeVirt virtual machines (VMs) using Multus. With this setup, VMs can communicate across nodes through dedicated secondary networks, which improves network segmentation and deployment flexibility in Kubernetes environments.

Prerequisites

Before you begin, ensure that the following requirements are met:

A running Kubernetes cluster
kubectl command-line tool configured to connect to your cluster
Calico and Multus installed and configured
KOSI is installed and authenticated successfully

Configuring Node Networking using Network Manager State (NMState)

Network Manager State (nmstate) is a declarative network configuration tool for Linux. It provides an API and CLI to manage network settings in a structured, state-driven manner.

In Kubernetes, nmstate is often used with NodeNetworkConfigurationPolicy (NNCP) to configure node networking dynamically, allowing for changes such as setting up VLANs, bridges, bonds, and other networking features.

1. Deploy the nmstate Operator

First, create the nmstate-values.yaml:

# nmstate-values.yaml
nmstateValues:
  namespace: nmstate

Deploy the nmstate operator and its CRDs using this file:

kosi install --hub kubeops kubeops/nmstate:2.0.1 -f nmstate-values.yaml

2. Configure Node Networking with NNCP

Create an nncp-values.yaml file to define network configurations dynamically, including bridges and routes:

# nncp-values.yaml
namespace: "nmstate"
nodes:
  - nodeName: cluster1master1
    nodeIP: 10.2.10.11
    bridgeIP: 10.10.1.1 # bridge IP for cluster1master1.
    bridgeSubnet: 10.10.1.0 # `/24` subnet for bridge
    nodeInterface: ens192 # use `ip a` or `ifconfig` to identify interface

  - nodeName: cluster1master2
    nodeIP: 10.2.10.12
    bridgeIP: 10.10.2.1
    bridgeSubnet: 10.10.2.0
    nodeInterface: ens192

  - nodeName: cluster1worker1
    nodeIP: 10.2.10.14
    bridgeIP: 10.10.3.1
    bridgeSubnet: 10.10.3.0
    nodeInterface: ens192

  - nodeName: cluster1worker2
    nodeIP: 10.2.10.15
    bridgeIP: 10.10.4.1
    bridgeSubnet: 10.10.4.0
    nodeInterface: ens192

Note:

If you’re adding a new node, assign it a unique bridgeIP and bridgeSubnet, that follow the existing addressing pattern.

For example, if your last node uses:

bridgeIP: 10.10.4.1

bridgeSubnet: 10.10.4.0/24

add the next node like this:
  - nodeName: <new_node>
    nodeIP: <node_ip>
    bridgeIP: 10.10.5.1
    bridgeSubnet: 10.10.5.0
    nodeInterface: ens192

Use this values file to deploy the node network configuration policies on all nodes:

kosi install --hub kubeops kubeops/nncp:2.0.1 -f nncp-values.yaml

3. Allow Forwarded Traffic on the Bridge

Run the following commands on the relevant nodes to allow traffic through the secondary bridge:

iptables -A FORWARD -i br-secondary -j ACCEPT
iptables -A FORWARD -o br-secondary -j ACCEPT

4. Verify Bridge Configuration and Node Connectivity

Verify the Bridge on Each Node

To confirm that the bridge exists and has the expected IP address:

ip a show br-secondary

Verify Attached Interfaces

List the bridge interfaces and verify that the expected interfaces are attached:

ip link show type bridge

Verify Routes and Connectivity

Check if the expected routes are in place:

ip route show

Test Connectivity Between Nodes

Test connectivity to another node’s bridge IP:

ping -c 4 <another_node_bridgeIP>

Install KubeVirt

For installation instructions, please refer to the KubeVirt Installation Guide

Networks and Interfaces

L3 Networking with Multus

To set up L3 networking with Multus, each worker node requires a separate NetworkAttachmentDefinition (NAD) with a unique IP address and subnet matching the node’s network configuration. This allows isolated network environments for each pod, enabling multi-interface support on pods and communication between nodes in the cluster.

Plan the Network

Before creating the NADs, define the IP ranges and subnets for each worker node to avoid conflicts and routing conflicts.

Example IP range mapping for worker nodes:

Node	Subnet	Example IP for Pod
cluster1worker1	10.10.3.0/24	10.10.3.10
cluster1worker2	10.10.4.0/24	10.10.4.10

Example NAD Configuration for Worker Node 1

This example shows how to configure a NetworkAttachmentDefinition (NAD) for worker node 1 (cluster1worker1), with a static IP range assigned to its pods:

# nad_cluster1worker1.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: secondary-network-node1 # Assign unique names to each node's NAD
  annotations:
    k8s.v1.cni.cncf.io/nodeSelector: '{"kubernetes.io/hostname": "cluster1worker1"}'
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "secondary-network",
      "type": "bridge",
      "bridge": "br-secondary",
      "ipam": {
        "type": "static",
        "addresses": [
          {
            "address": "10.10.3.10/24", # use correct worker node's subnet and mention the appropriate IP for pod's network
            "gateway": "10.10.3.1"
          }
        ],
        "routes": [
          {
            "dst": "10.10.4.0/24",
            "gw": "10.10.3.1"
          }
        ]
      }
    }

Apply the K8s Manifest Definition files

To deploy the NetworkAttachmentDefinition (NAD) objects for your nodes, apply the following Kubernetes manifest files:

kubectl apply -f nad_cluster1worker1.yaml
kubectl apply -f nad_cluster1worker2.yaml

Verify the created Network Attachment Definition Objects

To ensure that the NetworkAttachmentDefinition (NAD) objects have been successfully created, run the following command:

kubectl get net-attach-def -A

Deploy Virtual Machines (VMs) and Attach the Secondary Networks

Once the NADs are created, you can deploy Virtual Machines (VMs) and attach them to the secondary networks. The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached.

Network Selection from NADs

Each VM must be connected to the correct network defined by the NetworkAttachmentDefinition (NAD) for its corresponding node. For example, if you have created a NAD for cluster1worker1 named secondary-network-node1, ensure that the VM manifest refers to this network in the annotations section and correctly links to the networkName under the multus configuration.

Ensure that the network name in the manifest matches the NAD definition.

Create Image Pull Secret

Please create a secret for pulling image from registry.kubeops.net.

kubectl -n "l3-networking" create secret docker-registry kubeops-vm \
  --docker-server=registry.kubeops.net \
  --docker-username=<kubeops-user> \
  --docker-password=<harbor_token>

Example VM Manifest for Secondary Network

The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached to a specific NAD:

# vm1.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: "ubuntu-vm"
  namespace: "l3-networking"
  annotations:
    k8s.v1.cni.cncf.io/networks: secondary-network-node1 # Ensure this matches the NAD network name
spec:
  nodeSelector:
    kubernetes.io/hostname: cluster1worker1 # edit for scheduling node on other node
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/size: small
    spec:
      domain:
        cpu:
          cores: 1
        devices:
          disks:
            - name: containervolume
              disk:
                bus: virtio
            - name: cloudinitvolume
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: secondary-network
            bridge: {}
        resources:
          requests:
            memory: 2048M
      networks:
      - name: default
        pod: {}
      - name: secondary-network
        multus:
          networkName: secondary-network-node1 # Ensure this matches the NAD network name
      volumes:
        - name: containervolume
          containerDisk:
            image: registry.kubeops.net/kubeops-vm/kubeops/ubuntu-cloudimg-focal:2.0.1
            imagePullSecret: kubeops-vm
            imagePullPolicy: Always
        - name: cloudinitvolume
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              chpasswd:
                list: |
                  ubuntu:ubuntu
                  root:toor
                expire: False
              ssh_pwauth: True
              disable_root: false

Apply the VM Manifest

Once the NetworkAttachmentDefinition (NAD) objects have been created, you can deploy the Virtual Machines (VMs) and apply the manifests. The following commands apply the VM manifests to your Kubernetes cluster.

kubectl apply -f vm1.yaml
kubectl apply -f vm2.yaml

Verify the Networking

After the VMs start successfully, verify that the secondary network interfaces and routes are configured correctly.

On each VM

1. Check the IP Address

First, check the IP address assigned to the pod (VM). This is crucial to ensure that the secondary network interface has been correctly assigned an IP.

kubectl exec -n <namespace> <pod-1> -- ip a

2. Assign IP Address if Not Automatically Assigned (Optional)

kubectl exec -n <namespace> <pod-1> -- sudo dhclient enp2s0

3. Check the Routes

Verify the routing configuration on the VM to ensure that traffic can flow through the secondary network. Use the following command:

kubectl exec -n <namespace> <pod-1> -- ip r

This will display the routing table for the VM and confirm if the secondary network is correctly set up.

Verify VM Communication

After ensuring the network settings are correct, test the communication between VMs by pinging the secondary network IP of another pod.

Test Connectivity Between VMs

Use the ping command to check if the VMs can communicate over the secondary network. Replace <pod2_secondary_network_IP> with the actual IP address of the secondary network interface on the second VM.

kubectl exec -n <namespace> <pod-1> -- ping -c 2 <pod2_secondary_network_IP>

If the ping is successful, it confirms that the VMs are able to communicate over the secondary network.

Additional Notes:

Ensure that you use the correct namespace and pod name while executing the commands.

Make sure the secondary network interface is correctly configured on each VM, and the IP address and routes are properly assigned.

If communication fails, check for issues in the NetworkAttachmentDefinition (NAD) configuration or verify that the correct CNI plugins are running.

Kubeopsvm: How to back up VMs and vm snapshots with Velero

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This guide describes how to configure Velero to back up and restore KubeVirt virtual machines in a Kubernetes cluster.

Velero stores backup data in an S3-compatible object storage backend. In this example, MinIO is used as the backup storage target.

Prerequisites

Note This feature requires a Compliance enterprise license.

Before you begin, ensure that the following prerequisites are met:

A running Kubernetes cluster is available.
Rook-Ceph is configured as the storage provider.
An S3-compatible object storage service is available.( in this case MinIO)
A KubeVirt virtual machine is running in the cluster

In following example MinIO is used.

Step 1 - Configure Velero

Define the Velero configuration in your values file:

veleroValues:
 namespace: "velero"
 accessKeyId: "minio"
 secretAccessKey: "minio123"
 backupLocationConfig:
   s3Url: "http://10.2.10.110:30090" # nodeport of minio

Enable Velero in the kubeopsctl.yaml file :

velero: true

Apply the configuration:

kubeopsctl apply -f kubeopsctl.yaml

Step 2 - Verify S3 Bucket Access

The bucket for Velero ( default Velero ) in the s3 storage should be accessible for Velero, so ensure that the bucket exists and that Velero has permission to write to it.

In this example, the bucket is configured so that Velero has write access:

Step 3 - Create a Backup

To create a backup of the namespace containing the KubeVirt virtual machine, run:

velero backup get
velero backup create kubeops-vm-backup --include-namespaces <Namespace of kubevirt vm>

The name of the backup must be unique.

Step 4 - Verify the Backup

To review the backup status and details, run:

velero backup get
velero backup describe kubeops-vm-backup
velero backup logs kubeops-vm-backup

Confirm that the backup completed successfully before proceeding.

Step 5 - Restore the Backup

To restore the backup, run:

velero restore get
velero restore create kubeops-vm-restore --from-backup kubeops-vm-backup

The restore name must be unique.
After the restore is complete, verify that the virtual machine resources have been recreated successfully in the target namespace.

velero restore get
velero restore describe kubeops-vm-restore
velero restore logs kubeops-vm-restore

KubeOpsDoku – license-required

Kubeopsctl-2.0: Joining a Node in a Kubernetes cluster

Join Node Process:

Prerequisits

KOSI Login Recommendation

ETCD Backup Recommendation

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Example 1: Joining a Worker Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Kubeopsctl-2.0: Deleting a Node from a Kubernetes cluster

Deleting a Node from a Kubernetes cluster

Prerequisites

Worker

Scale down the rook-ceph-operator deployment to 0

Check which hosts and OSDs belong to each zone

Scale down the OSD deployment

Remove the OSD via ceph-tools

Check the MON running on the node

Delete OSD and MON deployments

Remove the deleted mon from the ceph tools

Delete the node from the kubernetes cluster

Scale the rook-ceph-operator deployment back to 1

Timing and health checks

Kubeopsctl-2.0: Upgrading a Kubernetes cluster

Kubernetes Version Upgrade Process:

Prerequisites

KOSI Login Recommendation

1. Pull required KOSI packages on your ADMIN

2. Change your target version inside the cluster-values

3. Start the upgrade

Example 1 - Upgrade all nodes in the cluster to a specific version

1. Pull required KOSI packages on your ADMIN

2. Update the target version in cluster-values.yaml

2. Validate your values and upgrade the cluster

Example 2 - Tranche upgrade of zones to a specific version

1. Pull required KOSI packages on your ADMIN

2. Adjust your cluster-values in zone1

3. Validate your values and upgrade the cluster

4. Adjust your cluster-values.yaml in zone2

5. Validate your values and upgrade the cluster

6. Adjust your cluster-values.yaml in zone3

7. Validate your values and upgrade the cluster

Kubeopsctl-2.0: Installing KubeOps Compliance applications

Prerequisits

KOSI Login Recommendation

Example 1: Installing Applications in a non-airgap-environment

Example 2: Installing Applications in an airgap-environment

Kubeopsctl-2.0: Updating KubeOps Compliance applications

Prerequisites

KOSI Login Recommendation

Updated KubeOpsctl

Example 1: Updating Applications in a non-airgap-environment

Example 2: Updating Applications in an airgap-environment

Kubeopsvm: How to install KubeOps Virtualization (kubevirt)

Requirements

Step 1 - Download KubeOpsVM

Step 2 - Remove the Taint of the Master Node (only necessary on single node cluster)

Step 3 - Create Kubevirt Configuration File

Step 4 - Install KubeOps-VM

Step 5 - Pull and Install KubeVirt Manager Package

Next Steps

Kubeopsvm: How to create and deploy a Custom Virtual Machine using KubeOps

Requirements

Additional Information

How to create and deploy a Custom Virtual Machine

Step 1 - Create a Configuration File for the Virtual Machine

Step 2 - Transfer the VM Image to your Local Registry

Step 3 - Deploy the Virtual Machine

Step 4 - Access the Custom VM

References

Kubeopsvm: How to access a virtual machine in a KubeOps Cluster

Requirements

Step 1 - Identify your Virtual Machine

Step 2 - Access the Virtual Machine

Step 1 - Create a Service forwarding the SSH port of the VM

Step 2 - Access the Custom VM

Kubeopsvm: How to create a ContainerDisk from a local image

Kubeopsvm: How to import a VM from vSphere and deploy it in a Cluster using KubeVirt

Requirements

2. Update the target version in `cluster-values.yaml`

2. Adjust your cluster-values in `zone1`

4. Adjust your `cluster-values.yaml` in `zone2`

6. Adjust your `cluster-values.yaml` `in zone3`