Documentation for production

• 1: How-To: Production Cluster with OIDC (Public IP & Airgap)
• 2: Supported Package Versions — Ubuntu / Debian
• 3: Supported Package Versions — Red Hat Enterprise Linux (RHEL)

1 - How-To: Production Cluster with OIDC (Public IP & Airgap)

How-To: Production Cluster with OIDC (Public IP & Airgap)

This how-to walks a customer through building a production-ready KubeOps Compliance environment from clean machines to a running, fully equipped, authenticated cluster: the complete platform stack (storage, registry, monitoring, logging, policy, dashboard, backup) with Keycloak providing OIDC authentication, and the dashboards reachable over HTTPS with valid certificates.

It covers the combinations a customer realistically needs:

• Network path — internet-connected (with public IPs) or airgap (KubeOps Compliance’s core strength).
• Certificates — automatic Let’s Encrypt (public IPs) or your own CA (private DNS / airgap).
• Ingress — nginx (default) or Traefik, including how to migrate.

kubeopsctl uses two configuration files, and it is important to keep them apart:

• cluster-values.yaml — defines the cluster itself (nodes, networking, virtual IP). It has no influence over which applications get installed.
• enterprise-values.yaml — defines the platform applications as a list of packages.

Overview: what you will build

A highly available cluster with the minimum supported production topology and the full platform stack:

• 1 admin node, 3 control-plane nodes, 3 worker nodes (each worker with extra disks for storage), across three zones
• a KubeVip virtual IP as the stable, highly available API endpoint
• the full stack via enterprise-values.yaml: Rook-Ceph (storage), Keycloak (OIDC), ingress-nginx or Traefik, cert-manager (TLS), Harbor (registry), Prometheus/Grafana (monitoring), OpenSearch + Logstash + Filebeat (logging), OPA Gatekeeper (policy), KubeOps Dashboard, Velero (backup)
• dashboards reachable via a domain name with valid certificates and Keycloak SSO

Choose your path

Two decisions shape the rest of this guide. Pick the row that matches your environment:

Why deployment differs:

• Let’s Encrypt (single-phase): cert-manager automatically creates a letsencrypt ClusterIssuer, and every ingress references it, so valid, browser-trusted certificates are issued as the components come up. Everything can be installed in one apply.
• Own CA (two-phase): the OIDC dashboard integration needs a valid certificate to already exist when it runs. If it runs against a self-signed/placeholder certificate, it falls back to a dummy secret and breaks. So you first bring up storage, registry, ingress and cert-manager, install your valid certificate manually, and only then install Keycloak and the rest.

The ingress controller (nginx or Traefik) is an independent choice — see section 8.

1. Prerequisites

At least seven machines: one admin, three control-plane, three worker.

Before you start:

• kubeopsctl 2.1.0 or newer (required for OIDC).
• Same username and password on every machine (cluster creation fails otherwise).
• Unique, lowercase hostname per machine.
• OS version pinned to a supported release.
• Network access: for the internet path, the cluster nodes reach the KubeOps registry directly. For the airgap path, only the admin node needs temporary outbound access to https://dispatcher.kubeops.net (KOSI hub) and https://registry.kubeops.net while kosi / kubeopsctl run; the cluster nodes stay isolated and pull all images from the Harbor that the platform stack deploys (section 5.1). KubeOps supports secure registries only.
• Storage disks for Rook-Ceph: every persistent component uses the rook-cephfs storage class, which is 3× replicated across the workers — so raw disk ≈ 3 × the sum of all PVC claims. For the full platform stack with 30-day retention, provide one dedicated, unformatted disk of at least ~300 GB per worker (500 GB recommended); see the sizing table in section 7.1. Also see the Rook-Ceph prerequisites.
• yq on all control-plane nodes (required for the OIDC dashboard integration) — yq install.
• Passwords prepared for Harbor (and its Postgres/Redis), Keycloak (admin + Postgres) and Grafana — never the documented defaults.
• DNS: for OIDC you need a DNS name for Keycloak and the dashboards. For Let’s Encrypt this must be public DNS; for an own CA it can be internal DNS.

2. Plan addressing and ports

Example node plan (203.0.113.x are placeholders from the documentation range, RFC 5737 — replace with your real IPs):

Default service ports:

Ports that must stay open between the cluster nodes (opened automatically by the KubeOps firewall setup):

3. Prepare all nodes

Run on every node unless noted.

3.1 Add the package repository (internet path)

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

wget https://packagerepo.kubeops.net/pgp-key.public
cat pgp-key.public | sudo gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/kubeops.gpg] https://packagerepo.kubeops.net/deb stable main' | sudo tee /etc/apt/sources.list.d/kubeops.list
sudo apt update

sudo dnf config-manager --add-repo https://packagerepo.kubeops.net/rpm/kubeops.repo

3.2 Operating system adaptations

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

sudo apt remove unattended-upgrades

# no special adaptations required

3.3 Make hostnames resolvable

Use DNS for all node hostnames, or add them to /etc/hosts on every node:

sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
203.0.113.10 admin
203.0.113.11 controlplane01
203.0.113.12 controlplane02
203.0.113.13 controlplane03
203.0.113.21 worker01
203.0.113.22 worker02
203.0.113.23 worker03
EOL_ETC_HOSTS

3.4 Distribute SSH keys

Passwordless SSH must work from each node to every other node. Root login is temporarily enabled and reverted in section 4.

printf 'PermitRootLogin yes\nPasswordAuthentication yes\n' | sudo tee /etc/ssh/sshd_config.d/permit_root_login.conf
sudo systemctl restart sshd

# create keys on all nodes
ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N ""

# copy the public key, scan host keys, and test login — for EVERY node
for host in admin controlplane01 controlplane02 controlplane03 worker01 worker02 worker03; do
  ssh-copy-id -i ~/.ssh/id_ed25519 "$host"
  ssh-keyscan "$host" >> ~/.ssh/known_hosts
  ssh "$host" exit
done
# repeat ssh-keyscan with the IP addresses as well

3.5 Distribute the sudoers file

The cluster is built and operated by the unprivileged user from the warning at the top of this guide — not as root. That user needs the elevated rights kubeopsctl requires, granted through a sudoers file. Replace myuser with your username and copy the file to /etc/sudoers.d/<username> on all nodes.

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/apt/sources.list.d/kubeops.list
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove unattended-upgrades
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove *
myuser ALL=(root) NOPASSWD: /usr/bin/lsof /var/lib/dpkg/lock*

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kosi*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kubeopsctl*.deb


# kube-vip image import
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# Kubeadm init
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# enable UFW
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reset
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 6443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 2379\:2380/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10250/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10259/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10257/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10256/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 30000\:32767/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 179/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 4789/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5473/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 51820/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 22/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5000/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5001/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 7443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw logging low
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw enable
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reload
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw status
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart systemd-networkd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable --now ufw

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# kubeadm init phase and kubeadm token create commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /usr/bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# upgrade nodes
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin
myuser ALL=(root) NOPASSWD: /bin/rm -f /etc/cni/net.d/87-podman-bridge.conflist

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubeadm*
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubectl*
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubelet*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark hold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark unhold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubelet* kubeadm* kubectl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeadm*

# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install passt_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conmon_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install catatonit_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install netavark_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install aardvark-dns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-image_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-common_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containernetworking-plugins_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libsubid4_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install uidmap_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libslirp0_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install slirp4netns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libyajl2_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install crun_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install fuse-overlayfs_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install buildah_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install podman_*.deb

# Podman Remove of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove podman buildah fuse-overlayfs crun libyajl2 slirp4netns libslirp0 uidmap libsubid4 containernetworking-plugins golang-github-containers-common golang-github-containers-image aardvark-dns netavark catatonit conmon passt

# Podman Installation & Update with apt
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install */pia/kosi_*

# runtime setup
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conntrack_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install runc_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containerd_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y iproute-tc

# Remove packages
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove containerd

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard_*.deb wireguard-tools_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard-tools-*.deb systemd-resolved-*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove wireguard wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y wireguard-tools*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y systemd-resolved*

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/local/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf config-manager --add-repo https\://packagerepo.kubeops.net/rpm/kubeops.repo
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kosi*.rpm, !/usr/bin/rpm --install kosi*[[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kubeopsctl*.rpm, !/usr/bin/rpm --install kubeopsctl*[[\:space\:]]*.rpm

# kube-vip/create-manifest.sh.tmpl
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# delete-node/delete-node.sh.tmpl
# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# kubeadm-init/kubeadm-init.sh.tmpl
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/sysconfig/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm
# if Helm is in repository
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y helm* 

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s/package.kosi
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# ssh remote kubeadm commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# local execution of kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
# Adjustments sed -i * 
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# kubeadm upgrade node
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin, !/bin/tar xzf cni-*[[\:space\:]]* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin, !/bin/tar xzf crictl-*[[\:space\:]]* -C /usr/bin
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubeadm*, !/usr/bin/dnf install -y kubeadm*[[\:space\:]]*, !/usr/bin/dnf install -y kubeadm*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*, !/bin/mv /usr/bin/kubelet /usr/bin/kubelet_*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubectl*, !/usr/bin/dnf install -y kubectl*[[\:space\:]]*, !/usr/bin/dnf install -y kubectl*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubelet*, !/usr/bin/dnf install -y kubelet*[[\:space\:]]*, !/usr/bin/dnf install -y kubelet*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubelet-* kubeadm-* kubectl-*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeadm-*


# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install aardvark-dns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install netavark-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install container-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libnet-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-libs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libslirp-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install slirp4netns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install yajl-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install crun-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containers-common-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install fuse-overlayfs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install shadow-utils-subid-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conmon-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install podman-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-*.rpm iproute-tc-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containerd.io-*.rpm

# Podman Remove of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase podman shadow-utils-subid fuse-overlayfs crun containers-common yajl slirp4netns libslirp criu-libs criu libnet container-selinux netavark aardvark-dns passt passt-selinux

# Podman Installation & Update with dnf
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y podman*, !/usr/bin/dnf install -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf update -y podman*,!/usr/bin/dnf update -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install */pia/kosi-*

# Allow installation of container-runtime
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm, !/usr/bin/rpm --install conntrack-tools-*[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm libnetfilter_cthelper-*.rpm libnetfilter_cttimeout-*.rpm libnetfilter_queue-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-tc-*.rpm, !/usr/bin/rpm --install iproute-tc-*[\:space\:]]*.rpm

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y iproute-tc

# Remove RPM packages
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase containerd.io

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/systemd/system/containerd.service.d/
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/systemd/system/containerd.service.d/override.conf*

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install wireguard-tools-*.rpm systemd-resolved-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y wireguard-tools*, !/usr/bin/dnf install -y wireguard-tools*[\:space\:]]*, !/usr/bin/dnf install -y wireguard-tools*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y systemd-resolved*, !/usr/bin/dnf install -y systemd-resolved*[\:space\:]]*, !/usr/bin/dnf install -y systemd-resolved*.rpm

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

3.6 Install kubeadm

• Operating-System:
• Ubuntu 24.04
• RHEL 9.6

sudo apt install -y kubeadm=<kubernetes-version>*

sudo dnf install -y --disableexcludes=kubeops-repo kubeadm-<kubernetes-version>

3.7 Configure time synchronization

• Ubuntu 24.04
• RHEL 9.6

sudo apt install -y chrony
sudo systemctl enable --now chrony

sudo dnf install -y chrony
sudo systemctl enable --now chronyd

Configure NTP servers in /etc/chrony.conf, sudo systemctl restart chronyd, then verify with chronyc tracking (leap status Normal, offset < 10 ms).

3.8 Install curl and yq

Install curl on every node, and yq on all control-plane nodes (required for the OIDC dashboard integration) per the yq installation guide.

4. Harden the nodes for production

Revert the SSH change from section 3.4 on every node:

sudo rm /etc/ssh/sshd_config.d/permit_root_login.conf
sudo systemctl restart sshd

Also: pin the OS version and tighten the firewall per section 2.

Pin the KOSI-installed packages. kubeopsctl installs a set of system packages (podman, containerd, the Kubernetes tools, etc.) at specific, validated versions. An OS update (apt upgrade / dnf update) can silently upgrade them and break the cluster. After section 6, pin/hold every one of these packages to the installed version so OS updates cannot move them.

• Ubuntu: sudo apt-mark hold <package> (e.g. podman, containerd, kubelet, kubeadm, kubectl)
• RHEL: exclude them from updates, e.g. sudo dnf update --exclude=podman,containerd,kubelet,kubeadm,kubectl, or use the versionlock plugin

5. Set up the admin tooling

Perform the rest from the admin node.

Install KOSI, then set KUBEOPSROOT, adjust the KOSI config, install the enterprise plugins, log in, and install kubeopsctl (≥ 2.1.0) — exactly as in the Setup Cluster guide:

# KOSI
sudo apt install -y kosi=<kosi-version>          # Ubuntu  | RHEL: sudo dnf install -y --disableexcludes=kubeops-repo <kosi-version>

# KUBEOPSROOT (append to ~/.bashrc, then: source ~/.bashrc)
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

# KOSI config + plugins
mkdir ~/kubeops && cd ~/kubeops
cp -R /var/kubeops/kosi/ . && cp -R /var/kubeops/plugins/ .
# edit $KUBEOPSROOT/kosi/config.yaml: set hub and the FULL plugins path

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.0.0
kosi login -u <yourUser>                          # a "registry temporarily not available" error here is expected

# kubeopsctl (>= 2.1.0 for OIDC)
sudo apt install -y kubeopsctl=<kubeopsctl-version>   # Ubuntu | RHEL: sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

5.1 (Airgap only) Stage the images in your local Harbor

In an airgapped environment the platform-stack images are served from your local Harbor registry. The flow is:

1. rook and harbor first. Only rook-ceph and harbor ship dedicated airgap packages that carry their own container images. Install these two first so storage and the registry exist.
2. Seed the remaining images. Once Harbor is up, kubeopsctl pulls every other tool’s images from registry.kubeops.net, pushes them into the deployed Harbor, and re-tags each reference to 127.0.0.1:30002/kubeops/<image> — Harbor reached over its NodePort 30002 on each node, project kubeops.
3. Private kubeops project. The kubeops Harbor project is private, so every tool is deployed with an image pull secret for it.
4. Point the config at local. Set airgap: true and packageRepository: local in cluster-values.yaml (section 6), and localRegistry: true in enterprise-values.yaml (section 7).

6. Create the cluster

The cluster-values.yaml describes only the cluster. The airgap flag is the key difference between the two network paths.

• Internet
• Airgap

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: true
virtualIP: <unused IP in the control-plane network>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: false
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true
zones:
- name: zone1
  nodes:
  - { name: <controlplane01 hostname>, iPAddress: <controlplane01 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker01 hostname>,       iPAddress: <worker01 ip>,       type: worker,       kubeVersion: <version> }
- name: zone2
  nodes:
  - { name: <controlplane02 hostname>, iPAddress: <controlplane02 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker02 hostname>,       iPAddress: <worker02 ip>,       type: worker,       kubeVersion: <version> }
- name: zone3
  nodes:
  - { name: <controlplane03 hostname>, iPAddress: <controlplane03 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker03 hostname>,       iPAddress: <worker03 ip>,       type: worker,       kubeVersion: <version> }

# file cluster-values.yaml  (airgap)
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: <your local Harbor>/kubeops    # your local secure registry
airgap: true
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: true
virtualIP: <unused IP in the control-plane network>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: false
packageRepository: local
changeCluster: true
zones:
  # ... same zone layout as the Internet tab ...

Then pull packages, install the runtime tooling, and create the cluster:

kubeopsctl pull                                                        # add --kubernetesVersion <x.xx.x> to pin a version
kosi install -p $KUBEOPSROOT/lima/podman_<version>.tgz -f cluster-values.yaml
kosi install -p $KUBEOPSROOT/lima/helm_<version>.tgz
kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml
kosi login -u <your username>
kubeopsctl apply -f cluster-values.yaml                                # creates the cluster (~10–15 min)

7. Configure the platform stack

The applications live in enterprise-values.yaml. Each package has a standard block (predefined keys) and an advanced block (any value from the underlying Helm chart). Standard values overwrite advanced values if both are set.

The full reference below enables the whole stack and is wired for OIDC dashboard access (each dashboard gets a hostname). It is not applied as-is in one step for every scenario — the ingress choice (section 8), the certificate scenario (section 9) and the deployment (section 10) determine how.

# file enterprise-values.yaml
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false            # airgap: true
packages:

# Storage — provides the rook-cephfs storage class used by everything below
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      hostname: <domain_name>
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
    advanced:
      # disk selection for Ceph goes here (rook-ceph-cluster Helm values),
      # e.g. storage.useAllDevices / storage.deviceFilter / per-node devices
      cluster:
      operator:

# Ingress controller — externalIPs binds it to your IP(s). For Traefik see section 8.
- name: ingress-nginx
  enabled: true
  values:
    standard:
      namespace: ingress-nginx
      externalIPs:
      - <ingress ip>             # public IP (Scenario A) or internal IP (Scenario B)
    advanced:

# Certificates
- name: cert-manager
  enabled: true
  values:
    standard:
      namespace: cert-manager
      replicaCount: 3
      logLevel: 2
      # Scenario B (own CA): point cert-manager at your own certificate secret
      # secretName: example-ca
    ca:
      # Scenario A (Let's Encrypt): cert-manager auto-creates a "letsencrypt" ClusterIssuer
      emailLetsEncrypt: <your_email@domain.com>   # MUST set (default example@example.com will not work)
      ingressName: nginx                          # MUST match the ingress controller (nginx or traefik)

# OIDC provider — Keycloak
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: keycloak
      storageClass: "rook-cephfs"
      nodePort: "30180"
      hostname: <domain_name>
      keycloak:
        auth:
          adminUser: admin
          adminPassword: "<strong password>"
          existingSecret: ""
      postgresql:
        auth:
          postgresPassword: "<strong password>"
          username: bn_keycloak
          password: "<strong password>"
          database: bitnami_keycloak
    advanced:

# Registry (also the local registry in airgap)
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "<strong password>"
      databasePassword: "<strong password>"
      redisPassword: "<strong password>"
      externalURL: https://<domain_name>
      nodePort: 30002
      hostname: <domain_name>
      harborPersistence:
        persistentVolumeClaim:
          registry:                                # full-stack images ~5 GB; rest = headroom for your own images — enable Harbor GC + a project quota (see section 7.1)
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:                                # also holds Trivy scan reports
            size: 5Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy:
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:

# Monitoring
- name: kube-prometheus-stack
  enabled: true
  values:
    standard:
      namespace: monitoring
      grafanaUsername: admin
      grafanaPassword: "<strong password>"
      grafanaResources:
        hostname: <domain_name>
        nodePort: 30211
        storageClass: "rook-cephfs"
        storage: 10Gi
      prometheusResources:
        hostname: <domain_name>
        nodePort: 32090
        storageClass: "rook-cephfs"
        storage: 50Gi
        retention: 30d
        retentionSize: "45GB"
    advanced:
      # Alertmanager ships with no persistent storage (emptyDir) — give it a PVC so silences/state survive restarts
      alertmanager:
        alertmanagerSpec:
          storage:
            volumeClaimTemplate:
              spec:
                storageClassName: rook-cephfs
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 2Gi

# Logging stack
- name: opensearch-os
  enabled: true
  values:
    standard:
      namespace: logging
      hostname: <domain_name>
      opensearchJavaOpts: "-Xmx512M -Xms512M"
      resources:
        requests:
          cpu: "250m"
          memory: "1024Mi"
        limits:
          cpu: "300m"
          memory: "3072Mi"
      persistence:                                 # size for your chosen log retention (see section 7.1)
        size: 40Gi
        storageClass: "rook-cephfs"
        accessModes: ["ReadWriteMany"]
      securityConfig:
        enabled: false
      replicas: "3"
    advanced:
      # Log retention: no lifecycle policy ships by default, so log indices grow UNBOUNDED.
      # This Job installs an ISM policy that deletes log indices older than the configured age and
      # re-runs idempotently on every `kubeopsctl apply` (upgrade-safe). See section 7.1.
      extraObjects:
      - apiVersion: batch/v1
        kind: Job
        metadata:
          name: kubeops-log-retention
          namespace: logging
        spec:
          ttlSecondsAfterFinished: 300
          backoffLimit: 6
          template:
            spec:
              restartPolicy: OnFailure
              containers:
              - name: apply-ism
                image: curlimages/curl:8.9.1        # airgap: reference this image from your local Harbor
                command: ["/bin/sh","-c"]
                args:
                - |
                  # OpenSearch endpoint — http + no auth matches securityConfig.enabled=false above;
                  # if you enable securityConfig, switch to https and pass admin credentials.
                  OS="http://opensearch-cluster-master.logging.svc:9200"
                  # >>> RETENTION: change "30d" to "60d", "90d", ... as needed <<<
                  cat >/tmp/policy.json <<'JSON'
                  {"policy":{"description":"Delete KubeOps log indices","default_state":"hot",
                   "states":[
                     {"name":"hot","actions":[],"transitions":[{"state_name":"delete","conditions":{"min_index_age":"30d"}}]},
                     {"name":"delete","actions":[{"delete":{}}],"transitions":[]}],
                   "ism_template":[{"index_patterns":["logstash-logs-*"],"priority":100}]}}
                  JSON
                  curl -sf -XPUT  "$OS/_plugins/_ism/policies/kubeops-log-retention" -H 'Content-Type: application/json' -d @/tmp/policy.json
                  curl -sf -XPOST "$OS/_plugins/_ism/add/logstash-logs-*"           -H 'Content-Type: application/json' -d '{"policy_id":"kubeops-log-retention"}' || true                  
- name: opensearch-dashboards
  enabled: true
  values:
    standard:
      namespace: logging
      hostname: <domain_name>
      nodePort: 30050
    advanced:
- name: logstash-os
  enabled: true
  values:
    standard:
      namespace: logging
      volumeClaimTemplate:
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 5Gi
        storageClass: "rook-cephfs"
    advanced:
      # Persistent queue so log bursts/backpressure are buffered on disk instead of dropped.
      # NOTE: this REPLACES the shipped logstash.yml — the first two lines are the existing defaults, keep them.
      logstashConfig:
        logstash.yml: |
          pipeline.ecs_compatibility: disabled
          log.level: info
          queue.type: persisted
          queue.max_bytes: 1gb          
- name: filebeat-os
  enabled: true
  values:
    standard:
      namespace: logging
    advanced:

# Policy
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:

# Cluster dashboard
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: <domain_name>
      service:
        nodePort: 30007
    advanced:

# Backup
- name: velero
  enabled: true
  values:
    standard:
      namespace: velero
      accessKeyId: "<s3 access key>"
      secretAccessKey: "<s3 secret key>"
      provider: "aws"
      bucket: "velero"
      useNodeAgent: true
      useVolumeSnapshots: true
      backupLocationConfig:                        # placeholder — point bucket/region/s3Url at an EXTERNAL S3 store (see section 14)
        region: "minio"
        s3ForcePathStyle: true
        s3Url: "http://minio.velero.svc:9000"
    advanced:
      # Out of the box Velero backs up only Kubernetes objects, on no schedule, deleted after 72h.
      # Add a recurring schedule and a retention TTL (720h = 30d):
      schedules:
        daily:
          schedule: "0 */6 * * *"
          template:
            ttl: "720h"
      # Back up PV data too. NOTE: the `standard` key defaultVolumesToFsBackup is emitted at top level
      # where this chart ignores it — it must live under `configuration:` (set it here in advanced).
      configuration:
        defaultVolumesToFsBackup: true

Secure the components: replace every password (or use existingSecret), set the public addresses (externalURL, hostname, ingress externalIPs), and restrict the dashboards/registry at the firewall to trusted networks.

7.1 Sizing, retention and backups

All persistent components use the rook-cephfs storage class, which is 3× replicated across the worker nodes (failureDomain: host, reclaimPolicy: Retain). Plan raw disk ≈ 3 × the sum of all PVC sizes and never fill Ceph beyond ~75–80 %.

Recommended minimal-but-stable sizes (7-node baseline, 30-day retention). These are the values used in the example above — adjust to your workload, since logging and monitoring volumes scale with the number of nodes/pods and with your retention.

The sum of usable PVCs is ≈ 215–250 Gi, so raw Ceph is ≈ 650–750 Gi; with headroom, plan ~900 Gi–1 TiB raw. Because the pools replicate across the three workers, provide a dedicated, unformatted disk of at least ~300 Gi per worker (recommended 500 Gi) for the Ceph OSDs. This supersedes the “50 GB disk per worker” figure in the prerequisites, which only covers an empty cluster.

Rule of thumb for logs (drives OpenSearch and Ceph): budget roughly 0.5–1 GB/day for a small cluster, × your retention days, × 2 for the index replica — and size the OpenSearch PVC to match.

• Log retention (OpenSearch) — KubeOps ships no index-lifecycle policy, so log indices grow unbounded and eventually force OpenSearch read-only. The opensearch-os package above therefore carries an extraObjects Job that installs an ISM policy deleting logstash-logs-* indices older than a configurable age. Set your retention in that block via min_index_age ("30d" / "60d" / "90d"). The Job only installs the policy once; from then on OpenSearch’s ISM plugin enforces it continuously on its own — deleting aged indices and auto-attaching the policy to every new logstash-logs-* index via ism_template. So retention keeps working between releases (even months apart), not only during an apply. The Job re-runs on every kubeopsctl apply, so the policy is also restored after a fresh reinstall.
• Metrics retention (Prometheus) — set in the standard block via prometheusResources.retention / .retentionSize (example: 30d / 45GB on a 50Gi PVC). Keep retentionSize a few GB below the PVC size for WAL/compaction headroom.
• Harbor cleanup (GC) — the Harbor chart configures no garbage collection, so the registry PVC keeps growing as tags/images accumulate. Deploy the GC-schedule Job shown below; once applied, Harbor runs GC on that schedule itself. To cap the private kubeops project, set a storage quota in the UI under Projects → kubeops → Configuration → storage quota (API: PUT /api/v2.0/quotas/<id>).
• Backups (Velero) — out of the box Velero captures only Kubernetes objects, on no schedule, deleted after 72h. The example enables the node agent and volume snapshots, adds a daily schedule with a 30-day ttl, and sets configuration.defaultVolumesToFsBackup: true so PV data is included — see section 14.

Harbor GC-schedule Job (apply once; Harbor then runs GC on the schedule itself):

# harbor-gc.yaml — apply with: kubectl -n harbor apply -f harbor-gc.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: kubeops-harbor-gc
  namespace: harbor
spec:
  ttlSecondsAfterFinished: 300
  backoffLimit: 10
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: harbor-gc
        image: curlimages/curl:8.9.1          # airgap: reference this image from your local Harbor
        command: ["/bin/sh","-c"]
        args:
        - |
          API="http://harbor-core.harbor.svc/api/v2.0"
          AUTH="-u admin:<harbor admin password>"
          for i in $(seq 1 60); do curl -s $AUTH "$API/health" >/dev/null 2>&1 && break; sleep 5; done
          # weekly GC (6-field cron), delete untagged artifacts — change the cron as needed
          BODY='{"parameters":{"delete_untagged":true},"schedule":{"type":"Custom","cron":"0 0 4 * * 0"}}'
          curl -s $AUTH -XPUT  "$API/system/gc/schedule" -H 'Content-Type: application/json' -d "$BODY" \
            || curl -s $AUTH -XPOST "$API/system/gc/schedule" -H 'Content-Type: application/json' -d "$BODY"          

8. Choose the ingress controller

KubeOps supports nginx (default) and Traefik. Whichever you use, the cert-manager ingressName must match it.

• ingress-nginx (default)
• Traefik

# values.yaml
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik
      externalIPs: []        # set your ingress IP(s)
    advanced: {}

# non-airgap
kosi search --hub kubeops --ps traefik
kosi install --hub kubeops kubeops/traefik:<version> -f values.yaml --dname traefik

# airgap
podman login <harbor ip>:<nodePort> -u <user> -p <pass> --tls-verify=false
kosi pull --hub kubeops kubeops/traefik:<version> -o traefik-<version>.tgz -r <harbor ip>:<nodePort>/kubeops -t localhost:<nodePort>/kubeops
kosi install --dname traefik -p traefik-<version>.tgz -f values.yaml

kosi list                                                              # find the nginx deployment name
kosi delete --hub kubeops kubeops/ingress-nginx:<version> -f enterprise-values.yaml --dname <name>
kubectl edit svc traefik -n traefik                                    # set web/websecure nodePorts (e.g. 30080/30443)

9. Certificates and external access

• Scenario A — Public IP + Let’s Encrypt
• Scenario B — Own CA (private DNS / airgap)

- name: cert-manager
  values:
    ca:
      emailLetsEncrypt: <your_email@domain.com>
      ingressName: nginx        # or traefik

# example only — use a certificate from your corporate CA in production
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt -subj "/CN=<your domain>"
kubectl create secret tls example-ca --key="tls.key" --cert="tls.crt" -n cert-manager

- name: cert-manager
  values:
    standard:
      secretName: example-ca

10. Deploy the platform stack

• Scenario A — single-phase (Let’s Encrypt)
• Scenario B — two-phase (own CA / airgap)

kubeopsctl apply -f enterprise-values.yaml

- name: rook-ceph      # enabled: true
- name: harbor         # enabled: true
- name: ingress-nginx  # enabled: true   (or install Traefik per section 8)
- name: cert-manager   # enabled: true

kubeopsctl apply -f enterprise-values.yaml
kubectl get storageclass      # rook-cephfs present

curl -I https://<domain_name>/

kubeopsctl apply -f enterprise-values.yaml

11. Single Sign-On for Harbor and Rook-Ceph

Most dashboards (Grafana, Prometheus, OpenSearch, KubeOps Dashboard) are wired to OIDC by the integration. Harbor and Rook-Ceph are configured separately and must be set up manually after Keycloak is running. Both use a Keycloak realm named kubeops-dashboards with a user kubeops.

Harbor

1. In Keycloak (kubeops-dashboards realm) create a confidential client harbor (standard flow on, direct access grants off) with redirect URI https://<domain_name>/c/oidc/callback and web origin <domain_name>; copy the client secret.
2. Create a secret with the client id/secret in the Harbor namespace.
3. In the Harbor package’s advanced values, inject the OIDC client id/secret as env vars and set CONFIG_OVERWRITE_JSON with auth_mode: oidc_auth, oidc_endpoint: https://<domain_name>/keycloak/realms/kubeops-dashboards, scopes openid,profile,email, oidc_auto_onboard: true.

Full procedure: SSO for Harbor.

Rook-Ceph

Rook-Ceph uses an oauth2-proxy in front of the Ceph dashboard:

1. In Keycloak create a confidential client for the Ceph dashboard (redirect URI https://<domain_name>/oauth2/callback), copy the client secret, and generate a cookie secret.
2. Create the oauth2-proxy-credentials secret and install the oauth2-proxy Helm chart in the rook-ceph namespace.
3. Enable external auth on the Ceph manager (ceph config-key set mgr/dashboard/external_auth ...), add the oauth2 auth annotations to the ceph-dashboard ingress, and create the oauth2-proxy ingress.

Full procedure: SSO for rook-ceph.

12. Verify the cluster

kubectl get nodes -o wide                 # all nodes Ready
kubectl get pods -A                       # control-plane, networking and platform pods Running
kubectl get storageclass                  # rook-cephfs present
kubectl get clusterissuer                 # letsencrypt issuer (Scenario A)
kubectl get certificate -A                # certificates Ready
kubectl get ingress -A                    # ingress objects for the dashboards

Confirm every node is Ready, all component pods are Running, certificates are Ready, and the dashboards answer over HTTPS with a trusted certificate.

13. Access the dashboards

Reachable through the ingress under your domain (replace <domain_name>):

For kubectl via OIDC, add the Keycloak provider to your kubeconfig (issuer URL, client id, a tool such as kubelogin) and bind Keycloak users/groups to least-privilege RBAC roles.

14. Back up the cluster (Velero)

Velero is part of the stack. It stores backups in an S3-compatible object store and can also back up persistent-volume data.

Point Velero at an external S3 store

The velero package’s standard keys map directly to Velero’s credentials and BackupStorageLocation. Set them in enterprise-values.yaml:

• AWS S3
• S3-compatible (MinIO / Ceph-RGW)

- name: velero
  values:
    standard:
      provider: "aws"
      bucket: "<your-bucket>"                 # must already exist
      accessKeyId: "<aws access key>"
      secretAccessKey: "<aws secret key>"
      backupLocationConfig:
        region: "<aws-region>"
        s3ForcePathStyle: false        # no s3Url for real AWS
      useNodeAgent: true
      useVolumeSnapshots: true

- name: velero
  values:
    standard:
      provider: "aws"                          # the AWS plugin also drives S3-compatible stores
      bucket: "<your-bucket>"                  # must already exist on the external store
      accessKeyId: "<access key>"
      secretAccessKey: "<secret key>"
      backupLocationConfig:
        region: "default"
        s3ForcePathStyle: true
        s3Url: "https://s3.example.com"
      useNodeAgent: true
      useVolumeSnapshots: true

The credentials become the Velero cloud-credentials secret automatically.

How backups work

• Scheduled — the daily schedule from the advanced.schedules block (section 7.1) runs automatically and each backup expires after its ttl (30 days in the example).
• On demand — velero backup create <name> (add --include-namespaces / --selector to scope it).
• What is captured — Kubernetes objects always; persistent-volume data as well, because useNodeAgent: true and configuration.defaultVolumesToFsBackup: true are set. Stateful components (Keycloak, Harbor Postgres) additionally dump their database to /backup/*.dump via a pre-backup hook, so the dump travels inside the volume backup.
• Inspect — velero backup get, velero backup describe <name> --details, velero backup logs <name>.

How restore (rollback) works

velero backup get                                          # list available backups
velero restore create --from-backup <BACKUP or SCHEDULE-TIMESTAMP>          # full restore
velero restore create --from-backup <BACKUP> --include-namespaces <ns>      # selective restore
velero restore describe <restore-name>                     # check status

• Rollback a namespace/app by restoring the relevant backup; Velero does not overwrite resources that still exist, so restore into a clean namespace.
• Disaster recovery (whole cluster) — build a fresh cluster, install KubeOps with the same external S3 credentials/bucket in the velero package, then velero backup get already lists your existing backups and you restore with velero restore create --from-backup <name>.
• Databases — after the namespace is restored, run the app-specific restore from the dumped file (e.g. Keycloak/Harbor pg_restore from /backup/*.dump).
• The rook-cephfs storage class must exist in the target cluster before restoring PVCs.

Per-component backup/restore commands (Harbor, Prometheus, the logging stack, OPA, Rook-Ceph …) with their exact namespace/label selectors, and the database-restore steps, are in the Backup and Restore guide.

15. Production operations notes

Network isolation with a default-deny NetworkPolicy

Every enterprise tool installs its own NetworkPolicy (applied directly by the installer, not through the Helm chart) that allows exactly the traffic that component needs — including DNS egress to kube-system / kube-dns. These allow-policies exist precisely so the platform keeps working once a deny-all policy is in place. Rook-Ceph ships an allow-all policy for its own namespace.

KubeOps does not apply a cluster-wide deny-all itself. For production we recommend adding a default-deny NetworkPolicy in every namespace so that traffic is blocked unless a policy explicitly allows it:

# default-deny-all.yaml — the namespace is taken from `kubectl -n` on apply
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

NetworkPolicy is namespaced, so roll it out across all namespaces, e.g.:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n "$ns" apply -f default-deny-all.yaml
done

Further operational notes

• Inter-node encryption: with Calico as the CNI, pod-to-pod traffic between nodes is encrypted with WireGuard (UDP 51820 must stay open between nodes — see section 2).
• API audit logging: kubeopsctl enables Kubernetes API audit logging at level: Metadata, written to /var/log/kubernetes/audit/ on the control-plane nodes. For stricter compliance needs, replace it with a richer audit policy and forward the logs into your logging stack.
• Monitoring: use Prometheus/Grafana with alerting; watch certificate expiry and Ceph capacity.
• Updates in tranches: drain and update one zone at a time; keep at least one control plane on the higher version during an upgrade.
• Keep packages pinned: keep the OS pinned and all KOSI-installed packages held at their validated versions (see Installed Package Versions for RHEL / Ubuntu) so an OS update cannot upgrade podman, containerd or the Kubernetes tools out from under the cluster.
• Rotate credentials for the platform components and keep them in secrets.

2 - Supported Package Versions — Ubuntu / Debian

This page lists the packages and versions available from the KubeOps package repository for Ubuntu 24.04 (the .deb pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Kubernetes

KubeOps tools

Helm

Container runtime and etcd

Pre-release versions

Pin the installed versions

After the packages are installed, hold them at the installed version so a later apt upgrade cannot upgrade them and break the cluster:

sudo apt-mark hold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

To release a hold later (for a controlled upgrade):

sudo apt-mark unhold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

3 - Supported Package Versions — Red Hat Enterprise Linux (RHEL)

This page lists the packages and versions available from the KubeOps package repository for RHEL 9.6 / 9.7 (the .rpm pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Kubernetes

Container runtime and etcd

KubeOps tools

Helm

Pre-release versions

Pin the installed versions

On RHEL the KubeOps package repository is configured so that its packages are excluded from normal dnf transactions — the install commands have to pass --disableexcludes=kubeops-repo to override that exclusion. As a result a regular dnf update already leaves these packages untouched, so the cluster is protected out of the box.

To lock the installed versions explicitly, use the versionlock plugin after the packages are installed:

sudo dnf install -y python3-dnf-plugin-versionlock
sudo dnf versionlock add \
  containerd.io podman kubelet kubeadm kubectl kubernetes-cni cri-tools \
  etcdctl etcdutl helm kosi kubeopsctl conntrack-tools

Alternatively, exclude them explicitly on each update:

sudo dnf update --exclude=containerd.io,podman,kubelet,kubeadm,kubectl,kubernetes-cni,cri-tools,etcdctl,etcdutl,helm,kosi,kubeopsctl

Dependencies

The repository also provides the following compatibility/dependency packages (32-bit i686 runtime libraries), which are pulled in automatically: conntrack-tools (1.4.7), glibc (2.34), glibc-gconv-extra (2.34), libgcc (11.5.0), libmnl (1.0.4), libnetfilter_cthelper (1.0.0), libnetfilter_cttimeout (1.0.0), libnetfilter_queue (1.0.5), libnfnetlink (1.0.1).