KubeOpsDoku – Documentation for production

Kubeopsctl-2.0: How-To: Production Cluster with OIDC (Public IP & Airgap)

Mon, 01 Jan 0001 00:00:00 +0000

How-To: Production Cluster with OIDC (Public IP & Airgap)

This how-to walks a customer through building a production-ready KubeOps Compliance environment from clean machines to a running, fully equipped, authenticated cluster: the complete platform stack (storage, registry, monitoring, logging, policy, dashboard, backup) with Keycloak providing OIDC authentication, and the dashboards reachable over HTTPS with valid certificates.

It covers the combinations a customer realistically needs:

Network path — internet-connected (with public IPs) or airgap (KubeOps Compliance’s core strength).
Certificates — automatic Let’s Encrypt (public IPs) or your own CA (private DNS / airgap).
Ingress — nginx (default) or Traefik, including how to migrate.

kubeopsctl uses two configuration files, and it is important to keep them apart:

cluster-values.yaml — defines the cluster itself (nodes, networking, virtual IP). It has no influence over which applications get installed.
enterprise-values.yaml — defines the platform applications as a list of packages.

Important OIDC authentication for the dashboards requires kubeopsctl 2.1.0 or newer. On older versions the OIDC integration described here is not available.

⚠ Warning This guide assumes Ubuntu 24.04 or RHEL 9.6 on all machines. For OpenSUSE MicroOS the installation flow differs — follow Prepare Cluster (MicroOS) and Setup Cluster (MicroOS) and apply the production adaptations from this guide.

Important Build the environment as a normal user, not as root. Every step is performed by an unprivileged user who has sudo rights through the sudoers file distributed in section 3.5. The same username (and password) must exist on every machine.

Overview: what you will build

A highly available cluster with the minimum supported production topology and the full platform stack:

1 admin node, 3 control-plane nodes, 3 worker nodes (each worker with extra disks for storage), across three zones
a KubeVip virtual IP as the stable, highly available API endpoint
the full stack via enterprise-values.yaml: Rook-Ceph (storage), Keycloak (OIDC), ingress-nginx or Traefik, cert-manager (TLS), Harbor (registry), Prometheus/Grafana (monitoring), OpenSearch + Logstash + Filebeat (logging), OPA Gatekeeper (policy), KubeOps Dashboard, Velero (backup)
dashboards reachable via a domain name with valid certificates and Keycloak SSO

Choose your path

Two decisions shape the rest of this guide. Pick the row that matches your environment:

Environment	Certificates	Deployment	Notes
Internet + public IP	Let’s Encrypt (automatic)	single-phase	cert-manager issues valid certs automatically
Internet, private DNS / no public IP	own CA (or LE via DNS-01)	two-phase	provide a valid cert manually first
Airgap	own CA (no public ACME reachable)	two-phase	packages served from a local Harbor registry

Why deployment differs:

Let’s Encrypt (single-phase): cert-manager automatically creates a letsencrypt ClusterIssuer, and every ingress references it, so valid, browser-trusted certificates are issued as the components come up. Everything can be installed in one apply.
Own CA (two-phase): the OIDC dashboard integration needs a valid certificate to already exist when it runs. If it runs against a self-signed/placeholder certificate, it falls back to a dummy secret and breaks. So you first bring up storage, registry, ingress and cert-manager, install your valid certificate manually, and only then install Keycloak and the rest.

The ingress controller (nginx or Traefik) is an independent choice — see section 8.

1. Prerequisites

At least seven machines: one admin, three control-plane, three worker.

OS	Minimum requirements
Red Hat Enterprise Linux 9.6 / 9.7	8 CPU cores, 16 GB memory, 50 GB disk
Ubuntu 24.04.02	8 CPU cores, 16 GB memory, 50 GB disk

Important These are the documented minimum. The full platform stack (Ceph, Harbor, OpenSearch, Prometheus) is resource- and storage-intensive — size worker nodes well above the minimum.

Note Only Ubuntu 24.04 and Red Hat Enterprise Linux 9.6 / 9.7 are supported. The node preparation detects the OS by its exact name, so RHEL derivatives (Rocky Linux, AlmaLinux, CentOS Stream) are not recognized and are unsupported.

Before you start:

kubeopsctl 2.1.0 or newer (required for OIDC).
Same username and password on every machine (cluster creation fails otherwise).
Unique, lowercase hostname per machine.
OS version pinned to a supported release.
Network access: for the internet path, the cluster nodes reach the KubeOps registry directly. For the airgap path, only the admin node needs temporary outbound access to https://dispatcher.kubeops.net (KOSI hub) and https://registry.kubeops.net while kosi / kubeopsctl run; the cluster nodes stay isolated and pull all images from the Harbor that the platform stack deploys (section 5.1). KubeOps supports secure registries only.
Storage disks for Rook-Ceph: every persistent component uses the rook-cephfs storage class, which is 3× replicated across the workers — so raw disk ≈ 3 × the sum of all PVC claims. For the full platform stack with 30-day retention, provide one dedicated, unformatted disk of at least ~300 GB per worker (500 GB recommended); see the sizing table in section 7.1. Also see the Rook-Ceph prerequisites.
yq on all control-plane nodes (required for the OIDC dashboard integration) — yq install.
Passwords prepared for Harbor (and its Postgres/Redis), Keycloak (admin + Postgres) and Grafana — never the documented defaults.
DNS: for OIDC you need a DNS name for Keycloak and the dashboards. For Let’s Encrypt this must be public DNS; for an own CA it can be internal DNS.

2. Plan addressing and ports

Example node plan (203.0.113.x are placeholders from the documentation range, RFC 5737 — replace with your real IPs):

Hostname	Role	Zone	IP
`admin`	Admin / management	–	`203.0.113.10`
`controlplane01`	Control plane	zone1	`203.0.113.11`
`controlplane02`	Control plane	zone2	`203.0.113.12`
`controlplane03`	Control plane	zone3	`203.0.113.13`
`worker01`	Worker	zone1	`203.0.113.21`
`worker02`	Worker	zone2	`203.0.113.22`
`worker03`	Worker	zone3	`203.0.113.23`
–	KubeVip virtual IP (API)	–	`203.0.113.100`

Note The KubeVip virtual IP must be unused and in the same network as the control-plane nodes; KubeVip announces it via ARP. Many cloud/hosting providers route public IPs individually rather than in a shared L2 segment — there, ARP-based KubeVip does not work and you need BGP mode or a provider load balancer. As an alternative to KubeVip, KubeOps can front the API servers with an HAProxy load balancer (KubeOps API port 7443).

Default service ports:

Service	Port / NodePort
Kubernetes API (via KubeVip)	`6443`
Ingress (HTTP / HTTPS)	`80` / `443`
Harbor	`30002`
Keycloak	`30180`
Grafana	`30211`
Prometheus	`32090`
OpenSearch Dashboards	`30050`
KubeOps Dashboard	`30007`
SSH	`22`

Ports that must stay open between the cluster nodes (opened automatically by the KubeOps firewall setup):

Purpose	Port(s)
Kubernetes API server	`6443`
etcd	`2379`, `2380`
kubelet	`10250`
kube-proxy	`10256`
kube-scheduler / controller-manager	`10257`, `10259`
NodePort range	`30000–32767`
Calico BGP / VXLAN / Typha	`179`, `4789/udp`, `5473`
Calico WireGuard (pod-to-pod encryption)	`51820/udp`
Local registries (airgap)	`5000`, `5001`
HAProxy API load balancer	`7443`
Rook-Ceph (mon / mgr / OSD)	`3300`, `6789`, `6800–7568`
SSH	`22`

⚠ Warning KubeOps configures the node firewall automatically (firewall: nftables, the default; ufw on Ubuntu). The generated nftables ruleset uses policy accept and only counts/allows the ports above — it is an allow-list annotation, not a default-deny firewall. On RHEL, firewalld is stopped and masked in favor of nftables. Inter-node traffic must stay open between the nodes but must not be exposed to the internet. For the Let’s Encrypt HTTP-01 challenge, inbound port 80 must be reachable from the internet. For production, restrict the dashboard NodePorts to trusted networks and consider a default-deny policy in a firewall in front of the nodes.

3. Prepare all nodes

Run on every node unless noted.

3.1 Add the package repository (internet path)

wget https://packagerepo.kubeops.net/pgp-key.public
cat pgp-key.public | sudo gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/kubeops.gpg] https://packagerepo.kubeops.net/deb stable main' | sudo tee /etc/apt/sources.list.d/kubeops.list
sudo apt update

sudo dnf config-manager --add-repo https://packagerepo.kubeops.net/rpm/kubeops.repo

Note Airgap: the KOSI packages are pulled from the KOSI hub (https://dispatcher.kubeops.net) and the container images from https://registry.kubeops.net. The admin node needs outbound access to both only while kosi / kubeopsctl run (pulling packages and seeding images) — not permanently. The admin node is the bridge that stages the packages and seeds the platform-stack images into your local Harbor (section 5.1), while the cluster nodes stay isolated. On the kubeopsctl side, airgap mode is simply airgap: true in cluster-values.yaml (section 6).

3.2 Operating system adaptations

sudo apt remove unattended-upgrades

# no special adaptations required

3.3 Make hostnames resolvable

Use DNS for all node hostnames, or add them to /etc/hosts on every node:

sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
203.0.113.10 admin
203.0.113.11 controlplane01
203.0.113.12 controlplane02
203.0.113.13 controlplane03
203.0.113.21 worker01
203.0.113.22 worker02
203.0.113.23 worker03
EOL_ETC_HOSTS

3.4 Distribute SSH keys

Passwordless SSH must work from each node to every other node. Root login is temporarily enabled and reverted in section 4.

⚠ Warning This temporarily enables root login and password authentication. It must be reversed before the cluster goes live.

printf 'PermitRootLogin yes\nPasswordAuthentication yes\n' | sudo tee /etc/ssh/sshd_config.d/permit_root_login.conf
sudo systemctl restart sshd

# create keys on all nodes
ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N ""

# copy the public key, scan host keys, and test login — for EVERY node
for host in admin controlplane01 controlplane02 controlplane03 worker01 worker02 worker03; do
  ssh-copy-id -i ~/.ssh/id_ed25519 "$host"
  ssh-keyscan "$host" >> ~/.ssh/known_hosts
  ssh "$host" exit
done
# repeat ssh-keyscan with the IP addresses as well

3.5 Distribute the sudoers file

The cluster is built and operated by the unprivileged user from the warning at the top of this guide — not as root. That user needs the elevated rights kubeopsctl requires, granted through a sudoers file. Replace myuser with your username and copy the file to /etc/sudoers.d/<username> on all nodes.

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/apt/sources.list.d/kubeops.list
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove unattended-upgrades
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove *
myuser ALL=(root) NOPASSWD: /usr/bin/lsof /var/lib/dpkg/lock*

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kosi*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kubeopsctl*.deb


# kube-vip image import
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# Kubeadm init
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# enable UFW
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reset
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 6443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 2379\:2380/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10250/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10259/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10257/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10256/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 30000\:32767/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 179/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 4789/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5473/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 51820/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 22/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5000/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5001/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 7443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw logging low
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw enable
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reload
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw status
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart systemd-networkd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable --now ufw

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# kubeadm init phase and kubeadm token create commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /usr/bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# upgrade nodes
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin
myuser ALL=(root) NOPASSWD: /bin/rm -f /etc/cni/net.d/87-podman-bridge.conflist

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubeadm*
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubectl*
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubelet*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark hold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark unhold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubelet* kubeadm* kubectl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeadm*

# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install passt_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conmon_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install catatonit_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install netavark_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install aardvark-dns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-image_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-common_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containernetworking-plugins_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libsubid4_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install uidmap_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libslirp0_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install slirp4netns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libyajl2_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install crun_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install fuse-overlayfs_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install buildah_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install podman_*.deb

# Podman Remove of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove podman buildah fuse-overlayfs crun libyajl2 slirp4netns libslirp0 uidmap libsubid4 containernetworking-plugins golang-github-containers-common golang-github-containers-image aardvark-dns netavark catatonit conmon passt

# Podman Installation & Update with apt
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install */pia/kosi_*

# runtime setup
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conntrack_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install runc_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containerd_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y iproute-tc

# Remove packages
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove containerd

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard_*.deb wireguard-tools_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard-tools-*.deb systemd-resolved-*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove wireguard wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y wireguard-tools*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y systemd-resolved*

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/local/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf config-manager --add-repo https\://packagerepo.kubeops.net/rpm/kubeops.repo
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kosi*.rpm, !/usr/bin/rpm --install kosi*[[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kubeopsctl*.rpm, !/usr/bin/rpm --install kubeopsctl*[[\:space\:]]*.rpm

# kube-vip/create-manifest.sh.tmpl
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# delete-node/delete-node.sh.tmpl
# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# kubeadm-init/kubeadm-init.sh.tmpl
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/sysconfig/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm
# if Helm is in repository
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y helm* 

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s/package.kosi
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# ssh remote kubeadm commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# local execution of kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
# Adjustments sed -i * 
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# kubeadm upgrade node
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin, !/bin/tar xzf cni-*[[\:space\:]]* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin, !/bin/tar xzf crictl-*[[\:space\:]]* -C /usr/bin
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubeadm*, !/usr/bin/dnf install -y kubeadm*[[\:space\:]]*, !/usr/bin/dnf install -y kubeadm*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*, !/bin/mv /usr/bin/kubelet /usr/bin/kubelet_*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubectl*, !/usr/bin/dnf install -y kubectl*[[\:space\:]]*, !/usr/bin/dnf install -y kubectl*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubelet*, !/usr/bin/dnf install -y kubelet*[[\:space\:]]*, !/usr/bin/dnf install -y kubelet*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubelet-* kubeadm-* kubectl-*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeadm-*


# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install aardvark-dns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install netavark-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install container-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libnet-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-libs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libslirp-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install slirp4netns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install yajl-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install crun-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containers-common-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install fuse-overlayfs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install shadow-utils-subid-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conmon-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install podman-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-*.rpm iproute-tc-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containerd.io-*.rpm

# Podman Remove of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase podman shadow-utils-subid fuse-overlayfs crun containers-common yajl slirp4netns libslirp criu-libs criu libnet container-selinux netavark aardvark-dns passt passt-selinux

# Podman Installation & Update with dnf
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y podman*, !/usr/bin/dnf install -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf update -y podman*,!/usr/bin/dnf update -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install */pia/kosi-*

# Allow installation of container-runtime
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm, !/usr/bin/rpm --install conntrack-tools-*[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm libnetfilter_cthelper-*.rpm libnetfilter_cttimeout-*.rpm libnetfilter_queue-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-tc-*.rpm, !/usr/bin/rpm --install iproute-tc-*[\:space\:]]*.rpm

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y iproute-tc

# Remove RPM packages
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase containerd.io

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/systemd/system/containerd.service.d/
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/systemd/system/containerd.service.d/override.conf*

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install wireguard-tools-*.rpm systemd-resolved-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y wireguard-tools*, !/usr/bin/dnf install -y wireguard-tools*[\:space\:]]*, !/usr/bin/dnf install -y wireguard-tools*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y systemd-resolved*, !/usr/bin/dnf install -y systemd-resolved*[\:space\:]]*, !/usr/bin/dnf install -y systemd-resolved*.rpm

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

3.6 Install kubeadm

sudo apt install -y kubeadm=<kubernetes-version>*

sudo dnf install -y --disableexcludes=kubeops-repo kubeadm-<kubernetes-version>

3.7 Configure time synchronization

Important Accurate time is critical for OIDC: tokens are time-limited JWTs, and clock skew between nodes and Keycloak causes authentication to fail intermittently.

sudo apt install -y chrony
sudo systemctl enable --now chrony

sudo dnf install -y chrony
sudo systemctl enable --now chronyd

Configure NTP servers in /etc/chrony.conf, sudo systemctl restart chronyd, then verify with chronyc tracking (leap status Normal, offset < 10 ms).

3.8 Install curl and yq

Install curl on every node, and yq on all control-plane nodes (required for the OIDC dashboard integration) per the yq installation guide.

4. Harden the nodes for production

Revert the SSH change from section 3.4 on every node:

sudo rm /etc/ssh/sshd_config.d/permit_root_login.conf
sudo systemctl restart sshd

⚠ Warning Leaving root login enabled on publicly reachable nodes is a serious security risk. Confirm key-based SSH still works after reverting.

Also: pin the OS version and tighten the firewall per section 2.

Pin the KOSI-installed packages. kubeopsctl installs a set of system packages (podman, containerd, the Kubernetes tools, etc.) at specific, validated versions. An OS update (apt upgrade / dnf update) can silently upgrade them and break the cluster. After section 6, pin/hold every one of these packages to the installed version so OS updates cannot move them.

Ubuntu: sudo apt-mark hold <package> (e.g. podman, containerd, kubelet, kubeadm, kubectl)
RHEL: exclude them from updates, e.g. sudo dnf update --exclude=podman,containerd,kubelet,kubeadm,kubectl, or use the versionlock plugin

Note The exact package list and the validated versions for your release are documented on the separate Installed Package Versions pages for RHEL and Ubuntu / Debian. Pin all of them, not just podman.

5. Set up the admin tooling

Perform the rest from the admin node.

Install KOSI, then set KUBEOPSROOT, adjust the KOSI config, install the enterprise plugins, log in, and install kubeopsctl (≥ 2.1.0) — exactly as in the Setup Cluster guide:

# KOSI
sudo apt install -y kosi=<kosi-version>          # Ubuntu  | RHEL: sudo dnf install -y --disableexcludes=kubeops-repo <kosi-version>

# KUBEOPSROOT (append to ~/.bashrc, then: source ~/.bashrc)
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

# KOSI config + plugins
mkdir ~/kubeops && cd ~/kubeops
cp -R /var/kubeops/kosi/ . && cp -R /var/kubeops/plugins/ .
# edit $KUBEOPSROOT/kosi/config.yaml: set hub and the FULL plugins path

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.0.0
kosi login -u <yourUser>                          # a "registry temporarily not available" error here is expected

# kubeopsctl (>= 2.1.0 for OIDC)
sudo apt install -y kubeopsctl=<kubeopsctl-version>   # Ubuntu | RHEL: sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

5.1 (Airgap only) Stage the images in your local Harbor

In an airgapped environment the platform-stack images are served from your local Harbor registry. The flow is:

rook and harbor first. Only rook-ceph and harbor ship dedicated airgap packages that carry their own container images. Install these two first so storage and the registry exist.
Seed the remaining images. Once Harbor is up, kubeopsctl pulls every other tool’s images from registry.kubeops.net, pushes them into the deployed Harbor, and re-tags each reference to 127.0.0.1:30002/kubeops/<image> — Harbor reached over its NodePort 30002 on each node, project kubeops.
Private kubeops project. The kubeops Harbor project is private, so every tool is deployed with an image pull secret for it.
Point the config at local. Set airgap: true and packageRepository: local in cluster-values.yaml (section 6), and localRegistry: true in enterprise-values.yaml (section 7).

Note KubeOps supports secure registries only — Harbor must be reachable with a certificate your nodes trust (section 9).

6. Create the cluster

The cluster-values.yaml describes only the cluster. The airgap flag is the key difference between the two network paths.

⚠ Warning This file has to be adapted to your values before usage.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: true
virtualIP: <unused IP in the control-plane network>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: false
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true
zones:
- name: zone1
  nodes:
  - { name: <controlplane01 hostname>, iPAddress: <controlplane01 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker01 hostname>,       iPAddress: <worker01 ip>,       type: worker,       kubeVersion: <version> }
- name: zone2
  nodes:
  - { name: <controlplane02 hostname>, iPAddress: <controlplane02 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker02 hostname>,       iPAddress: <worker02 ip>,       type: worker,       kubeVersion: <version> }
- name: zone3
  nodes:
  - { name: <controlplane03 hostname>, iPAddress: <controlplane03 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker03 hostname>,       iPAddress: <worker03 ip>,       type: worker,       kubeVersion: <version> }

# file cluster-values.yaml  (airgap)
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: <your local Harbor>/kubeops    # your local secure registry
airgap: true
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: true
virtualIP: <unused IP in the control-plane network>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: false
packageRepository: local
changeCluster: true
zones:
  # ... same zone layout as the Internet tab ...

Then pull packages, install the runtime tooling, and create the cluster:

kubeopsctl pull                                                        # add --kubernetesVersion <x.xx.x> to pin a version
kosi install -p $KUBEOPSROOT/lima/podman_<version>.tgz -f cluster-values.yaml
kosi install -p $KUBEOPSROOT/lima/helm_<version>.tgz
kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml
kosi login -u <your username>
kubeopsctl apply -f cluster-values.yaml                                # creates the cluster (~10–15 min)

⚠ Warning Now pin all KOSI-installed packages (podman, containerd, the Kubernetes tools, …) to their installed versions so a later OS update cannot upgrade them — see section 4 and the Installed Package Versions pages (RHEL / Ubuntu). The package versions above (podman, helm) are examples — check the repository for the versions shipped with your release; the kubernetes-tools version must match the version you pulled.

7. Configure the platform stack

The applications live in enterprise-values.yaml. Each package has a standard block (predefined keys) and an advanced block (any value from the underlying Helm chart). Standard values overwrite advanced values if both are set.

The full reference below enables the whole stack and is wired for OIDC dashboard access (each dashboard gets a hostname). It is not applied as-is in one step for every scenario — the ingress choice (section 8), the certificate scenario (section 9) and the deployment (section 10) determine how.

⚠ Warning Replace every <domain_name>, every placeholder password and the S3 credentials before applying. Airgap: set localRegistry: true.

Full enterprise-values.yaml (all components)

# file enterprise-values.yaml
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false            # airgap: true
packages:

# Storage — provides the rook-cephfs storage class used by everything below
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      hostname: <domain_name>
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
    advanced:
      # disk selection for Ceph goes here (rook-ceph-cluster Helm values),
      # e.g. storage.useAllDevices / storage.deviceFilter / per-node devices
      cluster:
      operator:

# Ingress controller — externalIPs binds it to your IP(s). For Traefik see section 8.
- name: ingress-nginx
  enabled: true
  values:
    standard:
      namespace: ingress-nginx
      externalIPs:
      - <ingress ip>             # public IP (Scenario A) or internal IP (Scenario B)
    advanced:

# Certificates
- name: cert-manager
  enabled: true
  values:
    standard:
      namespace: cert-manager
      replicaCount: 3
      logLevel: 2
      # Scenario B (own CA): point cert-manager at your own certificate secret
      # secretName: example-ca
    ca:
      # Scenario A (Let's Encrypt): cert-manager auto-creates a "letsencrypt" ClusterIssuer
      emailLetsEncrypt: <your_email@domain.com>   # MUST set (default example@example.com will not work)
      ingressName: nginx                          # MUST match the ingress controller (nginx or traefik)

# OIDC provider — Keycloak
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: keycloak
      storageClass: "rook-cephfs"
      nodePort: "30180"
      hostname: <domain_name>
      keycloak:
        auth:
          adminUser: admin
          adminPassword: "<strong password>"
          existingSecret: ""
      postgresql:
        auth:
          postgresPassword: "<strong password>"
          username: bn_keycloak
          password: "<strong password>"
          database: bitnami_keycloak
    advanced:

# Registry (also the local registry in airgap)
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "<strong password>"
      databasePassword: "<strong password>"
      redisPassword: "<strong password>"
      externalURL: https://<domain_name>
      nodePort: 30002
      hostname: <domain_name>
      harborPersistence:
        persistentVolumeClaim:
          registry:                                # full-stack images ~5 GB; rest = headroom for your own images — enable Harbor GC + a project quota (see section 7.1)
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:                                # also holds Trivy scan reports
            size: 5Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy:
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:

# Monitoring
- name: kube-prometheus-stack
  enabled: true
  values:
    standard:
      namespace: monitoring
      grafanaUsername: admin
      grafanaPassword: "<strong password>"
      grafanaResources:
        hostname: <domain_name>
        nodePort: 30211
        storageClass: "rook-cephfs"
        storage: 10Gi
      prometheusResources:
        hostname: <domain_name>
        nodePort: 32090
        storageClass: "rook-cephfs"
        storage: 50Gi
        retention: 30d
        retentionSize: "45GB"
    advanced:
      # Alertmanager ships with no persistent storage (emptyDir) — give it a PVC so silences/state survive restarts
      alertmanager:
        alertmanagerSpec:
          storage:
            volumeClaimTemplate:
              spec:
                storageClassName: rook-cephfs
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 2Gi

# Logging stack
- name: opensearch-os
  enabled: true
  values:
    standard:
      namespace: logging
      hostname: <domain_name>
      opensearchJavaOpts: "-Xmx512M -Xms512M"
      resources:
        requests:
          cpu: "250m"
          memory: "1024Mi"
        limits:
          cpu: "300m"
          memory: "3072Mi"
      persistence:                                 # size for your chosen log retention (see section 7.1)
        size: 40Gi
        storageClass: "rook-cephfs"
        accessModes: ["ReadWriteMany"]
      securityConfig:
        enabled: false
      replicas: "3"
    advanced:
      # Log retention: no lifecycle policy ships by default, so log indices grow UNBOUNDED.
      # This Job installs an ISM policy that deletes log indices older than the configured age and
      # re-runs idempotently on every `kubeopsctl apply` (upgrade-safe). See section 7.1.
      extraObjects:
      - apiVersion: batch/v1
        kind: Job
        metadata:
          name: kubeops-log-retention
          namespace: logging
        spec:
          ttlSecondsAfterFinished: 300
          backoffLimit: 6
          template:
            spec:
              restartPolicy: OnFailure
              containers:
              - name: apply-ism
                image: curlimages/curl:8.9.1        # airgap: reference this image from your local Harbor
                command: ["/bin/sh","-c"]
                args:
                - |
                  # OpenSearch endpoint — http + no auth matches securityConfig.enabled=false above;
                  # if you enable securityConfig, switch to https and pass admin credentials.
                  OS="http://opensearch-cluster-master.logging.svc:9200"
                  # >>> RETENTION: change "30d" to "60d", "90d", ... as needed <<<
                  cat >/tmp/policy.json <<'JSON'
                  {"policy":{"description":"Delete KubeOps log indices","default_state":"hot",
                   "states":[
                     {"name":"hot","actions":[],"transitions":[{"state_name":"delete","conditions":{"min_index_age":"30d"}}]},
                     {"name":"delete","actions":[{"delete":{}}],"transitions":[]}],
                   "ism_template":[{"index_patterns":["logstash-logs-*"],"priority":100}]}}
                  JSON
                  curl -sf -XPUT  "$OS/_plugins/_ism/policies/kubeops-log-retention" -H 'Content-Type: application/json' -d @/tmp/policy.json
                  curl -sf -XPOST "$OS/_plugins/_ism/add/logstash-logs-*"           -H 'Content-Type: application/json' -d '{"policy_id":"kubeops-log-retention"}' || true                  
- name: opensearch-dashboards
  enabled: true
  values:
    standard:
      namespace: logging
      hostname: <domain_name>
      nodePort: 30050
    advanced:
- name: logstash-os
  enabled: true
  values:
    standard:
      namespace: logging
      volumeClaimTemplate:
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 5Gi
        storageClass: "rook-cephfs"
    advanced:
      # Persistent queue so log bursts/backpressure are buffered on disk instead of dropped.
      # NOTE: this REPLACES the shipped logstash.yml — the first two lines are the existing defaults, keep them.
      logstashConfig:
        logstash.yml: |
          pipeline.ecs_compatibility: disabled
          log.level: info
          queue.type: persisted
          queue.max_bytes: 1gb          
- name: filebeat-os
  enabled: true
  values:
    standard:
      namespace: logging
    advanced:

# Policy
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:

# Cluster dashboard
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: <domain_name>
      service:
        nodePort: 30007
    advanced:

# Backup
- name: velero
  enabled: true
  values:
    standard:
      namespace: velero
      accessKeyId: "<s3 access key>"
      secretAccessKey: "<s3 secret key>"
      provider: "aws"
      bucket: "velero"
      useNodeAgent: true
      useVolumeSnapshots: true
      backupLocationConfig:                        # placeholder — point bucket/region/s3Url at an EXTERNAL S3 store (see section 14)
        region: "minio"
        s3ForcePathStyle: true
        s3Url: "http://minio.velero.svc:9000"
    advanced:
      # Out of the box Velero backs up only Kubernetes objects, on no schedule, deleted after 72h.
      # Add a recurring schedule and a retention TTL (720h = 30d):
      schedules:
        daily:
          schedule: "0 */6 * * *"
          template:
            ttl: "720h"
      # Back up PV data too. NOTE: the `standard` key defaultVolumesToFsBackup is emitted at top level
      # where this chart ignores it — it must live under `configuration:` (set it here in advanced).
      configuration:
        defaultVolumesToFsBackup: true

Secure the components: replace every password (or use existingSecret), set the public addresses (externalURL, hostname, ingress externalIPs), and restrict the dashboards/registry at the firewall to trusted networks.

7.1 Sizing, retention and backups

All persistent components use the rook-cephfs storage class, which is 3× replicated across the worker nodes (failureDomain: host, reclaimPolicy: Retain). Plan raw disk ≈ 3 × the sum of all PVC sizes and never fill Ceph beyond ~75–80 %.

Recommended minimal-but-stable sizes (7-node baseline, 30-day retention). These are the values used in the example above — adjust to your workload, since logging and monitoring volumes scale with the number of nodes/pods and with your retention.

Component	PVC (usable)	Notes
Harbor registry	40Gi	full-stack images ≈ 5 GB; the rest is headroom for your own images
Harbor database	5Gi	also holds Trivy scan reports
Harbor redis / trivy / jobLog	1 / 5 / 1 Gi
Keycloak Postgres	8Gi
Prometheus	50Gi	`retention: 30d`, `retentionSize: "45GB"`
Grafana	10Gi
Alertmanager	2Gi	default ships as emptyDir — add the PVC
OpenSearch (each of 3 nodes)	40Gi	scales with your log retention
Logstash	5Gi	persistent queue
Velero	external S3/MinIO	not on Ceph

The sum of usable PVCs is ≈ 215–250 Gi, so raw Ceph is ≈ 650–750 Gi; with headroom, plan ~900 Gi–1 TiB raw. Because the pools replicate across the three workers, provide a dedicated, unformatted disk of at least ~300 Gi per worker (recommended 500 Gi) for the Ceph OSDs. This supersedes the “50 GB disk per worker” figure in the prerequisites, which only covers an empty cluster.

Rule of thumb for logs (drives OpenSearch and Ceph): budget roughly 0.5–1 GB/day for a small cluster, × your retention days, × 2 for the index replica — and size the OpenSearch PVC to match.

Log retention (OpenSearch) — KubeOps ships no index-lifecycle policy, so log indices grow unbounded and eventually force OpenSearch read-only. The opensearch-os package above therefore carries an extraObjects Job that installs an ISM policy deleting logstash-logs-* indices older than a configurable age. Set your retention in that block via min_index_age ("30d" / "60d" / "90d"). The Job only installs the policy once; from then on OpenSearch’s ISM plugin enforces it continuously on its own — deleting aged indices and auto-attaching the policy to every new logstash-logs-* index via ism_template. So retention keeps working between releases (even months apart), not only during an apply. The Job re-runs on every kubeopsctl apply, so the policy is also restored after a fresh reinstall.
Metrics retention (Prometheus) — set in the standard block via prometheusResources.retention / .retentionSize (example: 30d / 45GB on a 50Gi PVC). Keep retentionSize a few GB below the PVC size for WAL/compaction headroom.
Harbor cleanup (GC) — the Harbor chart configures no garbage collection, so the registry PVC keeps growing as tags/images accumulate. Deploy the GC-schedule Job shown below; once applied, Harbor runs GC on that schedule itself. To cap the private kubeops project, set a storage quota in the UI under Projects → kubeops → Configuration → storage quota (API: PUT /api/v2.0/quotas/<id>).
Backups (Velero) — out of the box Velero captures only Kubernetes objects, on no schedule, deleted after 72h. The example enables the node agent and volume snapshots, adds a daily schedule with a 30-day ttl, and sets configuration.defaultVolumesToFsBackup: true so PV data is included — see section 14.

Harbor GC-schedule Job (apply once; Harbor then runs GC on the schedule itself):

# harbor-gc.yaml — apply with: kubectl -n harbor apply -f harbor-gc.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: kubeops-harbor-gc
  namespace: harbor
spec:
  ttlSecondsAfterFinished: 300
  backoffLimit: 10
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: harbor-gc
        image: curlimages/curl:8.9.1          # airgap: reference this image from your local Harbor
        command: ["/bin/sh","-c"]
        args:
        - |
          API="http://harbor-core.harbor.svc/api/v2.0"
          AUTH="-u admin:<harbor admin password>"
          for i in $(seq 1 60); do curl -s $AUTH "$API/health" >/dev/null 2>&1 && break; sleep 5; done
          # weekly GC (6-field cron), delete untagged artifacts — change the cron as needed
          BODY='{"parameters":{"delete_untagged":true},"schedule":{"type":"Custom","cron":"0 0 4 * * 0"}}'
          curl -s $AUTH -XPUT  "$API/system/gc/schedule" -H 'Content-Type: application/json' -d "$BODY" \
            || curl -s $AUTH -XPOST "$API/system/gc/schedule" -H 'Content-Type: application/json' -d "$BODY"

Important Upgrades: your enterprise-values.yaml is the source of truth — keep it under version control and re-run kubeopsctl apply to upgrade. Every standard and advanced value is re-applied to helm upgrade, so your sizing and retention settings are not overwritten. The OpenSearch ISM Job (shipped via extraObjects) re-runs automatically on each apply; the Harbor GC schedule lives in Harbor’s database and survives rolling upgrades. Both the ISM policy and the GC schedule are enforced continuously by OpenSearch/Harbor themselves between releases — you only re-apply them after a fresh reinstall (the GC Job with kubectl apply).

Note The retention (ISM) and Harbor-GC Jobs are deployed from this guide today. Native integration of these settings into the KubeOps packages (and the Velero defaultVolumesToFsBackup mapping) is planned for a future release; until then, configure them as shown here.

8. Choose the ingress controller

KubeOps supports nginx (default) and Traefik. Whichever you use, the cert-manager ingressName must match it.

Keep the ingress-nginx package enabled in enterprise-values.yaml (as in the full example) and set ingressName: nginx in cert-manager. Nothing else is needed.

Traefik is installed as its own KOSI package (not via the enterprise-values.yaml package list). Create a values.yaml:

# values.yaml
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik
      externalIPs: []        # set your ingress IP(s)
    advanced: {}

Install it (set ingressName: traefik in cert-manager):

# non-airgap
kosi search --hub kubeops --ps traefik
kosi install --hub kubeops kubeops/traefik:<version> -f values.yaml --dname traefik

# airgap
podman login <harbor ip>:<nodePort> -u <user> -p <pass> --tls-verify=false
kosi pull --hub kubeops kubeops/traefik:<version> -o traefik-<version>.tgz -r <harbor ip>:<nodePort>/kubeops -t localhost:<nodePort>/kubeops
kosi install --dname traefik -p traefik-<version>.tgz -f values.yaml

Migrating from nginx to Traefik: install Traefik (above), then remove the old nginx-ingress and, if you need to reuse its NodePorts, edit the Traefik service to match:

kosi list                                                              # find the nginx deployment name
kosi delete --hub kubeops kubeops/ingress-nginx:<version> -f enterprise-values.yaml --dname <name>
kubectl edit svc traefik -n traefik                                    # set web/websecure nodePorts (e.g. 30080/30443)

9. Certificates and external access

Dashboards reachable from the internet with valid, browser-trusted certificates.

Bind the ingress to your public IP (externalIPs in the ingress package / Traefik values.yaml).
Configure the issuer — cert-manager creates a letsencrypt ClusterIssuer automatically and every ingress references it; you only provide:

- name: cert-manager
  values:
    ca:
      emailLetsEncrypt: <your_email@domain.com>
      ingressName: nginx        # or traefik

Create public DNS A-records pointing <domain_name> at your ingress public IP.
Keep hostname: <domain_name> on each dashboard package.

Because valid certificates are issued automatically as the components come up, you can install everything in one apply (single-phase — section 10).

⚠ Warning

Let’s Encrypt: the HTTP-01 challenge needs inbound port 80 and public DNS. Let’s Encrypt enforces rate limits — test against the staging endpoint first. Wildcard certificates require DNS-01, not HTTP-01.

The cluster is reachable only inside your network, or has no public ACME access (airgap). You provide a valid certificate yourself.

Create a TLS secret from a certificate your clients already trust and point cert-manager at it:

# example only — use a certificate from your corporate CA in production
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt -subj "/CN=<your domain>"
kubectl create secret tls example-ca --key="tls.key" --cert="tls.crt" -n cert-manager

- name: cert-manager
  values:
    standard:
      secretName: example-ca

Distribute the CA certificate to all clients and to the kube-apiserver so it trusts the Keycloak OIDC issuer. Set externalIPs and the hostname fields to your internal IPs/DNS names.

Because the valid certificate must exist before the OIDC dashboard setup runs, this scenario uses the two-phase deployment (section 10).

10. Deploy the platform stack

With Let’s Encrypt, valid certificates are issued automatically, so the whole stack can be applied at once. Keep the packages in dependency order in enterprise-values.yaml — storage and the cert/ingress components are defined before Keycloak, which is defined before the dashboards — then:

kubeopsctl apply -f enterprise-values.yaml

Wait until all components are deployed and running.

Note

If your build does not resolve install order automatically, bring up rook-ceph (storage) and then keycloak before the dashboards — see the staged approach in the Scenario B tab.

The OIDC dashboard integration needs a valid certificate to already exist; otherwise it creates a dummy secret. Deploy in two phases, controlled via the enabled flag of each package.

Phase 1 — storage, registry, ingress, cert-manager. Enable only these (others enabled: false) and apply:

- name: rook-ceph      # enabled: true
- name: harbor         # enabled: true
- name: ingress-nginx  # enabled: true   (or install Traefik per section 8)
- name: cert-manager   # enabled: true

kubeopsctl apply -f enterprise-values.yaml
kubectl get storageclass      # rook-cephfs present

Phase 1.5 — install the valid certificate (manual). Create the TLS secret and set cert-manager secretName (section 9, Scenario B), re-apply phase 1, and verify the certificate is trusted:

curl -I https://<domain_name>/

⚠ Warning

Do not proceed until a valid certificate is in place. Installing Keycloak/the dashboards without it creates a dummy secret and the OIDC integration will not work.

Phase 2 — Keycloak and the rest. Enable keycloak (it must be ready before the dashboards) and the remaining packages, then apply again:

kubeopsctl apply -f enterprise-values.yaml

11. Single Sign-On for Harbor and Rook-Ceph

Most dashboards (Grafana, Prometheus, OpenSearch, KubeOps Dashboard) are wired to OIDC by the integration. Harbor and Rook-Ceph are configured separately and must be set up manually after Keycloak is running. Both use a Keycloak realm named kubeops-dashboards with a user kubeops.

Harbor

In Keycloak (kubeops-dashboards realm) create a confidential client harbor (standard flow on, direct access grants off) with redirect URI https://<domain_name>/c/oidc/callback and web origin <domain_name>; copy the client secret.
Create a secret with the client id/secret in the Harbor namespace.
In the Harbor package’s advanced values, inject the OIDC client id/secret as env vars and set CONFIG_OVERWRITE_JSON with auth_mode: oidc_auth, oidc_endpoint: https://<domain_name>/keycloak/realms/kubeops-dashboards, scopes openid,profile,email, oidc_auto_onboard: true.

Full procedure: SSO for Harbor.

Rook-Ceph

Rook-Ceph uses an oauth2-proxy in front of the Ceph dashboard:

In Keycloak create a confidential client for the Ceph dashboard (redirect URI https://<domain_name>/oauth2/callback), copy the client secret, and generate a cookie secret.
Create the oauth2-proxy-credentials secret and install the oauth2-proxy Helm chart in the rook-ceph namespace.
Enable external auth on the Ceph manager (ceph config-key set mgr/dashboard/external_auth ...), add the oauth2 auth annotations to the ceph-dashboard ingress, and create the oauth2-proxy ingress.

Full procedure: SSO for rook-ceph.

Note The externalURL/hostname must match the DNS name exactly, and the OIDC client id/secret in Keycloak must match the values you configure in Harbor and oauth2-proxy.

12. Verify the cluster

kubectl get nodes -o wide                 # all nodes Ready
kubectl get pods -A                       # control-plane, networking and platform pods Running
kubectl get storageclass                  # rook-cephfs present
kubectl get clusterissuer                 # letsencrypt issuer (Scenario A)
kubectl get certificate -A                # certificates Ready
kubectl get ingress -A                    # ingress objects for the dashboards

Confirm every node is Ready, all component pods are Running, certificates are Ready, and the dashboards answer over HTTPS with a trusted certificate.

13. Access the dashboards

Reachable through the ingress under your domain (replace <domain_name>):

Dashboard	Path
Harbor	`<domain_name>/` (base domain)
Grafana	`<domain_name>/grafana`
Rook-Ceph	`<domain_name>/ceph-dashboard`
OpenSearch	`<domain_name>/opensearch`
Keycloak	`<domain_name>/keycloak`
KubeOps Dashboard	`<domain_name>/kubeops-dashboard`
Prometheus	`<domain_name>/prometheus`

For kubectl via OIDC, add the Keycloak provider to your kubeconfig (issuer URL, client id, a tool such as kubelogin) and bind Keycloak users/groups to least-privilege RBAC roles.

14. Back up the cluster (Velero)

Velero is part of the stack. It stores backups in an S3-compatible object store and can also back up persistent-volume data.

Important Out of the box Velero backs up only Kubernetes objects, on no schedule, and deletes each backup after 72h (node agent and volume snapshots are off). For production, drive this from enterprise-values.yaml (see section 7.1) rather than ad-hoc CLI: set useNodeAgent: true and useVolumeSnapshots: true in the standard block, and in advanced add a recurring schedules entry with a 30-day ttl plus configuration.defaultVolumesToFsBackup: true so PV data is included. These settings are re-applied on every kubeopsctl apply, so they survive upgrades.

Point Velero at an external S3 store

⚠ Warning Store backups off-cluster. The shipped s3Url: http://minio.velero.svc:9000 is only a placeholder — a backup that lives inside the very cluster it protects is useless for disaster recovery. Use an external S3 bucket (or a MinIO/Ceph-RGW instance outside this cluster). The bucket must already exist.

The velero package’s standard keys map directly to Velero’s credentials and BackupStorageLocation. Set them in enterprise-values.yaml:

- name: velero
  values:
    standard:
      provider: "aws"
      bucket: "<your-bucket>"                 # must already exist
      accessKeyId: "<aws access key>"
      secretAccessKey: "<aws secret key>"
      backupLocationConfig:
        region: "<aws-region>"
        s3ForcePathStyle: false        # no s3Url for real AWS
      useNodeAgent: true
      useVolumeSnapshots: true

- name: velero
  values:
    standard:
      provider: "aws"                          # the AWS plugin also drives S3-compatible stores
      bucket: "<your-bucket>"                  # must already exist on the external store
      accessKeyId: "<access key>"
      secretAccessKey: "<secret key>"
      backupLocationConfig:
        region: "default"
        s3ForcePathStyle: true
        s3Url: "https://s3.example.com"
      useNodeAgent: true
      useVolumeSnapshots: true

The credentials become the Velero cloud-credentials secret automatically.

Note If the external endpoint uses a private CA, Velero needs its CA bundle — this is not exposed as a standard key. Use a publicly trusted certificate, or add caCert to the BackupStorageLocation manually (kubectl edit backupstoragelocation default -n velero); note the manual edit is not values-persistent.

How backups work

Scheduled — the daily schedule from the advanced.schedules block (section 7.1) runs automatically and each backup expires after its ttl (30 days in the example).
On demand — velero backup create <name> (add --include-namespaces / --selector to scope it).
What is captured — Kubernetes objects always; persistent-volume data as well, because useNodeAgent: true and configuration.defaultVolumesToFsBackup: true are set. Stateful components (Keycloak, Harbor Postgres) additionally dump their database to /backup/*.dump via a pre-backup hook, so the dump travels inside the volume backup.
Inspect — velero backup get, velero backup describe <name> --details, velero backup logs <name>.

How restore (rollback) works

velero backup get                                          # list available backups
velero restore create --from-backup <BACKUP or SCHEDULE-TIMESTAMP>          # full restore
velero restore create --from-backup <BACKUP> --include-namespaces <ns>      # selective restore
velero restore describe <restore-name>                     # check status

Rollback a namespace/app by restoring the relevant backup; Velero does not overwrite resources that still exist, so restore into a clean namespace.
Disaster recovery (whole cluster) — build a fresh cluster, install KubeOps with the same external S3 credentials/bucket in the velero package, then velero backup get already lists your existing backups and you restore with velero restore create --from-backup <name>.
Databases — after the namespace is restored, run the app-specific restore from the dumped file (e.g. Keycloak/Harbor pg_restore from /backup/*.dump).
The rook-cephfs storage class must exist in the target cluster before restoring PVCs.

Per-component backup/restore commands (Harbor, Prometheus, the logging stack, OPA, Rook-Ceph …) with their exact namespace/label selectors, and the database-restore steps, are in the Backup and Restore guide.

15. Production operations notes

Network isolation with a default-deny NetworkPolicy

Every enterprise tool installs its own NetworkPolicy (applied directly by the installer, not through the Helm chart) that allows exactly the traffic that component needs — including DNS egress to kube-system / kube-dns. These allow-policies exist precisely so the platform keeps working once a deny-all policy is in place. Rook-Ceph ships an allow-all policy for its own namespace.

KubeOps does not apply a cluster-wide deny-all itself. For production we recommend adding a default-deny NetworkPolicy in every namespace so that traffic is blocked unless a policy explicitly allows it:

# default-deny-all.yaml — the namespace is taken from `kubectl -n` on apply
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

NetworkPolicy is namespaced, so roll it out across all namespaces, e.g.:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n "$ns" apply -f default-deny-all.yaml
done

⚠ Warning

The tool-provided allow-policies are intentionally permissive (several allow egress to 0.0.0.0/0 on specific ports) — a deny-all NetworkPolicy complements, but does not replace, the node firewall (section 2).
The KubeOps Dashboard does not currently apply its own allow-policy, so under a strict deny-all you must add an allow-NetworkPolicy for its namespace manually (at least DNS egress and ingress from the ingress controller).
When you deploy your own workloads into a managed or new namespace, add explicit allow-NetworkPolicy rules (DNS, ingress controller, dependent services) or they cannot communicate.

Further operational notes

Inter-node encryption: with Calico as the CNI, pod-to-pod traffic between nodes is encrypted with WireGuard (UDP 51820 must stay open between nodes — see section 2).
API audit logging: kubeopsctl enables Kubernetes API audit logging at level: Metadata, written to /var/log/kubernetes/audit/ on the control-plane nodes. For stricter compliance needs, replace it with a richer audit policy and forward the logs into your logging stack.
Monitoring: use Prometheus/Grafana with alerting; watch certificate expiry and Ceph capacity.
Updates in tranches: drain and update one zone at a time; keep at least one control plane on the higher version during an upgrade.
Keep packages pinned: keep the OS pinned and all KOSI-installed packages held at their validated versions (see Installed Package Versions for RHEL / Ubuntu) so an OS update cannot upgrade podman, containerd or the Kubernetes tools out from under the cluster.
Rotate credentials for the platform components and keep them in secrets.

Kubeopsctl-2.0: Supported Package Versions — Ubuntu / Debian

Mon, 01 Jan 0001 00:00:00 +0000

This page lists the packages and versions available from the KubeOps package repository for Ubuntu 24.04 (the .deb pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Important Not all of these versions are installed at once. The Kubernetes version is selected with kubernetesVersion in cluster-values.yaml (and kubeopsctl pull --kubernetesVersion <x.y.z>); kubeopsctl then installs the matching kubeadm, kubelet, kubectl, cri-tools and kubernetes-cni. For OIDC (dashboard SSO) you need kubeopsctl 2.1.0.0 or newer.

Kubernetes

Package	Supported versions
`kubeadm`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubelet`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubectl`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubernetes-cni`	1.4.0, 1.5.1, 1.6.0, 1.7.1, 1.8.0, 1.9.1
`cri-tools`	1.30.0, 1.31.0, 1.32.0, 1.33.0, 1.34.0, 1.35.0, 1.36.0

KubeOps tools

Package	Supported versions
`kosi`	2.12.1.0, 2.13.0.1, 2.13.0.2
`kubeopsctl`	2.0.0.0, 2.0.1.0, 2.0.2.0, 2.0.3.0, 2.0.4.0, 2.0.5.0, 2.1.0.0

Helm

Package	Supported versions
`helm`	3.17.4

Container runtime and etcd

Package	Supported versions
`containerd`	1.7.24
`etcdctl`	3.5.16, 3.5.21
`etcdutl`	3.5.16, 3.5.21
`podman`	4.9.3

Pre-release versions

⚠ Warning The following are pre-release builds (Alpha / Beta / RC) and are not intended for production.

Package	Pre-release versions
`kosi`	2.14.0.4-Beta0, 2.15.0.4-Alpha5, 2.15.0.4-Beta0
`kubeopsctl`	2.2.0.0-Alpha5, 2.2.0.0-Beta0

Pin the installed versions

After the packages are installed, hold them at the installed version so a later apt upgrade cannot upgrade them and break the cluster:

sudo apt-mark hold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

Note kubeopsctl already holds kubelet, kubeadm and kubectl itself (it runs apt-mark hold/unhold on them around installs and upgrades), so for those three the hold is managed for you. The command above additionally covers kubernetes-cni, cri-tools, helm, kosi and kubeopsctl.

To release a hold later (for a controlled upgrade):

sudo apt-mark unhold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

Note containerd is installed from the operating system repository or as a local package. podman is installed as a KOSI lima package. Both packages are included in the hold and unhold commands above. To generate the exact hold list from your repository, run the list-package-versions.sh helper with --pin.

Kubeopsctl-2.0: Supported Package Versions — Red Hat Enterprise Linux (RHEL)

Mon, 01 Jan 0001 00:00:00 +0000

This page lists the packages and versions available from the KubeOps package repository for RHEL 9.6 / 9.7 (the .rpm pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Kubernetes

Package	Supported versions
`kubeadm`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubelet`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubectl`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubernetes-cni`	1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.1, 1.8.0, 1.9.1
`cri-tools`	1.30.1, 1.31.1, 1.32.0, 1.33.0, 1.34.0, 1.35.0, 1.36.0

Container runtime and etcd

Package	Supported versions
`containerd.io`	2.1.5
`podman`	5.2.2
`etcdctl`	3.5.16, 3.5.21
`etcdutl`	3.5.16, 3.5.21

KubeOps tools

Package	Supported versions
`kosi`	2.12.1.0, 2.13.0.1, 2.13.0.2
`kubeopsctl`	2.0.1.0, 2.0.2.0, 2.0.3.0, 2.0.4.0, 2.0.5.0, 2.1.0.0

Helm

Package	Supported versions
`helm`	3.17.4

Pre-release versions

⚠ Warning The following are pre-release builds (Alpha / Beta / RC) and are not intended for production.

Package	Pre-release versions
`kosi`	2.14.0.4_Beta0, 2.15.0.4_Alpha5, 2.15.0.4_Beta0
`kubeopsctl`	2.2.0.0_Alpha5, 2.2.0.0_Beta0

Pin the installed versions

On RHEL the KubeOps package repository is configured so that its packages are excluded from normal dnf transactions — the install commands have to pass --disableexcludes=kubeops-repo to override that exclusion. As a result a regular dnf update already leaves these packages untouched, so the cluster is protected out of the box.

⚠ Warning Confirm the exact exclude= configuration in kubeops.repo (and /etc/dnf/dnf.conf) for your release. The versionlock step below is an additional safeguard on top of that exclusion.

To lock the installed versions explicitly, use the versionlock plugin after the packages are installed:

sudo dnf install -y python3-dnf-plugin-versionlock
sudo dnf versionlock add \
  containerd.io podman kubelet kubeadm kubectl kubernetes-cni cri-tools \
  etcdctl etcdutl helm kosi kubeopsctl conntrack-tools

Alternatively, exclude them explicitly on each update:

sudo dnf update --exclude=containerd.io,podman,kubelet,kubeadm,kubectl,kubernetes-cni,cri-tools,etcdctl,etcdutl,helm,kosi,kubeopsctl

Dependencies

The repository also provides the following compatibility/dependency packages (32-bit i686 runtime libraries), which are pulled in automatically: conntrack-tools (1.4.7), glibc (2.34), glibc-gconv-extra (2.34), libgcc (11.5.0), libmnl (1.0.4), libnetfilter_cthelper (1.0.0), libnetfilter_cttimeout (1.0.0), libnetfilter_queue (1.0.5), libnfnetlink (1.0.1).

Note These are operating-system libraries. Do not version-lock glibc and the other system libraries — they should continue to receive OS security updates. Pin only the KubeOps and Kubernetes stack packages listed above.