This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

How to Guides

Welcome to our comprehensive How-To Guide for using kubeops. Whether youre a beginner aiming to understand the basics or an experienced user looking to fine-tune your skills, this guide is designed to provide you with detailed step-by-step instructions on how to navigate and utilize all the features of kubeops effectively.

1: Documentation for production

1.1: How-To: Production Cluster with OIDC (Public IP & Airgap)
1.2: Supported Package Versions — Ubuntu / Debian
1.3: Supported Package Versions — Red Hat Enterprise Linux (RHEL)

2: Joining a Node in a Kubernetes cluster
3: Deleting a Node from a Kubernetes cluster
4: Single Sign-On with Keycloak

4.1: Single Sign-On (SSO) with Keycloak for Harbor
4.2: Single Sign-On (SSO) with Keycloak for KubeOps Dashboard
4.3: Single Sign-On (SSO) with Keycloak for OpenSearch
4.4: Single Sign-On (SSO) with Keycloak for rook-ceph

5: Upgrading a Kubernetes cluster
6: Installing KubeOps Compliance applications
7: Updating KubeOps Compliance applications
8: Harbor Deployment with CloudNativePG
9: Ingress Configuration
10: Accessing Dashboards installed with KubeOps
11: Accessing Dashboards with OIDC
12: Changing a User Password in OpenSearch
13: Backup and restoring artifacts
14: Add certificate as trusted
15: Deploy Package on a cluster
16: How to migrate from nginx to traefik ingress

In the following sections, you will find everything from initial setup and configuration, to advanced tips and tricks that will help you get the most out of the software. Our aim is to assist you in becoming proficient with kubeops, enhancing both your productivity and your user experience.

Lets get started on your journey to mastering kubeops!

1 - Documentation for production

Learn how to deploy a production ready cluster

1.1 - How-To: Production Cluster with OIDC (Public IP & Airgap)

End-to-end how-to for customers to build a production-ready KubeOps Compliance environment with the full platform stack and OIDC via Keycloak — covering internet/public-IP and airgap, Let’s Encrypt and own-CA certificates, and nginx or Traefik ingress.

How-To: Production Cluster with OIDC (Public IP & Airgap)

This how-to walks a customer through building a production-ready KubeOps Compliance environment from clean machines to a running, fully equipped, authenticated cluster: the complete platform stack (storage, registry, monitoring, logging, policy, dashboard, backup) with Keycloak providing OIDC authentication, and the dashboards reachable over HTTPS with valid certificates.

It covers the combinations a customer realistically needs:

Network path — internet-connected (with public IPs) or airgap (KubeOps Compliance’s core strength).
Certificates — automatic Let’s Encrypt (public IPs) or your own CA (private DNS / airgap).
Ingress — nginx (default) or Traefik, including how to migrate.

kubeopsctl uses two configuration files, and it is important to keep them apart:

cluster-values.yaml — defines the cluster itself (nodes, networking, virtual IP). It has no influence over which applications get installed.
enterprise-values.yaml — defines the platform applications as a list of packages.

Important OIDC authentication for the dashboards requires kubeopsctl 2.1.0 or newer. On older versions the OIDC integration described here is not available.

⚠ Warning This guide assumes Ubuntu 24.04 or RHEL 9.6 on all machines. For OpenSUSE MicroOS the installation flow differs — follow Prepare Cluster (MicroOS) and Setup Cluster (MicroOS) and apply the production adaptations from this guide.

Important Build the environment as a normal user, not as root. Every step is performed by an unprivileged user who has sudo rights through the sudoers file distributed in section 3.5. The same username (and password) must exist on every machine.

Overview: what you will build

A highly available cluster with the minimum supported production topology and the full platform stack:

1 admin node, 3 control-plane nodes, 3 worker nodes (each worker with extra disks for storage), across three zones
a KubeVip virtual IP as the stable, highly available API endpoint
the full stack via enterprise-values.yaml: Rook-Ceph (storage), Keycloak (OIDC), ingress-nginx or Traefik, cert-manager (TLS), Harbor (registry), Prometheus/Grafana (monitoring), OpenSearch + Logstash + Filebeat (logging), OPA Gatekeeper (policy), KubeOps Dashboard, Velero (backup)
dashboards reachable via a domain name with valid certificates and Keycloak SSO

Choose your path

Two decisions shape the rest of this guide. Pick the row that matches your environment:

Environment	Certificates	Deployment	Notes
Internet + public IP	Let’s Encrypt (automatic)	single-phase	cert-manager issues valid certs automatically
Internet, private DNS / no public IP	own CA (or LE via DNS-01)	two-phase	provide a valid cert manually first
Airgap	own CA (no public ACME reachable)	two-phase	packages served from a local Harbor registry

Why deployment differs:

Let’s Encrypt (single-phase): cert-manager automatically creates a letsencrypt ClusterIssuer, and every ingress references it, so valid, browser-trusted certificates are issued as the components come up. Everything can be installed in one apply.
Own CA (two-phase): the OIDC dashboard integration needs a valid certificate to already exist when it runs. If it runs against a self-signed/placeholder certificate, it falls back to a dummy secret and breaks. So you first bring up storage, registry, ingress and cert-manager, install your valid certificate manually, and only then install Keycloak and the rest.

The ingress controller (nginx or Traefik) is an independent choice — see section 8.

1. Prerequisites

At least seven machines: one admin, three control-plane, three worker.

OS	Minimum requirements
Red Hat Enterprise Linux 9.6 / 9.7	8 CPU cores, 16 GB memory, 50 GB disk
Ubuntu 24.04.02	8 CPU cores, 16 GB memory, 50 GB disk

Important These are the documented minimum. The full platform stack (Ceph, Harbor, OpenSearch, Prometheus) is resource- and storage-intensive — size worker nodes well above the minimum.

Note Only Ubuntu 24.04 and Red Hat Enterprise Linux 9.6 / 9.7 are supported. The node preparation detects the OS by its exact name, so RHEL derivatives (Rocky Linux, AlmaLinux, CentOS Stream) are not recognized and are unsupported.

Before you start:

kubeopsctl 2.1.0 or newer (required for OIDC).
Same username and password on every machine (cluster creation fails otherwise).
Unique, lowercase hostname per machine.
OS version pinned to a supported release.
Network access: for the internet path, the cluster nodes reach the KubeOps registry directly. For the airgap path, only the admin node needs temporary outbound access to https://dispatcher.kubeops.net (KOSI hub) and https://registry.kubeops.net while kosi / kubeopsctl run; the cluster nodes stay isolated and pull all images from the Harbor that the platform stack deploys (section 5.1). KubeOps supports secure registries only.
Storage disks for Rook-Ceph: every persistent component uses the rook-cephfs storage class, which is 3× replicated across the workers — so raw disk ≈ 3 × the sum of all PVC claims. For the full platform stack with 30-day retention, provide one dedicated, unformatted disk of at least ~300 GB per worker (500 GB recommended); see the sizing table in section 7.1. Also see the Rook-Ceph prerequisites.
yq on all control-plane nodes (required for the OIDC dashboard integration) — yq install.
Passwords prepared for Harbor (and its Postgres/Redis), Keycloak (admin + Postgres) and Grafana — never the documented defaults.
DNS: for OIDC you need a DNS name for Keycloak and the dashboards. For Let’s Encrypt this must be public DNS; for an own CA it can be internal DNS.

2. Plan addressing and ports

Example node plan (203.0.113.x are placeholders from the documentation range, RFC 5737 — replace with your real IPs):

Hostname	Role	Zone	IP
`admin`	Admin / management	–	`203.0.113.10`
`controlplane01`	Control plane	zone1	`203.0.113.11`
`controlplane02`	Control plane	zone2	`203.0.113.12`
`controlplane03`	Control plane	zone3	`203.0.113.13`
`worker01`	Worker	zone1	`203.0.113.21`
`worker02`	Worker	zone2	`203.0.113.22`
`worker03`	Worker	zone3	`203.0.113.23`
–	KubeVip virtual IP (API)	–	`203.0.113.100`

Note The KubeVip virtual IP must be unused and in the same network as the control-plane nodes; KubeVip announces it via ARP. Many cloud/hosting providers route public IPs individually rather than in a shared L2 segment — there, ARP-based KubeVip does not work and you need BGP mode or a provider load balancer. As an alternative to KubeVip, KubeOps can front the API servers with an HAProxy load balancer (KubeOps API port 7443).

Default service ports:

Service	Port / NodePort
Kubernetes API (via KubeVip)	`6443`
Ingress (HTTP / HTTPS)	`80` / `443`
Harbor	`30002`
Keycloak	`30180`
Grafana	`30211`
Prometheus	`32090`
OpenSearch Dashboards	`30050`
KubeOps Dashboard	`30007`
SSH	`22`

Ports that must stay open between the cluster nodes (opened automatically by the KubeOps firewall setup):

Purpose	Port(s)
Kubernetes API server	`6443`
etcd	`2379`, `2380`
kubelet	`10250`
kube-proxy	`10256`
kube-scheduler / controller-manager	`10257`, `10259`
NodePort range	`30000–32767`
Calico BGP / VXLAN / Typha	`179`, `4789/udp`, `5473`
Calico WireGuard (pod-to-pod encryption)	`51820/udp`
Local registries (airgap)	`5000`, `5001`
HAProxy API load balancer	`7443`
Rook-Ceph (mon / mgr / OSD)	`3300`, `6789`, `6800–7568`
SSH	`22`

⚠ Warning KubeOps configures the node firewall automatically (firewall: nftables, the default; ufw on Ubuntu). The generated nftables ruleset uses policy accept and only counts/allows the ports above — it is an allow-list annotation, not a default-deny firewall. On RHEL, firewalld is stopped and masked in favor of nftables. Inter-node traffic must stay open between the nodes but must not be exposed to the internet. For the Let’s Encrypt HTTP-01 challenge, inbound port 80 must be reachable from the internet. For production, restrict the dashboard NodePorts to trusted networks and consider a default-deny policy in a firewall in front of the nodes.

3. Prepare all nodes

Run on every node unless noted.

3.1 Add the package repository (internet path)

wget https://packagerepo.kubeops.net/pgp-key.public
cat pgp-key.public | sudo gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/kubeops.gpg] https://packagerepo.kubeops.net/deb stable main' | sudo tee /etc/apt/sources.list.d/kubeops.list
sudo apt update

sudo dnf config-manager --add-repo https://packagerepo.kubeops.net/rpm/kubeops.repo

Note Airgap: the KOSI packages are pulled from the KOSI hub (https://dispatcher.kubeops.net) and the container images from https://registry.kubeops.net. The admin node needs outbound access to both only while kosi / kubeopsctl run (pulling packages and seeding images) — not permanently. The admin node is the bridge that stages the packages and seeds the platform-stack images into your local Harbor (section 5.1), while the cluster nodes stay isolated. On the kubeopsctl side, airgap mode is simply airgap: true in cluster-values.yaml (section 6).

3.2 Operating system adaptations

sudo apt remove unattended-upgrades

# no special adaptations required

3.3 Make hostnames resolvable

Use DNS for all node hostnames, or add them to /etc/hosts on every node:

sudo tee /etc/hosts <<EOL_ETC_HOSTS
127.0.0.1 localhost
203.0.113.10 admin
203.0.113.11 controlplane01
203.0.113.12 controlplane02
203.0.113.13 controlplane03
203.0.113.21 worker01
203.0.113.22 worker02
203.0.113.23 worker03
EOL_ETC_HOSTS

3.4 Distribute SSH keys

Passwordless SSH must work from each node to every other node. Root login is temporarily enabled and reverted in section 4.

⚠ Warning This temporarily enables root login and password authentication. It must be reversed before the cluster goes live.

printf 'PermitRootLogin yes\nPasswordAuthentication yes\n' | sudo tee /etc/ssh/sshd_config.d/permit_root_login.conf
sudo systemctl restart sshd

# create keys on all nodes
ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N ""

# copy the public key, scan host keys, and test login — for EVERY node
for host in admin controlplane01 controlplane02 controlplane03 worker01 worker02 worker03; do
  ssh-copy-id -i ~/.ssh/id_ed25519 "$host"
  ssh-keyscan "$host" >> ~/.ssh/known_hosts
  ssh "$host" exit
done
# repeat ssh-keyscan with the IP addresses as well

3.5 Distribute the sudoers file

The cluster is built and operated by the unprivileged user from the warning at the top of this guide — not as root. That user needs the elevated rights kubeopsctl requires, granted through a sudoers file. Replace myuser with your username and copy the file to /etc/sudoers.d/<username> on all nodes.

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/gpg --dearmor -o /usr/share/keyrings/kubeops.gpg
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/apt/sources.list.d/kubeops.list
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove unattended-upgrades
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove *
myuser ALL=(root) NOPASSWD: /usr/bin/lsof /var/lib/dpkg/lock*

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kosi*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kosi*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt-get install -y kubeopsctl*
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install kubeopsctl*.deb


# kube-vip image import
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# Kubeadm init
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# enable UFW
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reset
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 6443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 2379\:2380/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10250/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10259/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10257/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 10256/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 30000\:32767/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 179/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 4789/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5473/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 51820/udp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 22/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5000/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 5001/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw allow 7443/tcp
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw logging low
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw enable
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw reload
myuser ALL=(root) NOPASSWD: /usr/sbin/ufw status
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart systemd-networkd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable --now ufw

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# kubeadm init phase and kubeadm token create commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /usr/bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# upgrade nodes
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin
myuser ALL=(root) NOPASSWD: /bin/rm -f /etc/cni/net.d/87-podman-bridge.conflist

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubeadm*
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubectl*
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl

myuser ALL=(root) NOPASSWD: /usr/bin/dpkg install -y kubelet*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet

myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark hold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt-mark unhold kubelet kubeadm kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubelet* kubeadm* kubectl*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y kubeadm*

# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install passt_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conmon_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install catatonit_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install netavark_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install aardvark-dns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-image_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install golang-github-containers-common_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containernetworking-plugins_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libsubid4_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install uidmap_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libslirp0_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install slirp4netns_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install libyajl2_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install crun_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install fuse-overlayfs_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install buildah_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install podman_*.deb

# Podman Remove of local .deb-Package
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove podman buildah fuse-overlayfs crun libyajl2 slirp4netns libslirp0 uidmap libsubid4 containernetworking-plugins golang-github-containers-common golang-github-containers-image aardvark-dns netavark catatonit conmon passt

# Podman Installation & Update with apt
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y podman
myuser ALL=(root) NOPASSWD: /usr/bin/apt update
myuser ALL=(root) NOPASSWD: /usr/bin/apt remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install */pia/kosi_*

# runtime setup
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install conntrack_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install runc_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install containerd_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y iproute-tc

# Remove packages
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove containerd

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard_*.deb wireguard-tools_*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --install wireguard-tools-*.deb systemd-resolved-*.deb
myuser ALL=(root) NOPASSWD: /usr/bin/dpkg --remove wireguard wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y wireguard-tools*
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y systemd-resolved*

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/apt install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/local/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

# Preperation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf config-manager --add-repo https\://packagerepo.kubeops.net/rpm/kubeops.repo
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/hosts

# Setup
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kosi*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kosi*.rpm, !/usr/bin/rpm --install kosi*[[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*, !/usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeopsctl*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install kubeopsctl*.rpm, !/usr/bin/rpm --install kubeopsctl*[[\:space\:]]*.rpm

# kube-vip/create-manifest.sh.tmpl
myuser ALL=(root) NOPASSWD: /bin/cp kube-vip.yaml /etc/kubernetes/manifests/kube-vip.yaml

# delete-node/delete-node.sh.tmpl
# systemctl commands
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable containerd

# kubeadm reset
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm reset --force

# kubeadm-init/kubeadm-init.sh.tmpl
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init --upload-certs --config cluster-config.yaml

# remove folders
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /etc/kubernetes
myuser ALL=(root) NOPASSWD: /bin/rm -fr /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/etcd
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/lib/kubelet
myuser ALL=(root) NOPASSWD: /bin/rm -fr /var/kubeops

# reboot
myuser ALL=(root) NOPASSWD: /sbin/reboot now

# disable swap
myuser ALL=(root) NOPASSWD: /usr/sbin/swapoff --all
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask swap.target
myuser ALL=(root) NOPASSWD: /bin/sed -e * -i /etc/fstab

# nftables enable/restart
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now nftables
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart nftables

# copy nftables configs
myuser ALL=(root) NOPASSWD: /bin/cp nftables.conf /etc/sysconfig/nftables.conf

# firewalld control
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl stop firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl disable firewalld
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl mask firewalld

# Install/update Helm
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/bin
myuser ALL=(root) NOPASSWD: /bin/cp helm /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/helm
# if Helm is in repository
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y helm* 

# Delete Helm
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/helm

# k9s/package.kosi
myuser ALL=(root) NOPASSWD: /bin/cp k9s /usr/bin/
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/k9s
myuser ALL=(root) NOPASSWD: /bin/rm -f /usr/bin/k9s

# crictl pull images
myuser ALL=(root) NOPASSWD:SETENV: /usr/bin/crictl pull *

# ssh remote kubeadm commands
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm init phase upload-certs --upload-certs
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command --certificate-key *

# local execution of kubeadm join
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm join *

# kubernetes admin.conf handling
myuser ALL=(root) NOPASSWD: /bin/cp /etc/kubernetes/admin.conf /home/*/.kube/config
myuser ALL=(root) NOPASSWD: /bin/chown [0-9]*\:[0-9]* /home/*/.kube/config

# scheduler config copy
myuser ALL=(root) NOPASSWD: /bin/cp scheduler-config.yaml /etc/kubernetes/scheduler-config.yaml

# scheduler manifest patching
myuser ALL=(root) NOPASSWD: /bin/grep -q * /etc/kubernetes/manifests/kube-scheduler.yaml
# Adjustments sed -i * 
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i * /etc/kubernetes/manifests/kube-scheduler.yaml

# restart services
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart containerd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm token create --print-join-command

# create kubernetes manifests folder
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/kubernetes/manifests

# modeprobe br_netfilter
myuser ALL=(root) NOPASSWD: /bin/cp br_netfilter.conf /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /bin/chmod 644 /etc/modules-load.d/br_netfilter.conf
myuser ALL=(root) NOPASSWD: /sbin/modprobe br_netfilter
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload

# kubernetes image pull
myuser ALL=(root) NOPASSWD: /usr/bin/ctr *

# kubernetes version upgrade
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade plan --ignore-preflight-errors=all
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade apply * --ignore-preflight-errors=all --yes,  !/usr/bin/kubeadm upgrade apply *[[\:space\:]]* --ignore-preflight-errors=all --yes

# kubeadm upgrade node
myuser ALL=(root) NOPASSWD: /bin/cp /home/*/.kube/config /etc/kubernetes/admin.conf
myuser ALL=(root) NOPASSWD: /usr/bin/kubeadm upgrade node

# kubernetes-tools-packages
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf cni-* -C /opt/cni/bin, !/bin/tar xzf cni-*[[\:space\:]]* -C /opt/cni/bin
myuser ALL=(root) NOPASSWD: /bin/tar xzf crictl-* -C /usr/bin, !/bin/tar xzf crictl-*[[\:space\:]]* -C /usr/bin
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubeadm*, !/usr/bin/dnf install -y kubeadm*[[\:space\:]]*, !/usr/bin/dnf install -y kubeadm*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubeadm /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubeadm
myuser ALL=(root) NOPASSWD: /bin/test -f /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/mv /usr/bin/kubelet /usr/bin/kubelet_*, !/bin/mv /usr/bin/kubelet /usr/bin/kubelet_*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /bin/cp kubelet /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubectl*, !/usr/bin/dnf install -y kubectl*[[\:space\:]]*, !/usr/bin/dnf install -y kubectl*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubectl /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /bin/chmod +x /usr/bin/kubectl
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y kubelet*, !/usr/bin/dnf install -y kubelet*[[\:space\:]]*, !/usr/bin/dnf install -y kubelet*.rpm
myuser ALL=(root) NOPASSWD: /bin/cp kubelet.service /usr/lib/systemd/system/kubelet.service
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/lib/systemd/system/kubelet.service.d
myuser ALL=(root) NOPASSWD: /bin/cp 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now kubelet
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubelet-* kubeadm-* kubectl-*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y --disableexcludes=kubeops-repo kubeadm-*


# Allow HAProxy and load-balancer
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /usr/local/etc/haproxy
myuser ALL=(root) NOPASSWD: /bin/cp haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
myuser ALL=(root) NOPASSWD: /bin/cp load-balancer.yaml /etc/kubernetes/manifests/load-balancer.yaml
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /mnt/registry
myuser ALL=(root) NOPASSWD: /bin/cp docker-registry.yaml /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm /etc/kubernetes/manifests/docker-registry.yaml
myuser ALL=(root) NOPASSWD: /bin/rm -rf /mnt/registry
myuser ALL=(root) NOPASSWD: /usr/bin/crictl --namespace k8s.io images import local-registry-image.tar

# Podman Installation of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install passt-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install aardvark-dns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install netavark-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install container-selinux-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libnet-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install criu-libs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install libslirp-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install slirp4netns-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install yajl-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install crun-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containers-common-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install fuse-overlayfs-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install shadow-utils-subid-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conmon-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install podman-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-*.rpm iproute-tc-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install containerd.io-*.rpm

# Podman Remove of local .rpm-Package
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase podman shadow-utils-subid fuse-overlayfs crun containers-common yajl slirp4netns libslirp criu-libs criu libnet container-selinux netavark aardvark-dns passt passt-selinux

# Podman Installation & Update with dnf
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y podman*, !/usr/bin/dnf install -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf update -y podman*,!/usr/bin/dnf update -y podman*[[\:space\:]]*
myuser ALL=(root) NOPASSWD: /usr/bin/dnf remove -y podman

# Allow prepare-node
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install */pia/kosi-*

# Allow installation of container-runtime
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm, !/usr/bin/rpm --install conntrack-tools-*[\:space\:]]*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install conntrack-tools-*.rpm libnetfilter_cthelper-*.rpm libnetfilter_cttimeout-*.rpm libnetfilter_queue-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install iproute-tc-*.rpm, !/usr/bin/rpm --install iproute-tc-*[\:space\:]]*.rpm

# Enable repo and install containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y containerd.io
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y conntrack-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y iproute-tc

# Remove RPM packages
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase containerd.io

# Create directories
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/containerd/certs.d
myuser ALL=(root) NOPASSWD: /bin/mkdir -p /etc/systemd/system/containerd.service.d/
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/systemd/system/containerd.service.d/override.conf*

# Configure containerd
myuser ALL=(root) NOPASSWD: /usr/bin/tee /etc/containerd/config.toml
myuser ALL=(root) NOPASSWD: /bin/sed -i * /etc/containerd/config.toml

# Allow containerd insecure registry
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i -e * /etc/containerd/config.toml

# Enable and start containerd service
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now containerd

# Allow managing k8s sysctl configuration
myuser ALL=(root) NOPASSWD: /bin/cp k8s.conf /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/cp kubevirt.conf /etc/sysctl.d/kubevirt.conf
myuser ALL=(root) NOPASSWD: /usr/sbin/sysctl --system
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/k8s.conf
myuser ALL=(root) NOPASSWD: /bin/rm /etc/sysctl.d/kubevirt.conf

# Allow WireGuard package management
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --install wireguard-tools-*.rpm systemd-resolved-*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/rpm --erase wireguard-tools
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y wireguard-tools*, !/usr/bin/dnf install -y wireguard-tools*[\:space\:]]*, !/usr/bin/dnf install -y wireguard-tools*.rpm
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y systemd-resolved*, !/usr/bin/dnf install -y systemd-resolved*[\:space\:]]*, !/usr/bin/dnf install -y systemd-resolved*.rpm

# velero install on admin
myuser ALL=(root) NOPASSWD: /usr/bin/cp binary/velero /usr/bin

# chrony installation
myuser ALL=(root) NOPASSWD: /usr/bin/dnf install -y chrony
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart chronyd
myuser ALL=(root) NOPASSWD: /usr/bin/vi /etc/chrony/chrony.conf
myuser ALL=(root) NOPASSWD: /usr/bin/vim /etc/chrony/chrony.conf

# update apiserver on control plane
myuser ALL=(root) NOPASSWD: /usr/bin/cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
myuser ALL=(root) NOPASSWD: /usr/bin/sed -i /--oidc-/d /etc/kubernetes/manifests/kube-apiserver.yaml
myuser ALL=(root) NOPASSWD: /usr/bin/yq e * -i /etc/kubernetes/manifests/kube-apiserver.yaml

3.6 Install kubeadm

sudo apt install -y kubeadm=<kubernetes-version>*

sudo dnf install -y --disableexcludes=kubeops-repo kubeadm-<kubernetes-version>

3.7 Configure time synchronization

Important Accurate time is critical for OIDC: tokens are time-limited JWTs, and clock skew between nodes and Keycloak causes authentication to fail intermittently.

sudo apt install -y chrony
sudo systemctl enable --now chrony

sudo dnf install -y chrony
sudo systemctl enable --now chronyd

Configure NTP servers in /etc/chrony.conf, sudo systemctl restart chronyd, then verify with chronyc tracking (leap status Normal, offset < 10 ms).

3.8 Install curl and yq

Install curl on every node, and yq on all control-plane nodes (required for the OIDC dashboard integration) per the yq installation guide.

4. Harden the nodes for production

Revert the SSH change from section 3.4 on every node:

sudo rm /etc/ssh/sshd_config.d/permit_root_login.conf
sudo systemctl restart sshd

⚠ Warning Leaving root login enabled on publicly reachable nodes is a serious security risk. Confirm key-based SSH still works after reverting.

Also: pin the OS version and tighten the firewall per section 2.

Pin the KOSI-installed packages. kubeopsctl installs a set of system packages (podman, containerd, the Kubernetes tools, etc.) at specific, validated versions. An OS update (apt upgrade / dnf update) can silently upgrade them and break the cluster. After section 6, pin/hold every one of these packages to the installed version so OS updates cannot move them.

Ubuntu: sudo apt-mark hold <package> (e.g. podman, containerd, kubelet, kubeadm, kubectl)
RHEL: exclude them from updates, e.g. sudo dnf update --exclude=podman,containerd,kubelet,kubeadm,kubectl, or use the versionlock plugin

Note The exact package list and the validated versions for your release are documented on the separate Installed Package Versions pages for RHEL and Ubuntu / Debian. Pin all of them, not just podman.

5. Set up the admin tooling

Perform the rest from the admin node.

Install KOSI, then set KUBEOPSROOT, adjust the KOSI config, install the enterprise plugins, log in, and install kubeopsctl (≥ 2.1.0) — exactly as in the Setup Cluster guide:

# KOSI
sudo apt install -y kosi=<kosi-version>          # Ubuntu  | RHEL: sudo dnf install -y --disableexcludes=kubeops-repo <kosi-version>

# KUBEOPSROOT (append to ~/.bashrc, then: source ~/.bashrc)
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

# KOSI config + plugins
mkdir ~/kubeops && cd ~/kubeops
cp -R /var/kubeops/kosi/ . && cp -R /var/kubeops/plugins/ .
# edit $KUBEOPSROOT/kosi/config.yaml: set hub and the FULL plugins path

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.0.0
kosi login -u <yourUser>                          # a "registry temporarily not available" error here is expected

# kubeopsctl (>= 2.1.0 for OIDC)
sudo apt install -y kubeopsctl=<kubeopsctl-version>   # Ubuntu | RHEL: sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

5.1 (Airgap only) Stage the images in your local Harbor

In an airgapped environment the platform-stack images are served from your local Harbor registry. The flow is:

rook and harbor first. Only rook-ceph and harbor ship dedicated airgap packages that carry their own container images. Install these two first so storage and the registry exist.
Seed the remaining images. Once Harbor is up, kubeopsctl pulls every other tool’s images from registry.kubeops.net, pushes them into the deployed Harbor, and re-tags each reference to 127.0.0.1:30002/kubeops/<image> — Harbor reached over its NodePort 30002 on each node, project kubeops.
Private kubeops project. The kubeops Harbor project is private, so every tool is deployed with an image pull secret for it.
Point the config at local. Set airgap: true and packageRepository: local in cluster-values.yaml (section 6), and localRegistry: true in enterprise-values.yaml (section 7).

Note KubeOps supports secure registries only — Harbor must be reachable with a certificate your nodes trust (section 9).

6. Create the cluster

The cluster-values.yaml describes only the cluster. The airgap flag is the key difference between the two network paths.

⚠ Warning This file has to be adapted to your values before usage.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: true
virtualIP: <unused IP in the control-plane network>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: false
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true
zones:
- name: zone1
  nodes:
  - { name: <controlplane01 hostname>, iPAddress: <controlplane01 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker01 hostname>,       iPAddress: <worker01 ip>,       type: worker,       kubeVersion: <version> }
- name: zone2
  nodes:
  - { name: <controlplane02 hostname>, iPAddress: <controlplane02 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker02 hostname>,       iPAddress: <worker02 ip>,       type: worker,       kubeVersion: <version> }
- name: zone3
  nodes:
  - { name: <controlplane03 hostname>, iPAddress: <controlplane03 ip>, type: controlplane, kubeVersion: <version> }
  - { name: <worker03 hostname>,       iPAddress: <worker03 ip>,       type: worker,       kubeVersion: <version> }

# file cluster-values.yaml  (airgap)
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: <your local Harbor>/kubeops    # your local secure registry
airgap: true
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: true
virtualIP: <unused IP in the control-plane network>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: false
packageRepository: local
changeCluster: true
zones:
  # ... same zone layout as the Internet tab ...

Then pull packages, install the runtime tooling, and create the cluster:

kubeopsctl pull                                                        # add --kubernetesVersion <x.xx.x> to pin a version
kosi install -p $KUBEOPSROOT/lima/podman_<version>.tgz -f cluster-values.yaml
kosi install -p $KUBEOPSROOT/lima/helm_<version>.tgz
kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml
kosi login -u <your username>
kubeopsctl apply -f cluster-values.yaml                                # creates the cluster (~10–15 min)

⚠ Warning Now pin all KOSI-installed packages (podman, containerd, the Kubernetes tools, …) to their installed versions so a later OS update cannot upgrade them — see section 4 and the Installed Package Versions pages (RHEL / Ubuntu). The package versions above (podman, helm) are examples — check the repository for the versions shipped with your release; the kubernetes-tools version must match the version you pulled.

7. Configure the platform stack

The applications live in enterprise-values.yaml. Each package has a standard block (predefined keys) and an advanced block (any value from the underlying Helm chart). Standard values overwrite advanced values if both are set.

The full reference below enables the whole stack and is wired for OIDC dashboard access (each dashboard gets a hostname). It is not applied as-is in one step for every scenario — the ingress choice (section 8), the certificate scenario (section 9) and the deployment (section 10) determine how.

⚠ Warning Replace every <domain_name>, every placeholder password and the S3 credentials before applying. Airgap: set localRegistry: true.

Full enterprise-values.yaml (all components)

# file enterprise-values.yaml
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false            # airgap: true
packages:

# Storage — provides the rook-cephfs storage class used by everything below
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      hostname: <domain_name>
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
    advanced:
      # disk selection for Ceph goes here (rook-ceph-cluster Helm values),
      # e.g. storage.useAllDevices / storage.deviceFilter / per-node devices
      cluster:
      operator:

# Ingress controller — externalIPs binds it to your IP(s). For Traefik see section 8.
- name: ingress-nginx
  enabled: true
  values:
    standard:
      namespace: ingress-nginx
      externalIPs:
      - <ingress ip>             # public IP (Scenario A) or internal IP (Scenario B)
    advanced:

# Certificates
- name: cert-manager
  enabled: true
  values:
    standard:
      namespace: cert-manager
      replicaCount: 3
      logLevel: 2
      # Scenario B (own CA): point cert-manager at your own certificate secret
      # secretName: example-ca
    ca:
      # Scenario A (Let's Encrypt): cert-manager auto-creates a "letsencrypt" ClusterIssuer
      emailLetsEncrypt: <your_email@domain.com>   # MUST set (default example@example.com will not work)
      ingressName: nginx                          # MUST match the ingress controller (nginx or traefik)

# OIDC provider — Keycloak
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: keycloak
      storageClass: "rook-cephfs"
      nodePort: "30180"
      hostname: <domain_name>
      keycloak:
        auth:
          adminUser: admin
          adminPassword: "<strong password>"
          existingSecret: ""
      postgresql:
        auth:
          postgresPassword: "<strong password>"
          username: bn_keycloak
          password: "<strong password>"
          database: bitnami_keycloak
    advanced:

# Registry (also the local registry in airgap)
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "<strong password>"
      databasePassword: "<strong password>"
      redisPassword: "<strong password>"
      externalURL: https://<domain_name>
      nodePort: 30002
      hostname: <domain_name>
      harborPersistence:
        persistentVolumeClaim:
          registry:                                # full-stack images ~5 GB; rest = headroom for your own images — enable Harbor GC + a project quota (see section 7.1)
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:                                # also holds Trivy scan reports
            size: 5Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy:
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:

# Monitoring
- name: kube-prometheus-stack
  enabled: true
  values:
    standard:
      namespace: monitoring
      grafanaUsername: admin
      grafanaPassword: "<strong password>"
      grafanaResources:
        hostname: <domain_name>
        nodePort: 30211
        storageClass: "rook-cephfs"
        storage: 10Gi
      prometheusResources:
        hostname: <domain_name>
        nodePort: 32090
        storageClass: "rook-cephfs"
        storage: 50Gi
        retention: 30d
        retentionSize: "45GB"
    advanced:
      # Alertmanager ships with no persistent storage (emptyDir) — give it a PVC so silences/state survive restarts
      alertmanager:
        alertmanagerSpec:
          storage:
            volumeClaimTemplate:
              spec:
                storageClassName: rook-cephfs
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 2Gi

# Logging stack
- name: opensearch-os
  enabled: true
  values:
    standard:
      namespace: logging
      hostname: <domain_name>
      opensearchJavaOpts: "-Xmx512M -Xms512M"
      resources:
        requests:
          cpu: "250m"
          memory: "1024Mi"
        limits:
          cpu: "300m"
          memory: "3072Mi"
      persistence:                                 # size for your chosen log retention (see section 7.1)
        size: 40Gi
        storageClass: "rook-cephfs"
        accessModes: ["ReadWriteMany"]
      securityConfig:
        enabled: false
      replicas: "3"
    advanced:
      # Log retention: no lifecycle policy ships by default, so log indices grow UNBOUNDED.
      # This Job installs an ISM policy that deletes log indices older than the configured age and
      # re-runs idempotently on every `kubeopsctl apply` (upgrade-safe). See section 7.1.
      extraObjects:
      - apiVersion: batch/v1
        kind: Job
        metadata:
          name: kubeops-log-retention
          namespace: logging
        spec:
          ttlSecondsAfterFinished: 300
          backoffLimit: 6
          template:
            spec:
              restartPolicy: OnFailure
              containers:
              - name: apply-ism
                image: curlimages/curl:8.9.1        # airgap: reference this image from your local Harbor
                command: ["/bin/sh","-c"]
                args:
                - |
                  # OpenSearch endpoint — http + no auth matches securityConfig.enabled=false above;
                  # if you enable securityConfig, switch to https and pass admin credentials.
                  OS="http://opensearch-cluster-master.logging.svc:9200"
                  # >>> RETENTION: change "30d" to "60d", "90d", ... as needed <<<
                  cat >/tmp/policy.json <<'JSON'
                  {"policy":{"description":"Delete KubeOps log indices","default_state":"hot",
                   "states":[
                     {"name":"hot","actions":[],"transitions":[{"state_name":"delete","conditions":{"min_index_age":"30d"}}]},
                     {"name":"delete","actions":[{"delete":{}}],"transitions":[]}],
                   "ism_template":[{"index_patterns":["logstash-logs-*"],"priority":100}]}}
                  JSON
                  curl -sf -XPUT  "$OS/_plugins/_ism/policies/kubeops-log-retention" -H 'Content-Type: application/json' -d @/tmp/policy.json
                  curl -sf -XPOST "$OS/_plugins/_ism/add/logstash-logs-*"           -H 'Content-Type: application/json' -d '{"policy_id":"kubeops-log-retention"}' || true                  
- name: opensearch-dashboards
  enabled: true
  values:
    standard:
      namespace: logging
      hostname: <domain_name>
      nodePort: 30050
    advanced:
- name: logstash-os
  enabled: true
  values:
    standard:
      namespace: logging
      volumeClaimTemplate:
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 5Gi
        storageClass: "rook-cephfs"
    advanced:
      # Persistent queue so log bursts/backpressure are buffered on disk instead of dropped.
      # NOTE: this REPLACES the shipped logstash.yml — the first two lines are the existing defaults, keep them.
      logstashConfig:
        logstash.yml: |
          pipeline.ecs_compatibility: disabled
          log.level: info
          queue.type: persisted
          queue.max_bytes: 1gb          
- name: filebeat-os
  enabled: true
  values:
    standard:
      namespace: logging
    advanced:

# Policy
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:

# Cluster dashboard
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: <domain_name>
      service:
        nodePort: 30007
    advanced:

# Backup
- name: velero
  enabled: true
  values:
    standard:
      namespace: velero
      accessKeyId: "<s3 access key>"
      secretAccessKey: "<s3 secret key>"
      provider: "aws"
      bucket: "velero"
      useNodeAgent: true
      useVolumeSnapshots: true
      backupLocationConfig:                        # placeholder — point bucket/region/s3Url at an EXTERNAL S3 store (see section 14)
        region: "minio"
        s3ForcePathStyle: true
        s3Url: "http://minio.velero.svc:9000"
    advanced:
      # Out of the box Velero backs up only Kubernetes objects, on no schedule, deleted after 72h.
      # Add a recurring schedule and a retention TTL (720h = 30d):
      schedules:
        daily:
          schedule: "0 */6 * * *"
          template:
            ttl: "720h"
      # Back up PV data too. NOTE: the `standard` key defaultVolumesToFsBackup is emitted at top level
      # where this chart ignores it — it must live under `configuration:` (set it here in advanced).
      configuration:
        defaultVolumesToFsBackup: true

Secure the components: replace every password (or use existingSecret), set the public addresses (externalURL, hostname, ingress externalIPs), and restrict the dashboards/registry at the firewall to trusted networks.

7.1 Sizing, retention and backups

All persistent components use the rook-cephfs storage class, which is 3× replicated across the worker nodes (failureDomain: host, reclaimPolicy: Retain). Plan raw disk ≈ 3 × the sum of all PVC sizes and never fill Ceph beyond ~75–80 %.

Recommended minimal-but-stable sizes (7-node baseline, 30-day retention). These are the values used in the example above — adjust to your workload, since logging and monitoring volumes scale with the number of nodes/pods and with your retention.

Component	PVC (usable)	Notes
Harbor registry	40Gi	full-stack images ≈ 5 GB; the rest is headroom for your own images
Harbor database	5Gi	also holds Trivy scan reports
Harbor redis / trivy / jobLog	1 / 5 / 1 Gi
Keycloak Postgres	8Gi
Prometheus	50Gi	`retention: 30d`, `retentionSize: "45GB"`
Grafana	10Gi
Alertmanager	2Gi	default ships as emptyDir — add the PVC
OpenSearch (each of 3 nodes)	40Gi	scales with your log retention
Logstash	5Gi	persistent queue
Velero	external S3/MinIO	not on Ceph

The sum of usable PVCs is ≈ 215–250 Gi, so raw Ceph is ≈ 650–750 Gi; with headroom, plan ~900 Gi–1 TiB raw. Because the pools replicate across the three workers, provide a dedicated, unformatted disk of at least ~300 Gi per worker (recommended 500 Gi) for the Ceph OSDs. This supersedes the “50 GB disk per worker” figure in the prerequisites, which only covers an empty cluster.

Rule of thumb for logs (drives OpenSearch and Ceph): budget roughly 0.5–1 GB/day for a small cluster, × your retention days, × 2 for the index replica — and size the OpenSearch PVC to match.

Log retention (OpenSearch) — KubeOps ships no index-lifecycle policy, so log indices grow unbounded and eventually force OpenSearch read-only. The opensearch-os package above therefore carries an extraObjects Job that installs an ISM policy deleting logstash-logs-* indices older than a configurable age. Set your retention in that block via min_index_age ("30d" / "60d" / "90d"). The Job only installs the policy once; from then on OpenSearch’s ISM plugin enforces it continuously on its own — deleting aged indices and auto-attaching the policy to every new logstash-logs-* index via ism_template. So retention keeps working between releases (even months apart), not only during an apply. The Job re-runs on every kubeopsctl apply, so the policy is also restored after a fresh reinstall.
Metrics retention (Prometheus) — set in the standard block via prometheusResources.retention / .retentionSize (example: 30d / 45GB on a 50Gi PVC). Keep retentionSize a few GB below the PVC size for WAL/compaction headroom.
Harbor cleanup (GC) — the Harbor chart configures no garbage collection, so the registry PVC keeps growing as tags/images accumulate. Deploy the GC-schedule Job shown below; once applied, Harbor runs GC on that schedule itself. To cap the private kubeops project, set a storage quota in the UI under Projects → kubeops → Configuration → storage quota (API: PUT /api/v2.0/quotas/<id>).
Backups (Velero) — out of the box Velero captures only Kubernetes objects, on no schedule, deleted after 72h. The example enables the node agent and volume snapshots, adds a daily schedule with a 30-day ttl, and sets configuration.defaultVolumesToFsBackup: true so PV data is included — see section 14.

Harbor GC-schedule Job (apply once; Harbor then runs GC on the schedule itself):

# harbor-gc.yaml — apply with: kubectl -n harbor apply -f harbor-gc.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: kubeops-harbor-gc
  namespace: harbor
spec:
  ttlSecondsAfterFinished: 300
  backoffLimit: 10
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: harbor-gc
        image: curlimages/curl:8.9.1          # airgap: reference this image from your local Harbor
        command: ["/bin/sh","-c"]
        args:
        - |
          API="http://harbor-core.harbor.svc/api/v2.0"
          AUTH="-u admin:<harbor admin password>"
          for i in $(seq 1 60); do curl -s $AUTH "$API/health" >/dev/null 2>&1 && break; sleep 5; done
          # weekly GC (6-field cron), delete untagged artifacts — change the cron as needed
          BODY='{"parameters":{"delete_untagged":true},"schedule":{"type":"Custom","cron":"0 0 4 * * 0"}}'
          curl -s $AUTH -XPUT  "$API/system/gc/schedule" -H 'Content-Type: application/json' -d "$BODY" \
            || curl -s $AUTH -XPOST "$API/system/gc/schedule" -H 'Content-Type: application/json' -d "$BODY"

Important Upgrades: your enterprise-values.yaml is the source of truth — keep it under version control and re-run kubeopsctl apply to upgrade. Every standard and advanced value is re-applied to helm upgrade, so your sizing and retention settings are not overwritten. The OpenSearch ISM Job (shipped via extraObjects) re-runs automatically on each apply; the Harbor GC schedule lives in Harbor’s database and survives rolling upgrades. Both the ISM policy and the GC schedule are enforced continuously by OpenSearch/Harbor themselves between releases — you only re-apply them after a fresh reinstall (the GC Job with kubectl apply).

Note The retention (ISM) and Harbor-GC Jobs are deployed from this guide today. Native integration of these settings into the KubeOps packages (and the Velero defaultVolumesToFsBackup mapping) is planned for a future release; until then, configure them as shown here.

8. Choose the ingress controller

KubeOps supports nginx (default) and Traefik. Whichever you use, the cert-manager ingressName must match it.

Keep the ingress-nginx package enabled in enterprise-values.yaml (as in the full example) and set ingressName: nginx in cert-manager. Nothing else is needed.

Traefik is installed as its own KOSI package (not via the enterprise-values.yaml package list). Create a values.yaml:

# values.yaml
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik
      externalIPs: []        # set your ingress IP(s)
    advanced: {}

Install it (set ingressName: traefik in cert-manager):

# non-airgap
kosi search --hub kubeops --ps traefik
kosi install --hub kubeops kubeops/traefik:<version> -f values.yaml --dname traefik

# airgap
podman login <harbor ip>:<nodePort> -u <user> -p <pass> --tls-verify=false
kosi pull --hub kubeops kubeops/traefik:<version> -o traefik-<version>.tgz -r <harbor ip>:<nodePort>/kubeops -t localhost:<nodePort>/kubeops
kosi install --dname traefik -p traefik-<version>.tgz -f values.yaml

Migrating from nginx to Traefik: install Traefik (above), then remove the old nginx-ingress and, if you need to reuse its NodePorts, edit the Traefik service to match:

kosi list                                                              # find the nginx deployment name
kosi delete --hub kubeops kubeops/ingress-nginx:<version> -f enterprise-values.yaml --dname <name>
kubectl edit svc traefik -n traefik                                    # set web/websecure nodePorts (e.g. 30080/30443)

9. Certificates and external access

Dashboards reachable from the internet with valid, browser-trusted certificates.

Bind the ingress to your public IP (externalIPs in the ingress package / Traefik values.yaml).
Configure the issuer — cert-manager creates a letsencrypt ClusterIssuer automatically and every ingress references it; you only provide:

- name: cert-manager
  values:
    ca:
      emailLetsEncrypt: <your_email@domain.com>
      ingressName: nginx        # or traefik

Create public DNS A-records pointing <domain_name> at your ingress public IP.
Keep hostname: <domain_name> on each dashboard package.

Because valid certificates are issued automatically as the components come up, you can install everything in one apply (single-phase — section 10).

⚠ Warning

Let’s Encrypt: the HTTP-01 challenge needs inbound port 80 and public DNS. Let’s Encrypt enforces rate limits — test against the staging endpoint first. Wildcard certificates require DNS-01, not HTTP-01.

The cluster is reachable only inside your network, or has no public ACME access (airgap). You provide a valid certificate yourself.

Create a TLS secret from a certificate your clients already trust and point cert-manager at it:

# example only — use a certificate from your corporate CA in production
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt -subj "/CN=<your domain>"
kubectl create secret tls example-ca --key="tls.key" --cert="tls.crt" -n cert-manager

- name: cert-manager
  values:
    standard:
      secretName: example-ca

Distribute the CA certificate to all clients and to the kube-apiserver so it trusts the Keycloak OIDC issuer. Set externalIPs and the hostname fields to your internal IPs/DNS names.

Because the valid certificate must exist before the OIDC dashboard setup runs, this scenario uses the two-phase deployment (section 10).

10. Deploy the platform stack

With Let’s Encrypt, valid certificates are issued automatically, so the whole stack can be applied at once. Keep the packages in dependency order in enterprise-values.yaml — storage and the cert/ingress components are defined before Keycloak, which is defined before the dashboards — then:

kubeopsctl apply -f enterprise-values.yaml

Wait until all components are deployed and running.

Note

If your build does not resolve install order automatically, bring up rook-ceph (storage) and then keycloak before the dashboards — see the staged approach in the Scenario B tab.

The OIDC dashboard integration needs a valid certificate to already exist; otherwise it creates a dummy secret. Deploy in two phases, controlled via the enabled flag of each package.

Phase 1 — storage, registry, ingress, cert-manager. Enable only these (others enabled: false) and apply:

- name: rook-ceph      # enabled: true
- name: harbor         # enabled: true
- name: ingress-nginx  # enabled: true   (or install Traefik per section 8)
- name: cert-manager   # enabled: true

kubeopsctl apply -f enterprise-values.yaml
kubectl get storageclass      # rook-cephfs present

Phase 1.5 — install the valid certificate (manual). Create the TLS secret and set cert-manager secretName (section 9, Scenario B), re-apply phase 1, and verify the certificate is trusted:

curl -I https://<domain_name>/

⚠ Warning

Do not proceed until a valid certificate is in place. Installing Keycloak/the dashboards without it creates a dummy secret and the OIDC integration will not work.

Phase 2 — Keycloak and the rest. Enable keycloak (it must be ready before the dashboards) and the remaining packages, then apply again:

kubeopsctl apply -f enterprise-values.yaml

11. Single Sign-On for Harbor and Rook-Ceph

Most dashboards (Grafana, Prometheus, OpenSearch, KubeOps Dashboard) are wired to OIDC by the integration. Harbor and Rook-Ceph are configured separately and must be set up manually after Keycloak is running. Both use a Keycloak realm named kubeops-dashboards with a user kubeops.

Harbor

In Keycloak (kubeops-dashboards realm) create a confidential client harbor (standard flow on, direct access grants off) with redirect URI https://<domain_name>/c/oidc/callback and web origin <domain_name>; copy the client secret.
Create a secret with the client id/secret in the Harbor namespace.
In the Harbor package’s advanced values, inject the OIDC client id/secret as env vars and set CONFIG_OVERWRITE_JSON with auth_mode: oidc_auth, oidc_endpoint: https://<domain_name>/keycloak/realms/kubeops-dashboards, scopes openid,profile,email, oidc_auto_onboard: true.

Full procedure: SSO for Harbor.

Rook-Ceph

Rook-Ceph uses an oauth2-proxy in front of the Ceph dashboard:

In Keycloak create a confidential client for the Ceph dashboard (redirect URI https://<domain_name>/oauth2/callback), copy the client secret, and generate a cookie secret.
Create the oauth2-proxy-credentials secret and install the oauth2-proxy Helm chart in the rook-ceph namespace.
Enable external auth on the Ceph manager (ceph config-key set mgr/dashboard/external_auth ...), add the oauth2 auth annotations to the ceph-dashboard ingress, and create the oauth2-proxy ingress.

Full procedure: SSO for rook-ceph.

Note The externalURL/hostname must match the DNS name exactly, and the OIDC client id/secret in Keycloak must match the values you configure in Harbor and oauth2-proxy.

12. Verify the cluster

kubectl get nodes -o wide                 # all nodes Ready
kubectl get pods -A                       # control-plane, networking and platform pods Running
kubectl get storageclass                  # rook-cephfs present
kubectl get clusterissuer                 # letsencrypt issuer (Scenario A)
kubectl get certificate -A                # certificates Ready
kubectl get ingress -A                    # ingress objects for the dashboards

Confirm every node is Ready, all component pods are Running, certificates are Ready, and the dashboards answer over HTTPS with a trusted certificate.

13. Access the dashboards

Reachable through the ingress under your domain (replace <domain_name>):

Dashboard	Path
Harbor	`<domain_name>/` (base domain)
Grafana	`<domain_name>/grafana`
Rook-Ceph	`<domain_name>/ceph-dashboard`
OpenSearch	`<domain_name>/opensearch`
Keycloak	`<domain_name>/keycloak`
KubeOps Dashboard	`<domain_name>/kubeops-dashboard`
Prometheus	`<domain_name>/prometheus`

For kubectl via OIDC, add the Keycloak provider to your kubeconfig (issuer URL, client id, a tool such as kubelogin) and bind Keycloak users/groups to least-privilege RBAC roles.

14. Back up the cluster (Velero)

Velero is part of the stack. It stores backups in an S3-compatible object store and can also back up persistent-volume data.

Important Out of the box Velero backs up only Kubernetes objects, on no schedule, and deletes each backup after 72h (node agent and volume snapshots are off). For production, drive this from enterprise-values.yaml (see section 7.1) rather than ad-hoc CLI: set useNodeAgent: true and useVolumeSnapshots: true in the standard block, and in advanced add a recurring schedules entry with a 30-day ttl plus configuration.defaultVolumesToFsBackup: true so PV data is included. These settings are re-applied on every kubeopsctl apply, so they survive upgrades.

Point Velero at an external S3 store

⚠ Warning Store backups off-cluster. The shipped s3Url: http://minio.velero.svc:9000 is only a placeholder — a backup that lives inside the very cluster it protects is useless for disaster recovery. Use an external S3 bucket (or a MinIO/Ceph-RGW instance outside this cluster). The bucket must already exist.

The velero package’s standard keys map directly to Velero’s credentials and BackupStorageLocation. Set them in enterprise-values.yaml:

- name: velero
  values:
    standard:
      provider: "aws"
      bucket: "<your-bucket>"                 # must already exist
      accessKeyId: "<aws access key>"
      secretAccessKey: "<aws secret key>"
      backupLocationConfig:
        region: "<aws-region>"
        s3ForcePathStyle: false        # no s3Url for real AWS
      useNodeAgent: true
      useVolumeSnapshots: true

- name: velero
  values:
    standard:
      provider: "aws"                          # the AWS plugin also drives S3-compatible stores
      bucket: "<your-bucket>"                  # must already exist on the external store
      accessKeyId: "<access key>"
      secretAccessKey: "<secret key>"
      backupLocationConfig:
        region: "default"
        s3ForcePathStyle: true
        s3Url: "https://s3.example.com"
      useNodeAgent: true
      useVolumeSnapshots: true

The credentials become the Velero cloud-credentials secret automatically.

Note If the external endpoint uses a private CA, Velero needs its CA bundle — this is not exposed as a standard key. Use a publicly trusted certificate, or add caCert to the BackupStorageLocation manually (kubectl edit backupstoragelocation default -n velero); note the manual edit is not values-persistent.

How backups work

Scheduled — the daily schedule from the advanced.schedules block (section 7.1) runs automatically and each backup expires after its ttl (30 days in the example).
On demand — velero backup create <name> (add --include-namespaces / --selector to scope it).
What is captured — Kubernetes objects always; persistent-volume data as well, because useNodeAgent: true and configuration.defaultVolumesToFsBackup: true are set. Stateful components (Keycloak, Harbor Postgres) additionally dump their database to /backup/*.dump via a pre-backup hook, so the dump travels inside the volume backup.
Inspect — velero backup get, velero backup describe <name> --details, velero backup logs <name>.

How restore (rollback) works

velero backup get                                          # list available backups
velero restore create --from-backup <BACKUP or SCHEDULE-TIMESTAMP>          # full restore
velero restore create --from-backup <BACKUP> --include-namespaces <ns>      # selective restore
velero restore describe <restore-name>                     # check status

Rollback a namespace/app by restoring the relevant backup; Velero does not overwrite resources that still exist, so restore into a clean namespace.
Disaster recovery (whole cluster) — build a fresh cluster, install KubeOps with the same external S3 credentials/bucket in the velero package, then velero backup get already lists your existing backups and you restore with velero restore create --from-backup <name>.
Databases — after the namespace is restored, run the app-specific restore from the dumped file (e.g. Keycloak/Harbor pg_restore from /backup/*.dump).
The rook-cephfs storage class must exist in the target cluster before restoring PVCs.

Per-component backup/restore commands (Harbor, Prometheus, the logging stack, OPA, Rook-Ceph …) with their exact namespace/label selectors, and the database-restore steps, are in the Backup and Restore guide.

15. Production operations notes

Network isolation with a default-deny NetworkPolicy

Every enterprise tool installs its own NetworkPolicy (applied directly by the installer, not through the Helm chart) that allows exactly the traffic that component needs — including DNS egress to kube-system / kube-dns. These allow-policies exist precisely so the platform keeps working once a deny-all policy is in place. Rook-Ceph ships an allow-all policy for its own namespace.

KubeOps does not apply a cluster-wide deny-all itself. For production we recommend adding a default-deny NetworkPolicy in every namespace so that traffic is blocked unless a policy explicitly allows it:

# default-deny-all.yaml — the namespace is taken from `kubectl -n` on apply
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

NetworkPolicy is namespaced, so roll it out across all namespaces, e.g.:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n "$ns" apply -f default-deny-all.yaml
done

⚠ Warning

The tool-provided allow-policies are intentionally permissive (several allow egress to 0.0.0.0/0 on specific ports) — a deny-all NetworkPolicy complements, but does not replace, the node firewall (section 2).
The KubeOps Dashboard does not currently apply its own allow-policy, so under a strict deny-all you must add an allow-NetworkPolicy for its namespace manually (at least DNS egress and ingress from the ingress controller).
When you deploy your own workloads into a managed or new namespace, add explicit allow-NetworkPolicy rules (DNS, ingress controller, dependent services) or they cannot communicate.

Further operational notes

Inter-node encryption: with Calico as the CNI, pod-to-pod traffic between nodes is encrypted with WireGuard (UDP 51820 must stay open between nodes — see section 2).
API audit logging: kubeopsctl enables Kubernetes API audit logging at level: Metadata, written to /var/log/kubernetes/audit/ on the control-plane nodes. For stricter compliance needs, replace it with a richer audit policy and forward the logs into your logging stack.
Monitoring: use Prometheus/Grafana with alerting; watch certificate expiry and Ceph capacity.
Updates in tranches: drain and update one zone at a time; keep at least one control plane on the higher version during an upgrade.
Keep packages pinned: keep the OS pinned and all KOSI-installed packages held at their validated versions (see Installed Package Versions for RHEL / Ubuntu) so an OS update cannot upgrade podman, containerd or the Kubernetes tools out from under the cluster.
Rotate credentials for the platform components and keep them in secrets.

1.2 - Supported Package Versions — Ubuntu / Debian

The packages and versions provided by the KubeOps package repository for Ubuntu / Debian, and how to pin them.

This page lists the packages and versions available from the KubeOps package repository for Ubuntu 24.04 (the .deb pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Important Not all of these versions are installed at once. The Kubernetes version is selected with kubernetesVersion in cluster-values.yaml (and kubeopsctl pull --kubernetesVersion <x.y.z>); kubeopsctl then installs the matching kubeadm, kubelet, kubectl, cri-tools and kubernetes-cni. For OIDC (dashboard SSO) you need kubeopsctl 2.1.0.0 or newer.

Kubernetes

Package	Supported versions
`kubeadm`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubelet`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubectl`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubernetes-cni`	1.4.0, 1.5.1, 1.6.0, 1.7.1, 1.8.0, 1.9.1
`cri-tools`	1.30.0, 1.31.0, 1.32.0, 1.33.0, 1.34.0, 1.35.0, 1.36.0

KubeOps tools

Package	Supported versions
`kosi`	2.12.1.0, 2.13.0.1, 2.13.0.2
`kubeopsctl`	2.0.0.0, 2.0.1.0, 2.0.2.0, 2.0.3.0, 2.0.4.0, 2.0.5.0, 2.1.0.0

Helm

Package	Supported versions
`helm`	3.17.4

Container runtime and etcd

Package	Supported versions
`containerd`	1.7.24
`etcdctl`	3.5.16, 3.5.21
`etcdutl`	3.5.16, 3.5.21
`podman`	4.9.3

Pre-release versions

⚠ Warning The following are pre-release builds (Alpha / Beta / RC) and are not intended for production.

Package	Pre-release versions
`kosi`	2.14.0.4-Beta0, 2.15.0.4-Alpha5, 2.15.0.4-Beta0
`kubeopsctl`	2.2.0.0-Alpha5, 2.2.0.0-Beta0

Pin the installed versions

After the packages are installed, hold them at the installed version so a later apt upgrade cannot upgrade them and break the cluster:

sudo apt-mark hold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

Note kubeopsctl already holds kubelet, kubeadm and kubectl itself (it runs apt-mark hold/unhold on them around installs and upgrades), so for those three the hold is managed for you. The command above additionally covers kubernetes-cni, cri-tools, helm, kosi and kubeopsctl.

To release a hold later (for a controlled upgrade):

sudo apt-mark unhold \
  containerd podman \
  kubelet kubeadm kubectl kubernetes-cni cri-tools \
  helm kosi kubeopsctl

Note containerd is installed from the operating system repository or as a local package. podman is installed as a KOSI lima package. Both packages are included in the hold and unhold commands above. To generate the exact hold list from your repository, run the list-package-versions.sh helper with --pin.

1.3 - Supported Package Versions — Red Hat Enterprise Linux (RHEL)

The packages and versions provided by the KubeOps package repository for Red Hat Enterprise Linux, and how to pin them.

This page lists the packages and versions available from the KubeOps package repository for RHEL 9.6 / 9.7 (the .rpm pool). Use it to see which versions are supported and to pin the installed versions so an OS update cannot upgrade them.

Kubernetes

Package	Supported versions
`kubeadm`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubelet`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubectl`	1.30.8, 1.31.6, 1.32.2, 1.32.3, 1.32.9, 1.32.10, 1.33.3, 1.33.5, 1.33.10, 1.34.1, 1.34.6, 1.35.0, 1.36.1
`kubernetes-cni`	1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.1, 1.8.0, 1.9.1
`cri-tools`	1.30.1, 1.31.1, 1.32.0, 1.33.0, 1.34.0, 1.35.0, 1.36.0

Container runtime and etcd

Package	Supported versions
`containerd.io`	2.1.5
`podman`	5.2.2
`etcdctl`	3.5.16, 3.5.21
`etcdutl`	3.5.16, 3.5.21

KubeOps tools

Package	Supported versions
`kosi`	2.12.1.0, 2.13.0.1, 2.13.0.2
`kubeopsctl`	2.0.1.0, 2.0.2.0, 2.0.3.0, 2.0.4.0, 2.0.5.0, 2.1.0.0

Helm

Package	Supported versions
`helm`	3.17.4

Pre-release versions

⚠ Warning The following are pre-release builds (Alpha / Beta / RC) and are not intended for production.

Package	Pre-release versions
`kosi`	2.14.0.4_Beta0, 2.15.0.4_Alpha5, 2.15.0.4_Beta0
`kubeopsctl`	2.2.0.0_Alpha5, 2.2.0.0_Beta0

Pin the installed versions

On RHEL the KubeOps package repository is configured so that its packages are excluded from normal dnf transactions — the install commands have to pass --disableexcludes=kubeops-repo to override that exclusion. As a result a regular dnf update already leaves these packages untouched, so the cluster is protected out of the box.

⚠ Warning Confirm the exact exclude= configuration in kubeops.repo (and /etc/dnf/dnf.conf) for your release. The versionlock step below is an additional safeguard on top of that exclusion.

To lock the installed versions explicitly, use the versionlock plugin after the packages are installed:

sudo dnf install -y python3-dnf-plugin-versionlock
sudo dnf versionlock add \
  containerd.io podman kubelet kubeadm kubectl kubernetes-cni cri-tools \
  etcdctl etcdutl helm kosi kubeopsctl conntrack-tools

Alternatively, exclude them explicitly on each update:

sudo dnf update --exclude=containerd.io,podman,kubelet,kubeadm,kubectl,kubernetes-cni,cri-tools,etcdctl,etcdutl,helm,kosi,kubeopsctl

Dependencies

The repository also provides the following compatibility/dependency packages (32-bit i686 runtime libraries), which are pulled in automatically: conntrack-tools (1.4.7), glibc (2.34), glibc-gconv-extra (2.34), libgcc (11.5.0), libmnl (1.0.4), libnetfilter_cthelper (1.0.0), libnetfilter_cttimeout (1.0.0), libnetfilter_queue (1.0.5), libnfnetlink (1.0.1).

Note These are operating-system libraries. Do not version-lock glibc and the other system libraries — they should continue to receive OS security updates. Pin only the KubeOps and Kubernetes stack packages listed above.

2 - Joining a Node in a Kubernetes cluster

This guide outlines the steps to join a nodes to a cluster.

Note This guide requires a Compliance enterprise license.

To increase performance or add additional resource capacity to your cluster, adding a node to the cluster is the correct process. This process with kubeopsctl is very easy.
You can use the following steps to join control-plane nodes or worker-nodes to a Kubernetes cluster.

Join Node Process:

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

ETCD Backup Recommendation

Before performing changes on the control planes, it is recommended to create an ETCD backup. Refer to the official Kubernetes documentation for details here

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull a specific Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-controlplaneXX
    iPAddress: 10.2.10.XXX
    type: controlplane
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone1
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone1.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6       # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true             # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-controlplaneXX   # -> has to be changed
    iPAddress: 10.2.10.XXX      # -> has to be changed
    type: controlplane
    kubeVersion: 1.31.6         # -> check with actual version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. If you are ready just start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

Example 1: Joining a Worker Node to a Kubernetes Cluster

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion x.xx.x you can pull other Kubernetes versions.
Available Kubernetes versions are

kubeopsctl pull

2. Add your node definition/specifications in the cluster-values

Note If you want to join a node to your cluster the parameter: changeCluster has set to true

  - name: demo-workerXX
    iPAddress: 10.2.10.XX
    type: worker
    kubeVersion: 1.31.6

Note The parameter kubeVersion defines the kubernetes version in your cluster for the node. This must set to the same version which exists in your cluster.

3. Adjust your cluster-values in zone2
Adjust your cluster-values in comparison to the example below. Be sure to set the actual version in your values, as well as the target version in the nodes. In the snippet below it is just the zone2.

Note Take care of your format an syntax in the cluster-values.yaml

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6        # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true              # -> has to be set
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
  - name: demo-workerXX          # -> has to be changed
    iPAddress: 10.2.10.XX        # -> has to be changed
    type: worker
    kubeVersion: 1.31.6          # -> check with actual version     
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

3. Validate your values and join the node to the cluster

Once the cluster-values.yaml is created, check the values once again. Before joining the node, note the following:

Note

It is recommended if you join worker nodes to a cluster and rook-ceph is installed, to adjust your rook-ceph CRUSH map. Refer to the official rook-ceph documentation for details here. for updating the crush map, you should set in your enterprise-values.yaml:

...
packages:
- name: rook-ceph
  enabled: true
...

If you are ready, start the join node process with the command:

kubeopsctl apply -f cluster-values.yaml

If you need to apply the CRUSH map adjustment from above, use the following command instead:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

3 - Deleting a Node from a Kubernetes cluster

This guide outlines the steps to delete worker-nodes from a cluster, specifically how to proceed with rook-ceph and other KubeOps Compliance applications

Deleting a Node from a Kubernetes cluster

Note This guide requires a Compliance enterprise license.

In rare cases, it may be necessary to remove nodes from a Kubernetes cluster. This how-to guide explains the prerequisites and the key considerations to keep in mind before starting the node removal process.

You can use the following steps to delete nodes from a Kubernetes cluster.

Prerequisites

In order to run rook-ceph stable for a longer period your cluster needs at least 3 zones with each zone containing at least 1 worker-node
To check which mon and osd is running on the node you want to delete you can use the command kubectl get pods -n rook-ceph -o wide | grep <node-name> | grep "mon\|osd" | grep -v "osd-prepare" | awk '{print $1}'. As an output you get the mon and the osd running on that node. If you don’t get an output, you don’t have to delete the ressource and can skip to the “delete the node”-section

Worker

Important: Due to rook-ceph, a worker node must not be removed without following the steps below. In this example, worker01 (zone1) is removed from the cluster. Worker01 contains osd.0 and mon-c.

Scale down the rook-ceph-operator deployment to 0

This prevents new MONs or OSDs from being created.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=0

Check which hosts and OSDs belong to each zone

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

ID   CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
 -1         0.21478  root default
 -9         0.04880      zone zone1
 -7         0.04880          host worker01                              # worker01 is being removed
  0    ssd  0.04880              osd.0          up   1.00000  1.00000   # osd.0 is being removed
-15         0.04880          host worker04
  3    ssd  0.04880              osd.3          up   1.00000  1.00000
-11         0.10739      zone zone2
 -3         0.05859          host worker02
  1    ssd  0.05859              osd.1          up   0.95001  1.00000
-13         0.05859      zone zone3
 -5         0.05859          host worker03
  2    ssd  0.05859              osd.2          up   0.95001  1.00000

From this output you can see that osd.0 is part of worker01.

Scale down the OSD deployment

kubectl scale deploy -n rook-ceph rook-ceph-osd-<x> --replicas=0
# Example: kubectl scale deploy -n rook-ceph rook-ceph-osd-0 --replicas=0

Remove the OSD via ceph-tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- bash
# show OSD tree
ceph osd tree
# mark OSD out
ceph osd out <x>
# Example: ceph osd out 0
ceph osd purge <x> --yes-i-really-mean-it
# Example: ceph osd purge 0 --yes-i-really-mean-it
ceph auth del osd.<x>
# adjust CRUSH map
ceph osd crush remove <nodename>
# exit from ceph-tools
exit
# show OSD tree (now without the deleted node)
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph osd tree

Check the MON running on the node

Before deleting a MON deployment, check which MON is running on the node you want to remove.

kubectl -n rook-ceph get pods -l app=rook-ceph-mon -o wide

Delete OSD and MON deployments

After checking the OSD and MON running on the node, delete the corresponding deployments.

kubectl delete deploy -n rook-ceph rook-ceph-osd-<x> rook-ceph-mon-<y>

Example

kubectl delete deploy -n rook-ceph rook-ceph-osd-0 rook-ceph-mon-c

Remove the deleted mon from the ceph tools

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm <y>
# verify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

Example

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon rm c
# verfify
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph mon dump

This is the dump before executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b
2: [v2:192.168.196.110:3300/0,v1:192.168.196.110:6789/0] mon.c

This is the dump after executing the remove:

0: [v2:192.168.231.184:3300/0,v1:192.168.231.184:6789/0] mon.a
1: [v2:192.168.185.9:3300/0,v1:192.168.185.9:6789/0] mon.b

Delete the node from the kubernetes cluster

Remove the node you want to delete from the cluster-values.yaml file.
After updating the file, apply the deletion with the following command:

kubeopsctl apply --delete -f cluster-values.yaml

Example

The cluster-values.yaml without node1 but with node4

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6      
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker04
    iPAddress: 10.2.10.214
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.31.6       
- name: zone3
  nodes:
  - name: controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6  
  - name: worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

After updating the cluster-values.yaml, execute the delete command:

kubeopsctl apply --delete -f cluster-values.yaml

Scale the rook-ceph-operator deployment back to 1

This allows a new MON to be created automatically in zone2.

kubectl scale deploy rook-ceph-operator -n rook-ceph --replicas=1

Timing and health checks

The total duration depends on cluster size and node performance. Before proceeding, verify Ceph health and placement groups are clean.

kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph status
kubectl exec -it deploy/rook-ceph-tools -n rook-ceph -- ceph pg stat

Typical duration ranges from 15 to 120 minutes.

If you want to rejoin the same node, reset it to a time prior to joining the cluster. Only this way you can be sure, that no leftovers from the deletion process remain!

4 - Single Sign-On with Keycloak

Learn how to configure Keycloak for Single Sign-On, securely expose it using Kubernetes Ingress and TLS, and integrate it with kubeops and other Kubernetes applications.

In this guide, you will learn how to implement Single Sign-On (SSO) using Keycloak. We will walk through the complete flow—from understanding SSO for platforms and services such as Rook Ceph, Harbor, OpenSearch, and other Kubernetes applications, to configuring Keycloak, exposing it securely, and integrating it with kubeops.

By the end of this guide, you will be able to:

Understand how Keycloak enables centralized authentication
Configure Keycloak for SSO
Securely expose Keycloak using Kubernetes Ingress and TLS
Integrate Keycloak with kubeops for authentication and authorization
Validate and troubleshoot the SSO login flow

Let’s get started on enabling secure and seamless authentication with Keycloak.

4.1 - Single Sign-On (SSO) with Keycloak for Harbor

Learn how to configure Single Sign-On (SSO) for Harbor using Keycloak with OIDC in a Kubernetes environment.

This guide describes how to configure Harbor authentication using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Prerequisites

Before proceeding, ensure the following requirements are met:

A running Kubernetes cluster
kubectl installed and configured.
Keycloak is already installed and running
Keycloak is exposed using Kubernetes Ingress
A valid DNS record is configured for Keycloak and Harbor
TLS is enabled with a trusted Certificate Authority (CA)

Step 1: Prepare Keycloak (Realm, User, and Client)

In this step, we configure Keycloak for Harbor SSO. Keycloak is assumed to be already installed, exposed via Ingress, and reachable over HTTPS.

Create Realm

Ensure a realm named kubeops-dashboards exists.
If it does not exist, create it in the Keycloak admin console.

Realm name: kubeops-dashboards
Enabled: true

Create User

Ensure a user named kubeops exists in the kubeops-dashboards realm.
If the user does not exist, create it and set credentials.

Username: kubeops
Enabled: true
Set a permanent password

Create Client (Harbor)

Create a client for Harbor in the kubeops-dashboards realm.

Client ID: harbor
Client type: OpenID Connect
Client authentication: Enabled
Standard flow: Enabled
Direct access grants: Disabled

Valid Redirect URIs

Add the following redirect URI:

https://<your_DNS_name>/c/oidc/callback

Web Origins

<your_DNS_name>

Client Secret

After creating the client, copy the client secret.
This value will be used in the Harbor configuration:

oidc_client_id: harbor
oidc_client_secret: <CLIENT_SECRET>

Create Secret

⚠ Warning You may use a different name for the oidc-harbor secret. If you do so, adapt secretKeyRef.name in the Harbor values accordingly.

kubectl create secret generic oidc-harbor \
    --namespace <your_harbor_namespace> \
    --from-literal=client_id=<your_oidc_client_id> \
    --from-literal=client_secret=<your_oidc_client_secret>

Step 2: Prepare Harbor Values

The following kubeops package configuration enables Harbor and integrates it with Keycloak using OIDC authentication.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1

deleteNs: false
localRegistry: false

packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: <you_harbor_namespace>
        harborpass: "password"
        databasePassword: "password"
        redisPassword: "password"
        externalURL: https://<your_DNS_name>
        nodePort: 30002
        hostname: <your_DNS_name>
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi
              storageClass: "rook-cephfs"
            jobservice:
              jobLog:
                size: 1Gi
                storageClass: "rook-cephfs"
            database:
              size: 1Gi
              storageClass: "rook-cephfs"
            redis:
              size: 1Gi
              storageClass: "rook-cephfs"
            trivy:
              size: 5Gi
              storageClass: "rook-cephfs"

      advanced:
        core:
          extraEnvVars:
            - name: OIDC_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: oidc-harbor 
                  key: client_id
            - name: OIDC_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: oidc-harbor 
                  key: client_secret
            - name: CONFIG_OVERWRITE_JSON
              value: |
                {
                  "auth_mode": "oidc_auth",
                  "oidc_name": "keycloak",
                  "oidc_endpoint": "https://<your_DNS_name>/keycloak/realms/kubeops-dashboards",
                  "oidc_client_id": "$(OIDC_CLIENT_ID)",
                  "oidc_client_secret": "$(OIDC_CLIENT_SECRET)",
                  "oidc_scope": "openid,profile,email",
                  "oidc_verify_cert": true,
                  "oidc_auto_onboard": true
                }

Notes

Ensure the OIDC client in Keycloak matches the oidc_client_id and oidc_client_secret values.
The externalURL and hostname must match the Harbor DNS name exactly.
oidc_auto_onboard: true allows users to be created automatically in Harbor upon first login.

4.2 - Single Sign-On (SSO) with Keycloak for KubeOps Dashboard

Learn how to configure Single Sign-On (SSO) for KubeOps Dashboard using Keycloak with OIDC.

This guide describes how to configure KubeOps Dashboard using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Prerequisites

Before proceeding, ensure the following requirements are met:

A running Kubernetes cluster
kubectl installed and configured.
Keycloak is already installed and running

Step 1: Extract Keycloak CA certificate

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

On your admin host, run the OpenSSL command (kept exactly as provided):

  openssl s_client -showcerts -connect <your_DNS_name>:443 /dev/null | openssl x509 -outform PEM > keycloak-ca.crt

Copy the CA certificate to each master

scp @:/etc/kubernetes/pki/

Step 2: Update kube-apiserver yaml

On every master, edit the yaml : /etc/kubernetes/manifests/kube-apiserver.yaml

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

spec:

  containers:

  - command:


    - --oidc-issuer-url=https://<your_DNS_name>/keycloak/realms/master

    - --oidc-client-id=headlamp

    - --oidc-username-claim=preferred_username

    - --oidc-groups-claim=groups

    - "--oidc-username-prefix=oidc:"

    - "--oidc-groups-prefix=oidc:"

    - --oidc-ca-file=/etc/kubernetes/pki/keycloak-ca.crt

Step 3: Create a Keycloak client for Headlamp

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

Create a client for headlamp
- Client ID: headlamp
- Client type: OpenID Connect
- Access type: Confidential
- Client authentication: Enabled
- Standard flow: Enabled
- Direct access grants: Disabled
Valid Redirect URIs

Add the following redirect URI:
```
https://headlamp/<your_DNS_name>/*
```
Web Origins
```
<your_DNS_name>
```

Step 4: Create a client scope for Headlamp

Create a client scope
- Assigned Client Scope : headlamp-dedicated
For groups, use the Group Mapper in Keycloak:
- Mapper Type: groups
- Name: groups
- Token Claim Name: groups
- Add to ID token: ON
- Add to access token: ON
- Add to user info: ON
- Add to token introspection: ON

Step 5: Create a user Group and user in Keycloak

Create a group named headlamp (if doesn’t exist already) and user under the group.

Step 6: Create ClusterRoleBinding for Headlamp group

1.Use following yaml to create ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: headlamp-admin-user
subjects:
- kind: Group
  name: "oidc:headlamp" # Der 'sub' oder 'preferred_username' from the Keycloak-Token
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

The name “oidc:headlamp” needs to be the same as the group name.

Apply the ClusterRoleBinding file

    kubectl apply -f headlamp-clusterrolebinding.yaml

Step 7: Get client secret

After creating the client, copy the client secret.
This value will be used in the next step.

Step 8: Prepare Headlamp values (enterprise.yaml)

configure enterprise-yaml

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

packages:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      service:
        nodePort: 30007
      hostname: "<your_DNS_name>"
      path: "/"
    advanced:
      config:
        extraArgs:
          - "--in-cluster"
          - "--plugins-dir=/headlamp/plugins"
          - "--oidc-client-id=headlamp"
          - "--oidc-idp-issuer-url=https://<your_DNS_name>/keycloak/realms/master"
          - "--oidc-scopes=openid,profile,email"
          - "--insecure-ssl"
          - "--oidc-client-secret=<client-secret>"

Replace with the secret retrieved in Step 7.
-oidc-client-id must match the Keycloak client name (headlamp).

Step 9: Install Headlamp

Deploy Headlamp with the updated enterprise.yaml.

4.3 - Single Sign-On (SSO) with Keycloak for OpenSearch

Learn how to log in to OpenSearch and OpenSearch Dashboards with Single Sign-On (SSO) using Keycloak (OIDC).

This guide describes how to log in to OpenSearch Dashboards using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Unlike some other integrations, you do not have to create the Keycloak realm, client, roles or mapper by hand — the opensearch-dashboards package provisions them automatically during install/update. Username/password login remains available as a fallback.

Prerequisites

A running Kubernetes cluster with kubectl configured
Keycloak is installed and running (the keycloak package), exposed via Ingress and reachable over HTTPS
A valid DNS record and a trusted TLS certificate (e.g. Let’s Encrypt) for the ingress hostname
The OpenSearch hostname is set to that public host and is identical for the opensearch-os and opensearch-dashboards packages

Note If the hostname is a *.local name (not reachable), SSO is skipped automatically and only username/password login is configured.

Step 1: Configure the packages

Set the shared hostname and (optionally) the adminPassword for both OpenSearch packages in your enterprise-values.yaml:

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1

deleteNs: false
localRegistry: false

packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        namespace: logging
        hostname: <your_DNS_name>          # must match opensearch-dashboards and Keycloak host
        adminPassword: "<admin-password>"  # optional, default Password@@123456
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        namespace: logging
        nodePort: 30050
        hostname: <your_DNS_name>          # must match opensearch-os
        adminPassword: "<admin-password>"  # must match opensearch-os

Apply the configuration:

kubeopsctl apply -f enterprise-values.yaml

Step 2: What the package provisions automatically

During install/update the package configures Keycloak and OpenSearch for you:

Realm: kubeops-dashboards
Client: opensearch-dashboards (OpenID Connect, confidential, standard flow)
User: kubeops (default password password)
Realm roles: all_access, opensearch_dashboards_user
Protocol mapper: exposes the realm roles as a roles claim so OpenSearch can map them
OpenSearch backend security config: an openid authentication domain (Keycloak) plus a basic domain (username/password fallback), and a role mapping from the Keycloak realm roles to OpenSearch roles

The Keycloak admin credentials used for this are read from the keycloak-kubeops secret.

Step 3: Grant a user access

To let a Keycloak user log in to OpenSearch, assign them one of the realm roles in the kubeops-dashboards realm:

all_access — full access (maps to the OpenSearch all_access role)
opensearch_dashboards_user — dashboards access (maps to the kibana_user role)

You can use the pre-created kubeops user, or assign the role to your own Keycloak users.

Step 4: Log in

Open the OpenSearch Dashboards URL:

https://<your_DNS_name>/opensearch/

SSO: choose Log in with single sign-on → authenticate in Keycloak.
Fallback: use username admin and the adminPassword value (default Password@@123456).

Notes / Troubleshooting

hostname mismatch: the opensearch-os and opensearch-dashboards hostnames must be identical and must be the public Keycloak host, otherwise the OpenSearch backend cannot validate the token.
401 Unauthorized on SSO login: the OpenSearch backend could not validate the Keycloak token. Check that hostname is set and reachable from the OpenSearch pods, and that Keycloak’s TLS certificate is trusted (e.g. Let’s Encrypt). Verify the backend security config contains the openid domain:
```
curl -k -u admin:<adminPassword> \
  https://opensearch-cluster-master:9200/_plugins/_security/api/securityconfig
```
*.local hostname: SSO is intentionally skipped; only username/password login is available.
Changing the admin password: see Changing the OpenSearch password.

4.4 - Single Sign-On (SSO) with Keycloak for rook-ceph

Learn how to configure Single Sign-On (SSO) for rook-ceph using Keycloak with OIDC in a Kubernetes environment.

This guide describes how to configure rook-ceph authentication using Keycloak (OIDC) in a kubeops-managed Kubernetes environment.

Prerequisites

Before proceeding, ensure the following requirements are met:

A running Kubernetes cluster
kubectl installed and configured.
Keycloak is already installed and running
rook-ceph is already installed and running

Step 1: Prepare Keycloak (Realm, User)

To configure Keycloak for rook-ceph SSO

Create Realm

Ensure a realm named kubeops-dashboards exists.
If it does not exist, create it in the Keycloak admin console.

Realm name: kubeops-dashboards
Enabled: true

Create User

Ensure a user named kubeops exists in the kubeops-dashboards realm.
If the user does not exist, create it and set credentials.

Username: kubeops
Enabled: true
Set a permanent password

Step 2: Create Client (rook-ceph)

Create a client for rook-ceph in the kubeops-dashboards realm with following settings.

Client ID: ceph-dashboard
Client type: OpenID Connect
Access type: Confidential
Client authentication: Enabled
Standard flow: Enabled
Direct access grants: Disabled

Valid Redirect URIs

Add the following redirect URI:

https://<your_DNS_name>/oauth2/callback

Web Origins

Also update the web-origins

<your_DNS_name>

Step 3: Get Client Secret

In the Keycloak admin console, open the rook-ceph client and copy the client secret. This value will be used by oauth2-proxy and referenced in next steps:

oidc_client_id: ceph-dashboard
oidc_client_secret: <client-secret>

Generate a secure random cookie secret.

python3 -c 'import os,base64; print(base64.urlsafe_b64encode(os.urandom(32)).decode())'

Create a Kubernetes Secret containing OAuth2 credentials. Note: the example command below uses client-id=“ceph-dashboard” — verify this value matches your Keycloak client ID

Do not forget add the correct <client-secret> and <cookie-secret>

kubectl create secret generic oauth2-proxy-credentials \
  --from-literal=client-id="ceph-dashboard" \
  --from-literal=client-secret="<client-secret>" \
  --from-literal=cookie-secret="<cookie-secret>" \
  -n rook-ceph

Step 5: Prepare values for oauth2proxy

The following kubeops values configuration enables rook-ceph and integrates it with Keycloak using OIDC authentication. Use the client secret and Cookie secret derived in above steps here.

Important: Replace all <your_DNS_name> placeholders with your correct DNS name. Also use the correct <client-secret> and <cookie-secret>

global:
  # Global registry to pull the images from
  imageRegistry: ""
  # To help compatibility with other charts which use global.imagePullSecrets.
  imagePullSecrets: []
  #   - name: pullSecret1
  #   - name: pullSecret2
## Override the deployment namespace
##
namespaceOverride: ""
# Force the target Kubernetes version (it uses Helm `.Capabilities` if not set).
# This is especially useful for `helm template` as capabilities are always empty
# due to the fact that it doesn't query an actual cluster
kubeVersion:
# Oauth client configuration specifics
config:
  # Add config annotations
  annotations: {}
  # OAuth client ID
  clientID: "ceph-dashboard"
  # OAuth client secret
  clientSecret: "<client-secret>"
  # List of secret keys to include in the secret and expose as environment variables.
  # By default, all three secrets are required. To exclude certain secrets
  # (e.g., when using federated token authentication), remove them from this list.
  # Example to exclude client-secret:
  # requiredSecretKeys:
  #   - client-id
  #   - cookie-secret
  requiredSecretKeys:
    - client-id
    - client-secret
    - cookie-secret
  # Create a new secret with the following command
  # openssl rand -base64 32 | head -c 32 | base64
  # Use an existing secret for OAuth2 credentials (see secret.yaml for required fields)
  # Example:
  # existingSecret: secret
  cookieSecret: "<cookie-secret>"
  # The name of the cookie that oauth2-proxy will create
  # If left empty, it will default to the release name
  cookieName: ""
  google: {}
    # adminEmail: xxxx
    # useApplicationDefaultCredentials: true
    # targetPrincipal: xxxx
    # serviceAccountJson: xxxx
    # Alternatively, use an existing secret (see google-secret.yaml for required fields)
    # Example:
    # existingSecret: google-secret
    # groups: []
    # Example:
    #  - group1@example.com
    #  - group2@example.com
  # Default configuration, to be overridden
  configFile: |-
    provider = "keycloak-oidc"
    oidc_issuer_url = "https://<your_DNS_name>/keycloak/realms/master"
    email_domains = [ "*" ]
    upstreams = [ "file:///dev/null" ]    
   
    pass_user_headers = true
    set_xauthrequest = true
    pass_access_token = true
  # Custom configuration file: oauth2_proxy.cfg
  # configFile: |-
  #   pass_basic_auth = false
  #   pass_access_token = true
  # Use an existing config map (see configmap.yaml for required fields)
  # Example:
  # existingConfig: config
alphaConfig:
  enabled: false
  # Add config annotations
  annotations: {}
  # Arbitrary configuration data to append to the server section
  serverConfigData: {}
  # Arbitrary configuration data to append to the metrics section
  metricsConfigData: {}
  # Arbitrary configuration data to append
  configData: {}
  # Arbitrary configuration to append
  # This is treated as a Go template and rendered with the root context
  configFile: ""
  # Use an existing config map (see secret-alpha.yaml for required fields)
  existingConfig: ~
  # Use an existing secret
  existingSecret: "oauth2-proxy-credentials"
image:
  registry: ""
  repository: "oauth2-proxy/oauth2-proxy"
  # appVersion is used by default
  tag: ""
  pullPolicy: "IfNotPresent"
  command: []
# Optionally specify an array of imagePullSecrets.
# Secrets must be manually created in the namespace.
# ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets: []
  # - name: myRegistryKeySecretName
# Set a custom containerPort if required.
# This will default to 4180 if this value is not set and the httpScheme set to http
# This will default to 4443 if this value is not set and the httpScheme set to https
# containerPort: 4180
extraArgs:
  - --provider=keycloak-oidc
  - --set-xauthrequest=true
  - --pass-user-headers=true
  - --pass-access-token=true
  - --skip-oidc-discovery=true
  - --oidc-issuer-url=https://<your_DNS_name>/keycloak/realms/master
  - --login-url=https://<your_DNS_name>/keycloak/realms/master/protocol/openid-connect/auth
  - --redeem-url=https://<your_DNS_name>/keycloak/realms/master/protocol/openid-connect/token
  - --validate-url=https://<your_DNS_name>/keycloak/realms/master/protocol/openid-connect/userinfo
  - --oidc-jwks-url=https://<your_DNS_name>/keycloak/realms/master/protocol/openid-connect/certs
  - --ssl-insecure-skip-verify=true
  - --cookie-secure=true
extraEnv: []
envFrom: []
# Load environment variables from a ConfigMap(s) and/or Secret(s)
# that already exists (created and managed by you).
# ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables
#
# PS: Changes in these ConfigMaps or Secrets will not be automatically
#     detected and you must manually restart the relevant Pods after changes.
#
#  - configMapRef:
#      name: special-config
#  - secretRef:
#      name: special-config-secret
# -- Custom labels to add into metadata
customLabels: {}
# To authorize individual email addresses
# That is part of extraArgs but since this needs special treatment we need to do a separate section
authenticatedEmailsFile:
  enabled: false
  # Defines how the email addresses file will be projected, via a configmap or secret
  persistence: configmap
  # template is the name of the configmap what contains the email user list but has been configured without this chart.
  # It's a simpler way to maintain only one configmap (user list) instead changing it for each oauth2-proxy service.
  # Be aware the value name in the extern config map in data needs to be named to "restricted_user_access" or to the
  # provided value in restrictedUserAccessKey field.
  template: ""
  # The configmap/secret key under which the list of email access is stored
  # Defaults to "restricted_user_access" if not filled-in, but can be overridden to allow flexibility
  restrictedUserAccessKey: ""
  # One email per line
  # example:
  # restricted_access: |-
  #   name1@domain
  #   name2@domain
  # If you override the config with restricted_access it will configure a user list within this chart what takes care of the
  # config map resource.
  restricted_access: ""
  annotations: {}
  # helm.sh/resource-policy: keep
service:
  type: ClusterIP
  # when service.type is ClusterIP ...
  # clusterIP: 192.0.2.20
  # when service.type is LoadBalancer ...
  # loadBalancerIP: 198.51.100.40
  # loadBalancerSourceRanges: 203.0.113.0/24
  # when service.type is NodePort ...
  # nodePort: 80
  portNumber: 80
  # Protocol set on the service
  appProtocol: http
  annotations: {}
  # foo.io/bar: "true"
  # configure externalTrafficPolicy
  externalTrafficPolicy: ""
  # configure internalTrafficPolicy
  internalTrafficPolicy: ""
  # configure service target port
  targetPort: ""
  # Configures the service to use IPv4/IPv6 dual-stack.
  # Ref: https://kubernetes.io/docs/concepts/services-networking/dual-stack/
  ipDualStack:
    enabled: false
    ipFamilies: ["IPv6", "IPv4"]
    ipFamilyPolicy: "PreferDualStack"
  # Configure traffic distribution for the service
  # Ref: https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution
  trafficDistribution: ""
## Create or use ServiceAccount
serviceAccount:
  ## Specifies whether a ServiceAccount should be created
  enabled: true
  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the fullname template
  name:
  automountServiceAccountToken: true
  annotations: {}
  ## imagePullSecrets for the service account
  imagePullSecrets: []
    # - name: myRegistryKeySecretName
# Network policy settings.
networkPolicy:
  create: false
  ingress: []
  egress: []
ingress:
  enabled: false
  # className: nginx
  path: /
  # Only used if API capabilities (networking.k8s.io/v1) allow it
  pathType: ImplementationSpecific
  # Used to create an Ingress record.
  # hosts:
  # - chart-example.local
  # Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
  # Warning! The configuration is dependant on your current k8s API version capabilities (networking.k8s.io/v1)
  # extraPaths:
  # - path: /*
  #   pathType: ImplementationSpecific
  #   backend:
  #     service:
  #       name: ssl-redirect
  #       port:
  #         name: use-annotation
  labels: {}
  # annotations:
  #   kubernetes.io/ingress.class: nginx
  #   kubernetes.io/tls-acme: "true"
  # tls:
  # Secrets must be manually created in the namespace.
  # - secretName: chart-example-tls
  #   hosts:
  #     - chart-example.local
# Gateway API HTTPRoute configuration
# Ref: https://gateway-api.sigs.k8s.io/api-types/httproute/
gatewayApi:
  enabled: false
  # The name of the Gateway resource to attach the HTTPRoute to
  # Example:
  # gatewayRef:
  #   name: gateway
  #   namespace: gateway-system
  gatewayRef:
    name: ""
    namespace: ""
  # HTTPRoute rule configuration
  # rules:
  # - matches:
  #   - path:
  #       type: PathPrefix
  #       value: /
  rules: []
  # Hostnames to match in the HTTPRoute
  # hostnames:
  # - chart-example.local
  hostnames: []
  # Additional labels to add to the HTTPRoute
  labels: {}
  # Additional annotations to add to the HTTPRoute
  annotations: {}
resources: {}
  # limits:
  #   cpu: 100m
  #   memory: 300Mi
  # requests:
  #   cpu: 100m
  #   memory: 300Mi
# Container resize policy for runtime resource updates
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/
resizePolicy: []
  # - resourceName: cpu
  #   restartPolicy: NotRequired
  # - resourceName: memory
  #   restartPolicy: RestartContainer
extraVolumes: []
  # - name: ca-bundle-cert
  #   secret:
  #     secretName: <secret-name>
extraVolumeMounts: []
  # - mountPath: /etc/ssl/certs/
  #   name: ca-bundle-cert
# Additional containers to be added to the pod.
extraContainers: []
  #  - name: my-sidecar
  #    image: nginx:latest
# Additional Init containers to be added to the pod.
extraInitContainers: []
  #  - name: wait-for-idp
  #    image: my-idp-wait:latest
  #    command:
  #    - sh
  #    - -c
  #    - wait-for-idp.sh
priorityClassName: ""
# hostAliases is a list of aliases to be added to /etc/hosts for network name resolution
hostAliases: []
# - ip: "10.xxx.xxx.xxx"
#   hostnames:
#     - "auth.example.com"
# - ip: 127.0.0.1
#   hostnames:
#     - chart-example.local
#     - example.local
# [TopologySpreadConstraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) configuration.
# Ref: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#scheduling
# topologySpreadConstraints: []
# Affinity for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# affinity: {}
# Tolerations for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []
# Node labels for pod assignment
# Ref: https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector: {}
# Whether to use secrets instead of environment values for setting up OAUTH2_PROXY variables
proxyVarsAsSecrets: true
# Configure Kubernetes liveness and readiness probes.
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
# Disable both when deploying with Istio 1.0 mTLS. https://istio.io/help/faq/security/#k8s-health-checks
livenessProbe:
  enabled: true
  initialDelaySeconds: 0
  timeoutSeconds: 1
readinessProbe:
  enabled: true
  initialDelaySeconds: 0
  timeoutSeconds: 5
  periodSeconds: 10
  successThreshold: 1
# Configure Kubernetes security context for container
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
securityContext:
  enabled: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 2000
  runAsGroup: 2000
  seccompProfile:
    type: RuntimeDefault
deploymentAnnotations: {}
podAnnotations: {}
podLabels: {}
replicaCount: 1
revisionHistoryLimit: 10
strategy: {}
enableServiceLinks: true
## PodDisruptionBudget settings
## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
## One of maxUnavailable and minAvailable must be set to null.
podDisruptionBudget:
  enabled: true
  maxUnavailable: null
  minAvailable: 1
  # Policy for when unhealthy pods should be considered for eviction.
  # Valid values are "IfHealthyBudget" and "AlwaysAllow".
  # Ref: https://kubernetes.io/docs/tasks/run-application/configure-pdb/#unhealthy-pod-eviction-policy
  unhealthyPodEvictionPolicy: ""
## Horizontal Pod Autoscaling
## ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80
  annotations: {}
  # Configure HPA behavior policies for scaling if needed
  # Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configuring-scaling-behavior
  behavior: {}
    # scaleDown:
    #   stabilizationWindowSeconds: 300
    #   policies:
    #   - type: Percent
    #     value: 100
    #     periodSeconds: 15
    #   selectPolicy: Min
    # scaleUp:
    #   stabilizationWindowSeconds: 0
    #   policies:
    #   - type: Percent
    #     value: 100
    #     periodSeconds: 15
    #   - type: Pods
    #     value: 4
    #     periodSeconds: 15
    #   selectPolicy: Max
# Configure Kubernetes security context for pod
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
podSecurityContext: {}
# whether to use http or https
httpScheme: http
initContainers:
  # if the redis sub-chart is enabled, wait for it to be ready
  # before starting the proxy
  # creates a role binding to get, list, watch, the redis master pod
  # if service account is enabled
  waitForRedis:
    enabled: true
    image:
      repository: "alpine"
      tag: "latest"
      pullPolicy: "IfNotPresent"
    # uses the kubernetes version of the cluster
    # the chart is deployed on, if not set
    kubectlVersion: ""
    securityContext:
      enabled: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
      runAsGroup: 65534
      seccompProfile:
        type: RuntimeDefault
    timeout: 180
    resources: {}
      # limits:
      #   cpu: 100m
      #   memory: 300Mi
      # requests:
      #   cpu: 100m
      #   memory: 300Mi
# Additionally authenticate against a htpasswd file. Entries must be created with "htpasswd -B" for bcrypt encryption.
# Alternatively supply an existing secret which contains the required information.
htpasswdFile:
  enabled: false
  existingSecret: ""
  entries: []
  # One row for each user
  # example:
  # entries:
  #  - testuser:$2y$05$gY6dgXqjuzFhwdhsiFe7seM9q9Tile4Y3E.CBpAZJffkeiLaC21Gy
# Configure the session storage type, between cookie and redis
sessionStorage:
  # Can be one of the supported session storage cookie|redis
  type: cookie
  redis:
    # Name of the Kubernetes secret containing the redis & redis sentinel password values (see also `sessionStorage.redis.passwordKey`)
    existingSecret: ""
    # Redis password value. Applicable for all Redis configurations. Taken from redis subchart secret if not set. `sessionStorage.redis.existingSecret` takes precedence
    password: ""
    # Key of the Kubernetes secret data containing the redis password value. If you use the redis sub chart, make sure
    # this password matches the one used in redis-ha.redisPassword (see below).
    passwordKey: "redis-password"
    # Can be one of standalone|cluster|sentinel
    clientType: "standalone"
    standalone:
      # URL of redis standalone server for redis session storage (e.g. `redis://HOST[:PORT]`). Automatically generated if not set
      connectionUrl: ""
    cluster:
      # List of Redis cluster connection URLs. Array or single string allowed.
      connectionUrls: []
      # - "redis://127.0.0.1:8000"
      # - "redis://127.0.0.1:8001"
    sentinel:
      # Name of the Kubernetes secret containing the redis sentinel password value (see also `sessionStorage.redis.sentinel.passwordKey`). Default: `sessionStorage.redis.existingSecret`
      existingSecret: ""
      # Redis sentinel password. Used only for sentinel connection; any redis node passwords need to use `sessionStorage.redis.password`
      password: ""
      # Key of the Kubernetes secret data containing the redis sentinel password value
      passwordKey: "redis-sentinel-password"
      # Redis sentinel master name
      masterName: ""
      # List of Redis cluster connection URLs. Array or single string allowed.
      connectionUrls: []
      # - "redis://127.0.0.1:8000"
      # - "redis://127.0.0.1:8001"
# Enables and configure the automatic deployment of the redis-ha subchart
redis-ha:
  # provision an instance of the redis-ha sub-chart
  enabled: false
  # Redis specific helm chart settings, please see:
  # https://artifacthub.io/packages/helm/dandydev-charts/redis-ha#general-parameters
  #
  # Recommended:
  #
  # redisPassword: xxxxx
  # replicas: 1
  # persistentVolume:
  #   enabled: false
  #
  # If you install Redis using this sub chart, make sure that the password of the sub chart matches the password
  # you set in sessionStorage.redis.password (see above).
  #
  # If you want to use redis in sentinel mode see:
  # https://artifacthub.io/packages/helm/dandydev-charts/redis-ha#redis-sentinel-parameters
# Enables apiVersion deprecation checks
checkDeprecation: true
# Allows graceful shutdown
# terminationGracePeriodSeconds: 65
# lifecycle:
#   preStop:
#     exec:
#       command: [ "sh", "-c", "sleep 60" ]
metrics:
  # Enable Prometheus metrics endpoint
  enabled: true
  # Serve Prometheus metrics on this port
  port: 44180
  # when service.type is NodePort ...
  # nodePort: 44180
  # Protocol set on the service for the metrics port
  service:
    appProtocol: http
  serviceMonitor:
    # Enable Prometheus Operator ServiceMonitor
    enabled: false
    # Define the namespace where to deploy the ServiceMonitor resource
    namespace: ""
    # Prometheus Instance definition
    prometheusInstance: default
    # Prometheus scrape interval
    interval: 60s
    # Prometheus scrape timeout
    scrapeTimeout: 30s
    # Add custom labels to the ServiceMonitor resource
    labels: {}
    ## scheme: HTTP scheme to use for scraping. Can be used with `tlsConfig` for example if using istio mTLS.
    scheme: ""
    ## tlsConfig: TLS configuration to use when scraping the endpoint. For example if using istio mTLS.
    ## Of type: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#tlsconfig
    tlsConfig: {}
    ## bearerTokenFile: Path to bearer token file.
    bearerTokenFile: ""
    ## Used to pass annotations that are used by the Prometheus installed in your cluster to select Service Monitors to work with
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
    annotations: {}
    ## Metric relabel configs to apply to samples before ingestion.
    ## [Metric Relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs)
    metricRelabelings: []
    # - action: keep
    #   regex: 'kube_(daemonset|deployment|pod|namespace|node|statefulset).+'
    #   sourceLabels: [__name__]
    ## Relabel configs to apply to samples before ingestion.
    ## [Relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config)
    relabelings: []
    # - sourceLabels: [__meta_kubernetes_pod_node_name]
    #   separator: ;
    #   regex: ^(.*)$
    #   targetLabel: nodename
    #   replacement: $1
    #   action: replace
# Extra K8s manifests to deploy
extraObjects: []

step 6 : Install oauth 2 helm chart

Use following steps to install oauth2 using help chart.

    helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
    helm pull oauth2-proxy/oauth2-proxy
    tar -xzvf oauth2-proxy-10.7.0.tgz
    mv values.yaml oauth2-proxy/values.yaml
    helm install oauth2-proxy oauth2-proxy/ -n rook-ceph

step 7: Update rook-ceph configuration

Configure Ceph manager:

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

    ceph config-key set mgr/dashboard/external_auth true

    ceph config-key set mgr/dashboard/external_auth_header_name "X-Remote-User"

    ceph config-key set mgr/dashboard/external_auth_logout_url "https://<your_DNS_name>/oauth2/sign_out?rd=https://<your_DNS_name>/keycloak/realms/master/protocol/openid-connect/logout?client_id=ceph-dashboard"

step 8: update ceph-dashboard ingress

Configure the ceph-dashboard Ingress :

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

⚠ Warning

Starting with recent versions of the NGINX Ingress Controller, *-snippet annotations are disabled by default for security reasons. For the following to work, the Ingress Controller’s ConfigMap must be configured accordingly:

kubectl patch configmap ingress-nginx-controller \
  -n ingress \
  --type merge \
  -p '{"data":{"allow-snippet-annotations":"true","annotations-risk-level":"Critical"}}'

metadata:
  annotations:
    cert-manager.io/cluster-issuer: kubeops-ca-issuer
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: rook-ceph-cluster
    meta.helm.sh/release-namespace: rook-ceph

    nginx.ingress.kubernetes.io/auth-url: "https://<your_DNS_name>/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://<your_DNS_name>/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-Auth-Request-User"

    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Remote-User $upstream_http_x_auth_request_user;

Step 9: create oauth2 ingress

Create an Ingress for oauth2-proxy

Important: Replace all <your_DNS_name> placeholders with your correct DNS name

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oauth2-proxy-ingress
  namespace: rook-ceph
  annotations:
    kubernetes.io/ingress.class: nginx

spec:
  rules:
    - host: <your_DNS_name>
      http:
        paths:
          - path: /oauth2
            pathType: Prefix
            backend:
              service:
                name: oauth2-proxy
                port:
                  number: 80

5 - Upgrading a Kubernetes cluster

This guide outlines the steps to upgrade the Kubernetes version of a cluster, specifically demonstrating how to change the version using a configuration file.

Note This guide requires a Compliance enterprise license.

Upgrading a Kubernetes cluster is essential to maintain security, stability, and compatibility.Like Kubernetes itself, we adhere to the version skew policy and only allow upgrades between releases that differ by a single minor version. This ensures compatibility between components, reduces the risk of instability, and keeps the cluster in a supported and secure state.

For more information about the Version Skew Policy, see the official Kubernetes documentation:
Click here

Use the following steps to upgrade the Kubernetes version of a cluster.

Kubernetes Version Upgrade Process:

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1. Pull required KOSI packages on your ADMIN

If you do not specify a parameter, the Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion 1.34.1 you can pull an older Kubernetes version.
Available Kubernetes versions are

kubeopsctl pull --kubernetesVersion <x.xx.x>

2. Change your target version inside the cluster-values

Note The actual version is the current Kubernetes version in your cluster. The target version in the nodes, is version we want to do the upgrade (e.g., see this yaml file)

Note To upgrade your cluster, the parameter changeCluster must be set to true (e.g., see this yaml file)

3. Start the upgrade

kubeopsctl apply -f cluster-values.yaml

Important It is important to start the upgrade with the initial controlplane node. It is the first node with which the cluster was created.

Example 1 - Upgrade all nodes in the cluster to a specific version

In this example, a cluster is upgraded from Kubernetes version v1.33.5 to v1.34.1.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.34.1 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.34.1

2. Update the target version in `cluster-values.yaml`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version in the kubeVersion field of each node.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.33.5     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true           # -> important! Needs to be set for an upgrade
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.34.1       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.34.1       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.34.1       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.34.1       # -> target version

2. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully.
When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

Example 2 - Tranche upgrade of zones to a specific version

In this example, the cluster is upgraded in tranches. The order is:

zone1 (because it contains the initial control plane node),
then zone3,
and finally zone2.

Important It is important to start your tranche upgrade with zone1, because zone1 contains the initial control plane node.

1. Pull required KOSI packages on your ADMIN

Pull the kubernetes v1.33.5 packages on your ADMIN machine.

kubeopsctl pull --kubernetesVersion 1.33.5

2. Adjust your cluster-values in `zone1`

Adjust your cluster-values.yaml according to the example below. Make sure you set:

The current Kubernetes version in kubernetesVersion.
The target Kubernetes version only for the nodes in zone1. In the snippet below, only zone1 is configured with the target version.

Note Pay close attention to the format and syntax in cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false           
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

3. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

Note Tools like rook-ceph has no pdbs, so if you drain nodes for the kubernetes upgrade, rook ceph is temporarily unavailable. You should drain only one node at a time for the kubernetes upgrade.

4. Adjust your `cluster-values.yaml` in `zone2`

Next, change the target version of zone2:

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

5. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

6. Adjust your `cluster-values.yaml` `in zone3`

Finally, change the target version of zone 3.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2     # -> actual version
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.33.5       # ->target version
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.33.5       # -> target version
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.33.5       # -> target version
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.33.5       # -> target version

7. Validate your values and upgrade the cluster

After cluster-values.yaml is updated, review all values carefully. When you are ready, start the upgrade process:

kubeopsctl apply -f cluster-values.yaml

6 - Installing KubeOps Compliance applications

This guide outlines the steps to install KubeOps Compliance applications of a cluster.

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be individually configured as needed.

By separating the cluster values from the application values, the application values can be modified independently and installed at a later stage, providing greater flexibility and maintainability.

Prerequisits

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

⚠ Warning To ensure proper DNS resolution, all components of the logging stack — Filebeat, Logstash, OpenSearch, and OpenSearch Dashboards—must be deployed within the same Kubernetes namespace (or network domain).

Example 1: Installing Applications in a non-airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. For more information about available packages as well as parameters for each package check here.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.31.6         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2      
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and install the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the installation process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Installing Applications in an airgap-environment

To install the KubeOps Compliance Applications in an existing cluster follow the next steps:

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be installed. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application installation process
Important for only installation of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

The following file is only an example. Make sure to change the necessary values (ips, passwords, …) before usage

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                        # -> important
clusterName: myCluster
clusterUser: root
kubernetesVersion: 1.32.2         
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: local
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2      
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2      
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

7 - Updating KubeOps Compliance applications

This guide outlines the steps to update KubeOps Compliance applications of a cluster.

Note This guide requires a Compliance enterprise license.

There is a predefined selection of applications included with KubeOps Compliance. These applications ensure a production-ready cluster deployment and can be configured individually as needed.

By separating cluster values from application values, application values can be modified independently and installed later, providing greater flexibility and maintainability.

kubeopsctl automatically detects whether an application is already deployed and updates it accordingly. de]

Important If you have made changes to your deployments, reapply them afterward or secure them using the advancedValues configuration.

Prerequisites

Before performing any action with kubeopsctl, it is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

Updated KubeOpsctl

If you have an older kubeopsctl version installed, update it before starting with updating Compliance appliactions.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

Example 1: Updating Applications in a non-airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

3. The KubeOps Compliance Application update process
Important for only update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false                       # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

4. Validate your values and update the KubeOps Compliance Applications Once you finished defining your values, check them once again. If you are ready, just start the update process with the command:

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

Example 2: Updating Applications in an airgap-environment

To update the KubeOps Compliance Applications in an existing cluster follow the next steps:

Note All KubeOps Compliance applications versions are linked to the specific KubeOps Compliance version. A detailed version reference can be found here.

1. Define the Enterprise-Value-file

In the example value, the following applications are enabled:

opa-gatekeeper
rook-ceph
harbor
kubeops-dashboard

All other applications are disabled and will not be updated. Value-parameter will be explained in the references and can be found here.

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: true             # important for airgap, otherwise images are pulled from public registry
packages:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: gatekeeper-system
    advanced:
- name: rook-ceph
  enabled: true
  values:
    standard:
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
          mon:
            requests:
              cpu: "1"
              memory: "1Gi"
          osd:
            requests:
              cpu: "1"
              memory: "1Gi"
        dashboard:
          enabled: true
      operator:
        data:
          rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: harbor.local
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: kubeops-dashboard.local
      service:
        nodePort: 30007
    advanced:
- name: filebeat-os
  enabled: false
  values:
    standard:
      namespace: logging
    advanced:

2. Update kubeopsctl

If you have an older kubeopsctl version installed, update it using the following commands.

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
sudo apt update
sudo apt install -y kubeopsctl=<kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/rpm/
sudo dnf install -y --disableexcludes=kubeops-repo <kubeopsctl-version>

# kubeopsctl-version can be found under : https://packagerepo.kubeops.net/deb/pool/main/
wget https://packagerepo.kubeops.net/deb/pool/main/<kubeopsctl-version>.deb
sudo dpkg --install <kubeopsctl-version>.deb

# kubeopsctl-versions can be found under: https://packagerepo.kubeops.net/rpm
sudo rpm kubeopsctl
wget https://packagerepo.kubeops.net/rpm/<kubeopsctl-version>.rpm
sudo rpm --install <kubeopsctl-version>.rpm

2. Pull the KubeOps Compliance Applications packages
To pull the required application packages in the correct version for the release, use the following commands:

kubeopsctl pull -f enterprise-values.yaml --kubernetesVersion <x.xx.x>

kubeopsctl pull --tools enterprise-values.yaml --kubernetesVersion <x.xx.x>

Note The airgap-packages will be pulled automatically

3. The KubeOps Compliance Application update process
Important for only the update of the tools is that you have set your flag changeCluster to false in your cluster-values.yaml.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: true                          # -> important
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: local            # If you have incorporated package repository, replace “local” with the URL https://packagerepo.kubeops.net/
changeCluster: false                # -> important
zones:
- name: zone1
  nodes:
  - name: demo-controlplane01
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker01
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.31.6       
- name: zone2
  nodes:
  - name: demo-controlplane02
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker02
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.30.8       
- name: zone3
  nodes:
  - name: demo-controlplane03
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.31.6       
  - name: demo-worker03
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.31.6

⚠ Warning Both, Cluster-values.yaml and enterprise-values.yaml are required!

Note There is the possibility to do an Upgrade of the kubernetes version and do an installation of the KubeOps Compliance Tools

kubeopsctl apply -f cluster-values.yaml -f enterprise-values.yaml

8 - Harbor Deployment with CloudNativePG

Here is a brief overview of Harbor Deployment with CloudNativePG on Kubernetes using Kosi

This guide describes how to deploy Harbor on Kubernetes using a CloudNativePG (CNPG) PostgreSQL cluster managed by the CloudNativePG operator in an air‑gap and non-airgap environments.

Harbor Deployment with CloudNativePG in Non-airgap enviroment

Prerequisites

A running Kubernetes cluster.
kubectl, kosi, and kubeopsctl installed and configured.
You need to login with kosi . Refer to the official KOSI documentation for details here.

Step 1 — Install CloudNativePG operator

Deploy the operator with Kosi:

kosi install --hub kubeops kubeops/cloudnative-pg-operator:<kubeopsctl_version> --dname cnpg-operator

With this step, it Installs the CloudNativePG operator into the cluster and the operator manages PostgreSQL clusters and their lifecycle.

Step 2 — Create PostgreSQL cluster for Harbor

1. Apply the following Cluster manifest to create a Postgres cluster with 2 instances and 1Gi storage:

cat <<EOF | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg
  namespace: harbor
spec:
  instances: 2
  imagePullSecrets:
  - name: registry-pullsecret
  storage:
    size: 1Gi
EOF

2. Services and pods created for the cluster `cloudnative-pg`:

cloudnative-pg-rw → primary (read/write) cloudnative-pg-ro → replicas (read-only) cloudnative-pg-r → all pods

3. Verify pods are Running:

kubectl get pods -n harbor

Step 3 — Retrieve application user credentials

CNPG automatically creates a Secret named cloudnative-pg-app in the harbor namespace.

1. Verify the Secret exists:

kubectl get secret cloudnative-pg-app -n harbor

2. Decode the base64-encoded fields:

kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.username}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.password}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.dbname}" | base64 -d

Example values (for illustration only):

username: app
password: Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2
database: app

Important:
Use the non-superuser application credentials from this Secret in Harbor’s configuration.

Step 4 — Update Harbor tools.yaml for an external database

Edit your tools.yaml and set Harbor values under the helm chart configuration.
Example snippet:

- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "<DB_PASSWORD>"
      redisPassword: "Redis_Password"
      externalURL: <your_domain_name>
      nodePort: 30002
      hostname: <your_domain_name>
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
      database:
        type: external
        external:
          host: "cloudnative-pg-rw.harbor.svc.cluster.local"
          port: "5432"
          username: "app"
          password: "Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2"
          coreDatabase: "app"

Important: Use the -rw service host (cloudnative-pg-rw…) for write operations.
Do not use a superuser account.
Ensure the password matches the CNPG Secret.

Step 5 — Install Harbor with Kosi

1. Deploy Harbor using the updated tools.yaml:

kosi install --hub kubeops kubeops/harbor:2.0.3 -f tools.yaml --dname harbor

2. Verify Harbor pods:

kubectl get pods -n harbor

3. Access Harbor at: <your_domain_name>:30002 (or as configured)

Harbor Deployment with CloudNativePG in Airgap envoirnment

Prerequisites

A running Kubernetes cluster.
podman, kubectl, kosi, and kubeopsctl installed and configured.
You need to login with kosi . Refer to the official KOSI documentation for details here.

First, log in to your Harbor registry and pull the CloudNativePG operator package using kosi.

podman login <ip_address>:<nodePort> -u <user_name> -p <password> --tls-verify=false   
kosi pull --hub kubeops kubeops/cloudnative-pg-operator:<kubeopsctl_version> -o cloudnative-pg.tgz -r <ip_address>:<NodePort>/kubeops -t localhost:<NodePort>/kubeops

kosi install -p cloudnative-pg.tgz --dname cld-pg

Step 2: Manually Mirror the PostgreSQL Image to the KubeOps Project

In an air‑gapped environment, you must manually pull the PostgreSQL image from the external registry and push it to the KubeOps project in Harbor.

podman pull ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie

podman tag ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie <ip_address>:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie

podman push <ip_address>:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie --tls-verify=false

Step 3: Create the PostgreSQL Cluster for Harbor

Create a CNPG PostgreSQL cluster with two instances and 1Gi of storage in the harbor namespace:

cat <<EOF | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloudnative-pg
  namespace: harbor
spec:
  instances: 2
  storage:
    size: 1Gi
EOF

Step 4: Configure the Cluster to Use the Mirrored Image

Update the CNPG cluster to use the image you pushed to Harbor in Step 2.

1.List the cluster:

kubectl get cluster -n harbor

2.Edit the cluster:

kubectl edit cluster -n harbor cloudnative-pg

3.Replace the image name in the editor:

From:

ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie

To:

localhost:<NodePort>/kubeops/cloudnative-pg/postgresql:18.1-system-trixie

Step 5: Retrieve CNPG Application User Credentials

CNPG automatically creates a Secret named cloudnative-pg-app in the harbor namespace.

1. Verify the Secret exists:

kubectl get secret cloudnative-pg-app -n harbor

2. Decode the base64-encoded fields:

kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.username}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.password}" | base64 -d 
kubectl get secret cloudnative-pg-app -n harbor -o jsonpath="{.data.dbname}" | base64 -d

Example values (for illustration only):

username: app
password: Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2
database: app

Important:
Use the non-superuser application credentials from this Secret in Harbor’s configuration.

Step 6: Configure Harbor in enterprise.yaml

Update your enterprise-values.yaml to configure Harbor with the external CNPG database.
Example snippet:

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: harbor
        harborpass: "password"
        databasePassword: "<DB_PASSWORD>"
        redisPassword: "Redis_Password"
        externalURL: <your_domain_name>
        nodePort: 30002
        hostname: <your_domain_name>
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi
              storageClass: "rook-cephfs"
            jobservice:
              jobLog:
                size: 1Gi
                storageClass: "rook-cephfs"
            database:
              size: 1Gi
              storageClass: "rook-cephfs"
            redis:
              size: 1Gi
              storageClass: "rook-cephfs"
            trivy: 
              size: 5Gi
              storageClass: "rook-cephfs"
      advanced:
        database:
          type: external
          external:
            host: "cloudnative-pg-rw.harbor.svc.cluster.local"
            port: "5432"
            username: "app"
            password: "Hw2t7hXuKPfZrVjVDwCc4PeKTevlB7ORmzQeW50JtEqiwHl40xkxuhVHeRIU3fX2"
            coreDatabase: "app"

Important: Use the -rw service host (cloudnative-pg-rw…) for write operations.
Do not use a superuser account.
Ensure the password matches the CNPG Secret.

Apply the updated configuration:

kubeopsctl apply -f enterprise-values.yaml

Step 6 — Verify the Harbor Deployment

Check that Harbor pods are running in the harbor namespace:

kubectl get pods -n harbor

All Harbor pods should eventually reach a Running or Ready state

General Notes and best practices

Always decode the CNPG app Secret to obtain the correct username, password, and database name used by Harbor.
Use the cloudnative-pg-rw service for write traffic; use -ro services for reads if required.
Production recommendations (follow your organizational security policies):
Create a dedicated DB user (for example, harbor) rather than reusing the default app user.
Enable TLS for Postgres connections.
Implement backups (for example, CNPG Barman or object storage like S3).
To scale the CNPG cluster, update the instances field in the Cluster spec and reapply.

9 - Ingress Configuration

Here is a brief overview of how you can configure your ingress manually.

Manual configuration of the Nginx-Ingress-Controller

Right now the Ingress Controller Package is not fully configured. To make complete use of the Ingress capabilities of the cluster, the user needs to manually update some of the settings of the corresponding service.

Choosing an ingress controller. KubeOps ships two ingress controllers: ingress-nginx and traefik. NGINX has reached end of support, so new clusters should prefer Traefik and existing clusters should plan a migration. To replace nginx-ingress with Traefik (including how SSO, certificates and dashboards keep working), follow How to migrate from nginx to traefik ingress. The rest of this page covers the manual nginx service configuration.

Locating the service

The service in question is called “ingress-nginx-controller” and can be found in the same namespace as the ingress package itself. To locate the service across all namespaces, you could use the following command.

kubectl get service -A | grep ingress-nginx-controller

This command should return two entries of services, “ingress-nginx-controller” and “ingress-nginx-controller-admission”, though only the first one needs to be further adjusted.

Setting the Ingress-Controller service to type NodePort

To edit the service, you can use the following command, although the actual namespace may be different. This will change the service type to NodePort.

kubectl patch service ingress-nginx-controller -n ingress -p '{"spec":{"type":"NodePort"}}'

Kubernetes will now automatically assign unused portnumbers for the nodePort to allow http and https connections to the service. These can be retrieved by running the same command, used to locate the service. Alternatively, you can use the following command, which adds the portnumbers 30080 and 30443 for the respective protocols. By doing so, you have to make sure, that these portnumbers are not being used by any other NodePort service.

kubectl patch service ingress-nginx-controller -n ingress --type=json -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}, {"op":"add","path":"/spec/ports/0/nodePort","value":30080}, {"op":"add","path":"/spec/ports/1/nodePort","value":30443}]'

Configuring external IPs

If you have access to external IPs that route to one or more cluster nodes, you can expose your Kubernetes-Services of any type through these addresses. The command below shows how to add an external IP-Adress to the service with the example value of “192.168.0.1”. Keep in mind that this value has to be changed in order to fit your networking settings.

kubectl patch service ingress-nginx-controller -n ingress -p '{"spec":{"externalIPs":["192.168.0.1"]}}'

10 - Accessing Dashboards installed with KubeOps

A brief overview of how you can access dashboards.

To access an application dashboard, an SSH tunnel to one of the control planes is needed. The following dashboards are available and configured with the following NodePorts by default:

⚠ Warning Change default credentials before going to production.

Note The NodePorts and hostnames shown on this page reflect the default values. Both can be configured in the enterprise-values.yaml. See Fileformats for details.

NodePort

32090 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

To establish a tunnel, forward the NodePort from one of the control planes to your local machine. You can use the command line, PuTTY, or MobaXterm for this.
After that, the dashboard can be accessed at localhost:32090/prometheus/.

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/prometheus/.

No credentials are necessary for login.

NodePort

30211 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/grafana/.

username: the username set in the enterprise-values.yaml of Prometheus (default: user)
password: the password set in the enterprise-values.yaml of Prometheus (default: password)

NodePort

30050 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/opensearch/.

OpenSearch Dashboards supports Single Sign-On (SSO) via Keycloak as the primary login, with username/password as a fallback.

SSO (recommended): open https://<your-domain>/opensearch/ and choose Log in with single sign-on. You are redirected to Keycloak. See the SSO for OpenSearch guide for setup and how to grant users access.
Username / password (fallback):
- username: admin
- password: the value of adminPassword in the enterprise-values.yaml (default: Password@@123456)

Note

SSO is only wired up when the OpenSearch hostname points to a reachable Keycloak (non-.local). For *.local hostnames only the username/password login is available. See Changing the OpenSearch password to change admin.

NodePort

HTTP: 30002
HTTPS: 30003

Connecting via SSH Tunnel

To establish a tunnel, forward port 30002 from one of the control planes to your local machine. You can use the command line, PuTTY, or MobaXterm for this.
After that, the dashboard can be accessed at localhost:30002/.

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/.

username: admin
password: the password set in the enterprise-values.yaml (default: password)

NodePort

The Rook/Ceph dashboard has no fixed NodePort. To find the NodePort in use, run one of the following:

kubectl get svc -n rook-ceph

Look for the rook-ceph-mgr-dashboard-external-http service:

NAME                                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                                     AGE
rook-ceph-mgr-dashboard-external-http     NodePort    192.168.197.13    <none>        7000:31268/TCP                              21h

Or retrieve the NodePort directly:

echo $(kubectl get -n rook-ceph -o jsonpath="{.spec.ports[0].nodePort}" services rook-ceph-mgr-dashboard-external-http)

In the example above, the NodePort is 31268.

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/ceph-dashboard/.

echo Username: admin
echo Password: $(kubectl get secret rook-ceph-dashboard-password -n rook-ceph --template={{.data.password}} | base64 -d)

NodePort

30007 (if not set otherwise in the enterprise-values.yaml)

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/kubeops-dashboard/.

kubectl -n monitoring create token headlamp-admin

NodePort

30180

Connecting via SSH Tunnel

Connecting via Ingress

The dashboard can also be accessed via https://<your-domain>/keycloak/.

echo Username: $(kubectl get secret --namespace keycloak keycloak-kubeops -o jsonpath="{.data.ADMIN_USER}" | base64 -d)
echo Password: $(kubectl get secret --namespace keycloak keycloak-kubeops -o jsonpath="{.data.ADMIN_PASSWORD}" | base64 -d)

11 - Accessing Dashboards with OIDC

A brief overview of how you can access dashboards with OIDC.

This how-to guide explains how to enable OIDC-based access to your Kubernetes application dashboards using kubeopsctl. You will configure a single enterprise-values.yaml file that defines the required components. Once the configuration is applied, the dashboards for Harbor, Grafana, Rook-Ceph, OpenSearch, Keycloak, the KubeOps dashboard, and Prometheus become accessible via their configured hostnames and paths. Following the steps in this guide, you will set up a consistent, centralized OIDC integration and make your dashboards securely available through a browser.

Prerequisites

yq should be installed on all master nodes. For more infomration refer yq installation guide.
Keycloak should be installed before other packages.
Pull all the required packages with ‘kubectl pull’ command. For more information refer Pull Packages with Kubeopsctl

Step 1: Configure enterprise-values.yaml

In order to connect to dashboad with OIDC, you need to configure your enterprise-values.yaml file as below

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: rook-ceph
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access
      namespace: rook-ceph
      cluster:
        resources:
          mgr:
            requests:
              cpu: "500m"
              memory: "512Mi"
            mon:
              requests:
                cpu: "1"
                memory: "1Gi"
            osd:
              requests:
                cpu: "1"
                memory: "1Gi"
          dashboard:
            enabled: "true"
        operator:
          data:
            rookLogLevel: "DEBUG"
- name: harbor
  enabled: true
  values:
    standard:
      namespace: harbor
      harborpass: "password"
      databasePassword: "password"
      redisPassword: "password"
      externalURL: http://10.2.10.110:30002
      nodePort: 30002
      hostname: <domain_name>          # required for OIDC dashboard access
      harborPersistence:
        persistentVolumeClaim:
          registry:
            size: 40Gi
            storageClass: "rook-cephfs"
          jobservice:
            jobLog:
              size: 1Gi
              storageClass: "rook-cephfs"
          database:
            size: 1Gi
            storageClass: "rook-cephfs"
          redis:
            size: 1Gi
            storageClass: "rook-cephfs"
          trivy: 
            size: 5Gi
            storageClass: "rook-cephfs"
    advanced:
- name: cert-manager 
  enabled: true
  values:
    standard:
      namespace: cert-manager
      replicaCount: 3
      logLevel: 2
    advanced:
    ca:      
      emailLetsEncrypt: <email>
      ingressName: nginx
- name: ingress-nginx
  enabled: true
  values:
    standard:
      namespace: ingress-nginx
      externalIPs: 
        - <>                 
      advanced:
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: "keycloak"
      storageClass: "rook-cephfs"
      nodePort: "30180"
      hostname: <domain_name>          # required for OIDC dashboard access
      keycloak:
        auth:
          adminUser: admin
          adminPassword: topsecret
      postgresql:
        auth:
          postgresUserPassword: "changeme"
          username: bn_keycloak
          password: "changeme"
          database: bitnami_keycloak
        volumeSize: "8Gi"
    advanced:
- name: filebeat-os
  enabled: true
  values:
    standard:
      namespace: logging
    advanced:
- name: logstash-os
  enabled: true
  values:
    standard:
      namespace: logging
      volumeClaimTemplate:
        accessModes: 
          - ReadWriteMany
        resources:
          requests:
            storage: 1Gi
        storageClass: "rook-cephfs"
    advanced:
- name: opensearch-dashboards
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access 
      namespace: logging
      nodePort: 30050
    advanced:
- name: opensearch-os
  enabled: true
  values:
    standard:
      hostname: <domain_name>          # required for OIDC dashboard access
      namespace: logging
      opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
      resources:
        requests:
          cpu: "250m" # optional, default is 250m
          memory: "1024Mi" # optional, default is 1024Mi
        limits:
           cpu: "300m" # optional, default is 300m
           memory: "3072Mi" # optional, default is 3072Mi
      persistence:
        size: 4Gi # mandatory
        enabled: "true" # optional, default is true
        enableInitChown: "false" # optional, default is false
        labels:
          enabled: "false" # optional, default is false
        storageClass: "rook-cephfs" # optional, default is rook-cephfs
        accessModes:
          - "ReadWriteMany" # optional, default is {ReadWriteMany}
        securityConfig:
          enabled: "false" # optional, default value: false
        replicas: "3" # optional, default is 3
    advanced:
- name: kube-prometheus-stack
  enabled: true
  values:
    standard:
      namespace: monitoring
      grafanaUsername: admin
      grafanaPassword: topsecret
      retentionSize: "24GB"
      grafanaResources:
        hostname: <domain_name>          # required for OIDC dashboard access
        nodePort: 30211
        retention: 10d
        retentionSize: "24GB"
        storageClass: "rook-cephfs"
        storage: 25Gi
      prometheusResources:
        hostname: <domain_name>          # required for OIDC dashboard access
        retentionSize: "24GB"
        storageClass: "rook-cephfs"
        storage: 25Gi
    advanced:
- name: opa-gatekeeper
  enabled: true
  values:
    standard:
      namespace: opa-gatekeeper
    advanced:
- name: kubeops-dashboard
  enabled: true
  values:
    standard:
      namespace: monitoring
      hostname: <domain_name>          # required for OIDC dashboard access
      service:
        nodePort: 30007
    advanced:
- name: velero
  enabled: true
  values:
    standard:
      namespace: "velero"
      accessKeyId: "your_s3_storage_username"
      secretAccessKey: "your_s3_storage_password"
      useNodeAgent: false
      defaultVolumesToFsBackup: false
      provider: "aws"
      bucket: "velero"
      useVolumeSnapshots: false
      backupLocationConfig:
        region: "minio"
        s3ForcePathStyle: true
        s3Url: "http://minio.velero.svc:9000"
    advanced:

step 2: Apply the Configuration with ‘KubeOpctl’

Apply the enterprise-values.yaml configuration using the following command:

kubeopsctl apply -f enterprise-values.yaml

Wait until all components are deployed and running before proceeding.

Step 3: Connect to Dashboards via Hostname

To access the dashboards via OIDC, use the hostnames configured in your enterprise-values.yaml (uservalues). Harbor is an exception: you can access it directly using the base domain name.

Use the following paths, replacing <domain_name> with your configured domain:

<domain_name>/ for Harbor
<domain_name>/grafana for grafana
<domain_name>/ceph-dashboard for rook-ceph
<domain_name>/opensearch for opensearch
<domain_name>/keycloak for keycloak
<domain_name>/kubeops-dashboard for Kubeops Dashboard
<domain_name>/prometheus for Prometheus

12 - Changing a User Password in OpenSearch

Detailed instructions for changing the OpenSearch password.

This guide explains how to change passwords in OpenSearch.

Recommended: Change the `admin` password via the package value

The OpenSearch admin password is managed through the adminPassword package value. Set it for the opensearch-os package (and the matching opensearch-dashboards package) in your enterprise-values.yaml and run an update:

packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        adminPassword: "<your-new-password>" # must match opensearch-dashboards
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        adminPassword: "<your-new-password>" # must match opensearch-os

On the next kubeopsctl apply/update the new password is enforced automatically after the deployment. This also works on existing clusters and does not remove any other internal users.

Note Internal users created in the OpenSearch Dashboards Security UI persist across package updates. The update only enforces the admin password and reapplies the authentication/authorization configuration; it never overwrites your internal users.

Advanced: Change any user’s password manually via `securityadmin`

Use this method to change the password of any internal user (for example testuser or kibanaro) without a package update.

This process reads the live internal users from the security index (via securityadmin -backup), so every other user — including users you created in the OpenSearch Dashboards Security UI — is preserved. Do not build the file from the internal-users-config-secret: that secret only holds the packaged baseline users and would remove UI-created users when applied.

Prerequisites

Access to the Kubernetes cluster and kubectl.
The admin certificate is available in the pod at /usr/share/opensearch/config/certs/ (root-ca.pem, admin.pem, admin-key.pem) in a KubeOps deployment.

Set a shell variable for convenience (adjust the namespace and pod name as needed):

NS=logging; POD=opensearch-cluster-master-0

Step 1: Back up the current internal users (live state)

kubectl exec -it $POD -n $NS -- bash -c "\
    /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -backup /tmp/os-sec \
    -icl -nhnv \
    -cacert /usr/share/opensearch/config/certs/root-ca.pem \
    -cert /usr/share/opensearch/config/certs/admin.pem \
    -key /usr/share/opensearch/config/certs/admin-key.pem"

This writes the current configuration — including all internal users — to /tmp/os-sec/ in the pod.

Note securityadmin -backup may end with Configuration for 'audit' failed because of empty source and a non-zero exit code. This is harmless: internal_users.yml is written before that step, so the file you need is already present.

Step 2: Generate a new password hash for each user

Run this once per user and note the resulting hash:

kubectl exec -it $POD -n $NS -- bash -c "sh /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p <new_password>"

Step 3: Update the hashes in the backup file

Copy the file out, edit it locally, and copy it back:

kubectl cp $NS/$POD:/tmp/os-sec/internal_users.yml ./internal_users.yml
# edit ./internal_users.yml: replace the hash: value under each user you want to change
kubectl cp ./internal_users.yml $NS/$POD:/tmp/os-sec/internal_users.yml

Only change the hash: line of the target users; leave everyone else untouched:

testuser:
  hash: "<new-hash-for-testuser>"
  reserved: false
  backend_roles:
  - "testrole"
kibanaro:
  hash: "<new-hash-for-kibanaro>"
  reserved: false
  backend_roles:
  - "kibanauser"

Step 4: Apply only the internal users configuration

Note

The DN of the certificate specified with -cert must be listed under plugins.security.authcz.admin_dn in opensearch.yml. In a KubeOps deployment the admin certificate subject is C=DE,L=test,O=client,OU=client,CN=admin.

More information: Opensearch Documentation.

Apply only the internalusers config type. Do not use -cd (the whole directory) here: -cd requires the complete security config set (including nodes_dn.yml, allowlist.yml, audit.yml), which a KubeOps deployment does not ship, so it would fail with nodes_dn.yml ... No such file or directory.

kubectl exec -it $POD -n $NS -- bash -c "\
    /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -f /tmp/os-sec/internal_users.yml -t internalusers \
    -icl -nhnv \
    -cacert /usr/share/opensearch/config/certs/root-ca.pem \
    -cert /usr/share/opensearch/config/certs/admin.pem \
    -key /usr/share/opensearch/config/certs/admin-key.pem"

The change is written to the security index and takes effect immediately — no pod restart required.

Important

Once applied with securityadmin.sh, credentials are stored in the OpenSearch security index, which lives on the persistent volume and therefore survives pod and container restarts as well as package updates. Editing files inside the container without running securityadmin.sh only changes the local file and is not applied to the cluster. For the admin user, prefer the adminPassword package value described above — it is enforced again on every update and keeps all other internal users intact.

13 - Backup and restoring artifacts

In this article, we look at the backup procedure with Velero.

What is Velero?

Velero uses object storage to store backups and associated artifacts. It also optionally integrates supported block storage systems to snapshot your persistent volumes. Before beginning the installation process, you should identify the object storage provider and optional block storage provider(s) you’ll be using from the list of compatible providers.

Velero supports storage providers for both cloud-provider environments and on-premises environments.

Velero prerequisites:

Access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.
kubectl installed locally
Object Storage (S3, Cloud Provider Environment, On-Premises Environment)

Note : Compatible providers and on-premises documentation can be read on https://velero.io/docs

Install Velero

This command is an example on how you can install velero into your cluster:

velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.2.1 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000

NOTE:

s3Url has to be the url of your s3 storage login.

example for credentials-velero file:

[default]
aws_access_key_id = your_s3_storage_username
aws_secret_access_key = your_s3_storage_password

Backup the cluster

Scheduled Backups

This command creates a backup for the cluster every 6 hours:

velero schedule create cluster --schedule "0 */6 * * *"

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete cluster

Restore Scheduled Backup

This command restores the backup according to a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the cluster

velero backup create cluster

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Backup a specific deployment

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create filebeat --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “filebeat” every 6 hours:

velero schedule create filebeat --schedule "0 */6 * * *" --include-namespaces logging --selector app=filebeat-filebeat,release=filebeat --include-resources serviceaccount,deployment,daemonset,configmap,clusterrolebinding,clusterrole --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete filebeat

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create filebeat --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “filebeat”:

velero backup create filebeat --include-namespaces logging --selector app=filebeat-filebeat,release=filebeat --include-resources serviceaccount,deployment,daemonset,configmap,clusterrolebinding,clusterrole --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create logstash --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “logstash” every 6 hours:

velero schedule create logstash --schedule "0 */6 * * *" --include-namespaces logging --selector app=logstash-logstash,chart=logstash,release=logstash --include-resources StatefulSet,ServiceAccount,Service,Secret,RoleBinding,Role,PodSecurityPolicy,PodDisruptionBudget,Ingress,ConfigMap --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete logstash

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create logstash --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “logstash”:

velero backup create logstash --include-namespaces logging --selector app=logstash-logstash,chart=logstash,release=logstash --include-resources StatefulSet,ServiceAccount,Service,Secret,RoleBinding,Role,PodSecurityPolicy,PodDisruptionBudget,Ingress,ConfigMap --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “logging” every 6 hours:

velero schedule create opensearch --schedule "0 */6 * * *" --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “opensearch” every 6 hours:

velero schedule create opensearch --schedule "0 */6 * * *" --include-namespaces logging --selector app.kubernetes.io/instance=opensearch,app.kubernetes.io/name=opensearch --include-resources ConfigMap,Ingress,NetworkPolicy,PodDisruptionBudget,PodSecurityPolicy,Role,RoleBinding,Secret,Service,ServiceAccount,StatefulSet --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete opensearch

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “logging”:

velero backup create opensearch --include-namespaces logging --include-cluster-resources=true

This command creates a backup for the deployment “opensearch”:

velero backup create opensearch --include-namespaces logging --selector app.kubernetes.io/instance=opensearch,app.kubernetes.io/name=opensearch --include-resources ConfigMap,Ingress,NetworkPolicy,PodDisruptionBudget,PodSecurityPolicy,Role,RoleBinding,Secret,Service,ServiceAccount,StatefulSet --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “monitoring” every 6 hours:

velero schedule create prometheus --schedule "0 */6 * * *" --include-namespaces monitoring --include-cluster-resources=true

This command creates a backup for the deployment “prometheus” every 6 hours:

velero schedule create prometheus --schedule "0 */6 * * *" --include-namespaces monitoring --include-resources Alertmanager,Secret,Ingress,List,PodDisruptionBudget,Role,RoleBinding,PodSecurityPolicy,Service,ServiceAccount,ServiceMonitor,Endpoints,ConfigMap,ConfigMapList,ClusterRole,ClusterRoleBinding,SecretProviderClass,PodMonitor,Prometheus,Job,NetworkPolicy,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Issuer,Deployment,VerticalPodAutoscaler,ThanosRuler --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete prometheus

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “monitoring”:

velero backup create prometheus --include-namespaces monitoring --include-cluster-resources=true

This command creates a backup for the deployment “prometheus”:

velero backup create prometheus --include-namespaces monitoring --include-resources Alertmanager,Secret,Ingress,List,PodDisruptionBudget,Role,RoleBinding,PodSecurityPolicy,Service,ServiceAccount,ServiceMonitor,Endpoints,ConfigMap,ConfigMapList,ClusterRole,ClusterRoleBinding,SecretProviderClass,PodMonitor,Prometheus,Job,NetworkPolicy,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Issuer,Deployment,VerticalPodAutoscaler,ThanosRuler --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “harbor” every 6 hours:

velero schedule create harbor --schedule "0 */6 * * *" --include-namespaces harbor --include-cluster-resources=true

This command creates a backup for the deployment “harbor” every 6 hours:

velero schedule create harbor --schedule "0 */6 * * *" --include-namespaces harbor --include-resources ConfigMap,Deployment,PersistentVolumeClaim,Secret,Service,StatefulSet,Ingress,ServiceMonitor --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete harbor

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “harbor”:

velero backup create harbor --include-namespaces harbor --include-cluster-resources=true

This command creates a backup for the deployment “harbor”:

velero backup create harbor --include-namespaces harbor --include-resources ConfigMap,Deployment,PersistentVolumeClaim,Secret,Service,StatefulSet,Ingress,ServiceMonitor --include-cluster-resources=true --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “gatekeeper-system” every 6 hours:

velero schedule create gatekeeper --schedule "0 */6 * * *" --include-namespaces gatekeeper-system --include-cluster-resources=true

This command creates a backup for the deployment “gatekeeper” every 6 hours:

velero schedule create gatekeeper --schedule "0 */6 * * *" --include-namespaces gatekeeper-system --include-resources PodSecurityPolicy,ServiceAccount,Deployment,PodDisruptionBudget,ResourceQuota,ClusterRole,Role,ClusterRoleBinding,RoleBinding,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Secret,Service,Job --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete gatekeeper

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “gatekeeper-system”:

velero backup create gatekeeper --include-namespaces gatekeeper-system --include-cluster-resources=true

This command creates a backup for the deployment “gatekeeper-system”:

velero backup create gatekeeper --include-namespaces gatekeeper-system --include-resources PodSecurityPolicy,ServiceAccount,Deployment,PodDisruptionBudget,ResourceQuota,ClusterRole,Role,ClusterRoleBinding,RoleBinding,MutatingWebhookConfiguration,ValidatingWebhookConfiguration,Secret,Service,Job --include-cluster-resources=true --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

Scheduled Backups

This command creates a backup for the namespace “rook-ceph” every 6 hours:

velero schedule create rook-ceph --schedule "0 */6 * * *" --include-namespaces rook-ceph --include-cluster-resources=true

Get Schedules

This command lists all schedules for backups:

velero schedule get

Delete Schedules

This command deletes the specified schedule:

velero schedule delete rook-ceph

Restore Scheduled Backup

This command restores the backup from a schedule:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

Backup

This command creates a backup for the namespace “rook-ceph”:

velero backup create rook-ceph --include-namespaces rook-ceph --include-cluster-resources=true

Get Backups

This command lists all created backups:

velero backup get

Delete Backups

This commands deletes the specified backup:

velero backup delete <BACKUP NAME>

Restore Backup

This commands restores the specified backup:

velero restore create <RESOURCE NAME> --from-backup <BACKUP NAME>

restore databases

keycloak

create backup of keycloak namespace, in this example the backup is called keycloak1.

This command creates a backup for the namespace “rook-ceph”:

velero backup create keycloak1 --include-namespaces keycloak --include-cluster-resources=true

restore backup, in this example keycloak1

velero restore create keycloak1 --from-backup keycloak1

restore the database dump:

kubectl -n <keycloak-namespace> exec keycloak-postgres-0 -- pg_restore -v --jobs=4 --clean --if-exists -d bitnami_keycloak /backup/keycloak-db.dump

14 - Add certificate as trusted

This section outlines the process for adding a certificate as trusted by downloading it from the browser and installing it in the Trusted Root Certification Authorities on Windows or Linux systems.

1. Download the certificate

As soon as Chrome issues a certificate warning, click on Not secure to the left of the address bar.
Show the certificate (Click on Certificate is not valid).
Go to Details tab.
Click Export... at the bottom and save the certificate.

As soon as Firefox issues a certificate warning, click on Advanced....
View the certificate (Click on View Certificate).
Scroll down to Miscellaneous and save the certificate.

2. Install the certificate

Press Windows + R.
Enter mmc and click OK.
Click on File > Add/Remove snap-in....
Select Certificates in the Available snap-ins list and click on Add >, then on OK. Add the snap-in.
In the tree pane, open Certificates - Current user > Trusted Root Certification Authorities, then right-click Certificates and select All tasks > Import....
The Certificate Import Wizard opens here. Click on Next.
Select the previously saved certificate and click Next.
Click Next again in the next window.
Click on Finish. If a warning pops up, click on Yes.
The program can now be closed. Console settings do not need to be saved.
Clear browser cache and restart browser.

The procedures for using a browser to import a certificate as trusted (on Linux systems) vary depending on the browser and Linux distribution used. To manually cause a self-signed certificate to be trusted by a browser on a Linux system:

Distribution	Copy certificate here	Run following command to trust certificate
RedHat	`/etc/pki/ca-trust/source/anchors/`	`update-ca-trust extract`

Note

If the directory does not exist, create it.

Note

If you do not have the ca-certificates package, install it with your package manager.

15 - Deploy Package on a cluster

This guide explains how to deploy a package to a Kubernetes cluster with KOSI using either the Helm or Kubectl plugin.

You can install artifacts in your cluster in several ways. When creating a package, you can use one of the following plugins:

helm
kubectl
cmd
kosi

This guide demonstrates a complete, reproducible deployment of nginx-ingress using Helm and kosi.

Using the Helm Plugin

Pre-requisites

Before you begin, make sure the following tools are available in your environment:

helm
kosi

This example uses the official NGINX Helm repository and deploys nginx-ingress with Helm and KOSI.

Prepare the Files

In order to install an artifact with the Helm plugin, the Helm chart must first be downloaded.

To keep the environment clean and reproducible, first create a dedicated working directory and change into it:

mkdir nginx-ingress-kosi && cd nginx-ingress-kosi

The example uses the official NGINX Helm repository. Add the repository before performing any Helm operations:

helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update

To verify which chart versions are available, run:

helm search repo nginx-stable/nginx-ingress --versions

This guide uses chart version 0.16.1. Download the chart as a packaged Helm artifact:

helm pull nginx-stable/nginx-ingress --version 0.16.1

This command creates the following file in your working directory:

nginx-ingress-0.16.1.tgz

Create KOSI package

Once the chart is available locally, initialize the kosi package structure. The following command creates the necessary files in the current directory:

kosi create

The Helm chart remains the central input artifact for deployment, while configuration is handled via values.yaml. Helm already includes default values, which can be inspected or exported if needed:

helm show values nginx-stable/nginx-ingress --version 0.16.1 > values.yaml

Review and update values.yaml before packaging and deploying the chart. For example, in environments without a cloud-based LoadBalancer, you may need to change the service type in values.yaml.

controller:
  service:
    type: LoadBalancer

Only required overrides should be defined in the customized values.yaml file. To customize the deployment of the Helm chart, a new values.yaml file in the current directory must be created and edited with this content:

controller:
  service:
    type: NodePort

This override is merged with the chart defaults at runtime.

Helm charts do not directly contain the full list of container images required for deployment. To identify them, render the chart into Kubernetes manifests:

helm template nginx-ingress nginx-stable/nginx-ingress --version 0.16.1 > manifests.yaml

Then extract all referenced images from the rendered manifest:

grep image: manifests.yaml

For this example, the required image is:

nginx/nginx-ingress:3.0.1

With all artifacts identified, the package.kosi file defines what is required for deployment. It explicitly references:

- the Helm chart archive
- the custom values.yaml
- all required container images

nginx-ingress-0.16.1.tgz
values.yaml
package.kosi

Note

The downloaded Helm chart must also be located in the current directory. To customize the deployment of the Helm chart, the values.yaml file must be edited.

All files required by a task in the package must be named in the package.kosi file under files. The container images required by the Helm chart must also be listed in the package.kosi under containers.

In the example below, only two files are required for the installation: the Helm Chart for the nginx-ingress and the values.yaml to configure the deployment. To install nginx-ingress you will also need the nginx/nginx-ingress image with the tag 3.0.1.

To install nginx-ingress with the Helm plugin, call the plugin as shown in the example under install. The deployment configuration file is listed under values and the packed Helm chart is specified with the key tgz. Furthermore, it is also possible to specify the namespace in which the artifact should be deployed and the name of the deployment. The full documentation for the Helm plugin can be found here.

Use the following configuration:

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "deployexample1";
description = "It shows how to deploy an artifact to your cluster using the helm plugin.";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
        valuesFile = "values.yaml";
        nginxHelmChart="nginx-ingress-0.16.1.tgz";
}

containers =
{
        nginx = ["docker.io", "nginx/nginx-ingress", "3.0.1"];
}

install
{
        helm
        (
            command = "install";
            tgz = "nginx-ingress-0.16.1.tgz";
            values = "['values.yaml']";
            namespace = "dev";
            deploymentName = "nginx-ingress"
        );
}

Build the Package

Once the package.kosi file has been fully configured, all files must be combined into a kosi package. To do this, execute the following command in the directory where the package.kosi file is located.

kosi build

Push the Package

To make the generated kosi package available on other machines, it is pushed to the user’s private KubeOps Hub. To do this, the user must first log in to the hub and then push it to hub.

$ kosi login -u <username>
2023-02-04 11:19:43 Info:      KOSI version: 2.13.0_Alpha0
2023-02-04 11:19:43 Info:      Please enter password
****************
2023-02-04 11:19:26 Info:      Login Succeeded to Hub.
$ kosi push --hub kosi
2023-02-04 11:23:18 Info:      KOSI version: 2.13.0_Alpha0
2023-02-04 11:23:19 Info:      Push to Private Registry registry.preprod.kubeops.net/<username>/

Deploy the Package

Once the KOSI package has been created and published, it needs to be installed on the admin node. The following command will download and execute the package. The package name and version refer to the values defined in package.kosi with the keys name and version.

kosi install --hub <username> <username>/<packagename>:<version>

For the example package, the command would be: kosi install --hub <username> <username>/deployExample:0.1.0.

Using the Kubectl Plugin

Prerequisites

Before you begin, make sure the following tools and components are available:

kosi
the kubeops-kubernetes-plugins package installed on the admin node

This example deploys nginx-ingress from a Kubernetes manifest using the Kubectl plugin.

Create KOSI package

First you need to create a KOSI package. The following command creates the necessary files in the current directory:

kosi create

The NGINX ingress controller YAML manifest can either be automatically downloaded and applied directly with kubectl apply or it can be downloaded manually if you want to customize the deployment. The YAML manifest can be downloaded from the NGINX GitHub Repo and must be placed in the same directory as the files for the KOSI package.

All files required by a task in the package must be named in the package.kosi file under files. The container images required by the YAML manifest must also be listed in the package.kosi under containers. In the example below, only one file is required for the installation: the YAML manifest for the nginx-ingress controller. To install nginx-ingress you will also need the registry.k8s.io/ingress-nginx/controller image with the tag v1.5.1 and the image registry.k8s.io/ingress-nginx/kube-webhook-certgen with tag v20220916-gd32f8c343.

To install nginx-ingress with the Kubectl plugin, call the plugin as shown in the example under installs. The full documentation for the Kubectl plugin can be found here.

languageversion = "1.0.0";
apiversion = "kubernative/kubeops/sina/user/v4";
name = "deployexample2";
description = "It shows how to deploy an artifact to your cluster using the helm plugin.";
version = "0.1.0";
docs = "docs.tgz";
logo = "logo.png";

files =
{
     manifest: "deploy.yaml"
}

containers =
{
    nginx = ["registry.k8s.io", "ingress-nginx/controller", "v1.5.1"];
    certgen= ["registry.k8s.io","ingress-nginx/kube-webhook-certgen","v20220916-gd32f8c343"];
}

install
{
    kubectl
    (
      operation="apply",
      flags="-f deploy.yaml";
      sudo = true;
      sudoPassword="toor"
    );
}

Build the Package

Once the package.kosi file has been fully configured, all files must be combined into a KOSI package. To do this, execute the following command in the directory where the package.kosi file is located.

kosi build

Publish the Package

To make the generated KOSI package available on other machines, it is pushed to the user’s private KubeOps Hub. To do this, the user must first log in to the hub.

$ kosi login -u <username>
2023-02-04 11:19:43 Info:      kosi version: 2.13.0_Alpha0
2023-02-04 11:19:43 Info:      Please enter password
****************
2023-02-04 11:19:26 Info:      Login Succeeded to Hub.
$ kosi push --hub kosi
2023-02-04 11:23:18 Info:      kosi version: 2.13.0_Alpha0
2023-02-04 11:23:19 Info:      Push to Private Registry registry.preprod.kubeops.net/<username>/

Deploy the Package

kosi install --hub <username> <username>/<packagename>:<version>

For the example package, the command would be: kosi install --hub <username> <username>/deployExample:0.1.0.

16 - How to migrate from nginx to traefik ingress

Installation

Kubeops supports deploying Traefik as a dynamic ingress controller and reverse proxy. This guide describes a concise, safe migration from an existing nginx-ingress controller to Traefik and explains how to install Traefik and replace a deprecated nginx-ingress deployment. The migration from nginx to Traefik is straightforward; the steps below show the process in order.

Prerequisites

A running Kubernetes cluster with an existing nginx-ingress controller.
It is recommended to do a login with kosi. Refer to the official KOSI documentation for details here.

1.Create Values file

Create a values.yaml file for the Traefik installation:

# values.yaml
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik
      externalIPs: []
    advanced: {}

Note: Update externalIPs and any other values as required for your environment.

2.Install Traefik

non-airgap-environment

After creating values.yaml, install Traefik:

# get your desired version/s
kosi search --hub kubeops --ps traefik
# install traefik
kosi install --hub kubeops kubeops/traefik:<desired_version> -f values.yaml --dname traefik

Example Non-airgap

kosi install --hub kubeops kubeops/traefik:2.1.0_Beta0 -f values.yaml --dname traefik

Airgap-environment

# log in to Harbor
podman login <ip_address>:<NodePort> -u <username> -p <password> --tls-verify=false
# pull your desired version
kosi pull --hub kubeops kubeops/<desired_version> -o traefik-<desired_version>.tgz -r <ip_address>:<NodePort>/kubeops -t 127.0.0.1:<NodePort>/kubeops
 # install Traefik
kosi install --dname traefik -p traefik-:<desired_version>.tgz -f values.yaml

Example Airgap

kosi pull --hub kubeops kubeops/traefik:2.1.0_Beta0 -o traefik-2.1.0-Beta0.tgz -r 10.2.10.11:30002/kubeops -t 127.0.0.1:30002/kubeops
 
kosi install --dname traefik -p traefik-2.1.0-Beta0.tgz -f values.yaml

3.Verify Deployment

Verify that the Traefik pods and services are running in the traefik namespace:

kubectl get pods -n traefik
kubectl get svc -n traefik

3b. Switch applications to the Traefik ingress class

This is the load-bearing step of the migration. Traefik ships a backwards-compatibility shim (an IngressClass named nginx plus the experimental providers.kubernetesIngressNginx provider) so that existing ingressClassName: nginx ingresses should keep working unchanged. This shim is experimental and does not work reliably for all workloads — Keycloak in particular has been observed to be unreachable through the nginx class while Traefik is the controller. Do not rely on the shim as the primary mechanism; switch each application’s ingress to ingressClassName: traefik.

Every ingress package exposes a uniform standard.ingressClassName key (defaulting to nginx). Set it to traefik on each package, and set the cert-manager solver class:

- name: keycloak
  values:
    standard:
      ingressClassName: traefik
- name: kubeOpsDashboard
  values:
    standard:
      ingressClassName: traefik
- name: kube-prometheus-stack
  values:
    standard:
      ingressClassName: traefik
- name: harbor
  values:
    standard:
      ingressClassName: traefik
- name: rook-ceph
  values:
    standard:
      ingressClassName: traefik
- name: opensearch-dashboards
  values:
    standard:
      ingressClassName: traefik
- name: kubevirt-manager
  values:
    standard:
      ingressClassName: traefik
- name: cert-manager
  values:
    ca:
      ingressName: traefik         # ACME HTTP01 solver ingress class

cert-manager is the only exception to the standard.ingressClassName key: its solver class lives under values.ca.ingressName. Certificates and the cert-manager.io/cluster-issuer annotations are otherwise controller-agnostic.

Note: the plain OpenSearch data package (opensearch-os) and logstash ship their ingress disabled by default, so they need no change. If you enable them, set the class via that package’s advanced block (ingress.ingressClassName / ingress.className).

Apply the updated values and confirm every ingress now uses the Traefik class:

kubectl get ingress -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName

SSO / certificates / dashboards: SSO is app-level OIDC (Keycloak reachable at https://<host>/keycloak), so it is not tied to the ingress controller and keeps working once the class is switched. The same is true for cert-manager and dashboard access.

nginx-only annotations do not migrate. Annotations of the form nginx.ingress.kubernetes.io/* are ignored by Traefik. OpenSearch Dashboards ships buffer/ header-size annotations (proxy-buffer-size, large-client-header-buffers, …) that matter for large OIDC headers; if login fails with 4xx/431/502, add an equivalent Traefik buffering middleware / entrypoint transport setting via the package’s advanced block. Harbor auto-injects nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" when internalTLS.enabled — verify Harbor over TLS after the switch.

4.Remove old nginx-ingress deployment and service

# get version of installed nginx-ingress and its deployment name (--dname)
kosi list
# delete old nginx-ingress
kosi delete --hub kubeops kubeops/ingress-nginx:<installed_version> -f enterprise-values.yaml --dname <kosi_deployment_name>

Example


kosi delete --hub kubeops ingress-nginx:2.1.0_Beta0 -f values.yaml --dname ingress-nginx

Clean up the leftover nginx IngressClass. The ingress-nginx chart annotates its IngressClass with helm.sh/resource-policy: keep, so it survives kosi delete. Both ingress-nginx and the Traefik package define an IngressClass named nginx for controller k8s.io/ingress-nginx, so they must not run at the same time. After removing ingress-nginx, delete any leftover class if it is no longer wanted:
kubectl get ingressclass
kubectl delete ingressclass nginx   # only if you no longer need the nginx compat class

Edit Traefik service

If nginx-ingress used specific NodePorts that you need to reuse, update the Traefik Service:

kubectl edit svc traefik -n traefik

Note: Default NodePorts (for example, 31080 / 31443) might not be reachable in your environment. If these ports are not accessible, determine the NodePorts previously used by nginx-ingress (for example, 30080 / 30443) and configure Traefik to use the same ports.

Update the ports:

Adjust the ports section to match the previous nginx NodePorts if required:

ports:
- name: web
  nodePort: 30080
  port: 80
  targetPort: web
- name: websecure
  nodePort: 30443
  port: 443
  targetPort: websecure

Verify Port Change

kubectl get svc -n traefik

Note: Ensure that the nginx Service is removed or that its NodePorts are freed before reusing those NodePorts on the Traefik Service.

How to Guides

1 - Documentation for production

1.1 - How-To: Production Cluster with OIDC (Public IP & Airgap)

How-To: Production Cluster with OIDC (Public IP & Airgap)

Overview: what you will build

Choose your path

1. Prerequisites

2. Plan addressing and ports

3. Prepare all nodes

3.1 Add the package repository (internet path)

3.2 Operating system adaptations

3.3 Make hostnames resolvable

3.4 Distribute SSH keys

3.5 Distribute the sudoers file

3.6 Install kubeadm

3.7 Configure time synchronization

3.8 Install curl and yq

4. Harden the nodes for production

5. Set up the admin tooling

5.1 (Airgap only) Stage the images in your local Harbor

6. Create the cluster

7. Configure the platform stack

7.1 Sizing, retention and backups

8. Choose the ingress controller

9. Certificates and external access

10. Deploy the platform stack

11. Single Sign-On for Harbor and Rook-Ceph

Harbor

Rook-Ceph

12. Verify the cluster

13. Access the dashboards

14. Back up the cluster (Velero)

Point Velero at an external S3 store

How backups work

How restore (rollback) works

15. Production operations notes

Network isolation with a default-deny NetworkPolicy

Further operational notes

1.2 - Supported Package Versions — Ubuntu / Debian

Kubernetes

KubeOps tools

Helm

Container runtime and etcd

Pre-release versions

Pin the installed versions

1.3 - Supported Package Versions — Red Hat Enterprise Linux (RHEL)

Kubernetes

Container runtime and etcd

KubeOps tools

Helm

Pre-release versions

Pin the installed versions

Dependencies

2 - Joining a Node in a Kubernetes cluster

Join Node Process:

Prerequisits

KOSI Login Recommendation

ETCD Backup Recommendation

Example 1: Joining a Control-Plane Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

Example 1: Joining a Worker Node to a Kubernetes Cluster

3. Validate your values and join the node to the cluster

3 - Deleting a Node from a Kubernetes cluster

Deleting a Node from a Kubernetes cluster

Prerequisites

Worker

Scale down the rook-ceph-operator deployment to 0

Check which hosts and OSDs belong to each zone

Scale down the OSD deployment

Remove the OSD via ceph-tools

Check the MON running on the node

Delete OSD and MON deployments

Remove the deleted mon from the ceph tools

Delete the node from the kubernetes cluster

Scale the rook-ceph-operator deployment back to 1

Timing and health checks

4 - Single Sign-On with Keycloak

4.1 - Single Sign-On (SSO) with Keycloak for Harbor

Prerequisites

Step 1: Prepare Keycloak (Realm, User, and Client)

2. Update the target version in `cluster-values.yaml`

2. Adjust your cluster-values in `zone1`

4. Adjust your `cluster-values.yaml` in `zone2`

6. Adjust your `cluster-values.yaml` `in zone3`