Setup Cluster

6 minute read

Important: the following commands have to be executed on your cluster admin

⚠ Warning This guide assumes that you are a normal user and are not acting as the root user.

1. Install KOSI

sudo apt update
sudo apt install -y kosi=2.13*

sudo dnf install -y --disableexcludes=kubeops-repo kosi-2.13.0.2-0

# download kosi deb manually and install with
sudo dpkg --install kosi_2.13.0.2-1_amd64.deb

# download kosi rpm manually and install with
sudo rpm --install -v kosi-2.13.0.2-0.x86_64.rpm

2. Set the KUBEOPSROOT env var

Set KUBEOPSROOT and XDG_RUNTIME_DIR in ~/.bashrc

⚠ Warning This file entry has to be adapted to your values before usage.

# file ~/.bashrc
# Append these values to the end of your ~/.bashrc file
export KUBEOPSROOT=/home/<yourUser>/kubeops
export XDG_RUNTIME_DIR=$KUBEOPSROOT

Source .bashrc to apply the values

source ~/.bashrc
echo $KUBEOPSROOT
echo $XDG_RUNTIME_DIR

As a result you should see your KUBEOPSROOT-path two times.

3. Adjust KOSI Configuration

This creates a kubeops directory in your home directory and transfers all necessary files, e.g., the kosi-config and the plugins, to it.

mkdir ~/kubeops
cd ~/kubeops
cp -R /var/kubeops/kosi/ .
cp -R /var/kubeops/plugins/ .

The config.yaml is in your KUBEOPSROOT-path (typically in ~/kubeops/kosi)

Set hub in your kosi config to hub: https://dispatcher.kubeops.net/v4/dispatcher/

⚠ Warning This file entry has to be adapted to your values before usage.

Set the “plugins”-entry in your kosi config to plugins: /home/<yourUser>/kubeops/plugins, where is changed to your username

⚠ Warning This file has to be adapted to your values before usage.

# file $KUBEOPSROOT/kosi/config.yaml
apiversion: kubernative/sina/config/v2

spec:
  hub: https://dispatcher.kubeops.net/v4/dispatcher/ # <-- set hub url
  plugins: <your kubeopsroot>/kubeops/plugins/ # <-- set the path to your plugin folder (~ for home or $KUBEOPSROOT don't work, it has to be the full path)
  workspace: /tmp/kosi/process/
  logging: info
  housekeeping: false
  proxy: false

4. Install KOSI enterprise plugins

kosi install --hub kosi-enterprise kosi/enterprise-plugins:2.0.0

⚠ Warning This command has to be adapted to your values before usage.

kosi login -u <yourUser>

At this point it is normal if you get the following error message:
Error: The login to registry is temporary not available. Please try again later.
The reason for this is that podman is not yet installed.

6. Install kubeopsctl

sudo apt update
sudo apt install -y kubeopsctl=2.0*

sudo dnf install -y --disableexcludes=kubeops-repo kubeopsctl-2.0.1.0

# download kubeopsctl deb manually from https://kubeops.net and install with
sudo dpkg --install kubeopsctl_2.0.1.0-1_amd64.deb

# download kubeopsctl rpm manually from https://kubeops.net and install with
sudo rpm --install -v kubeopsctl-2.0.1.0-0.x86_64.rpm

7. Create a cluster-values.yaml configuration file

⚠ Warning This file has to be adapted to your values before usage.

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: <your cluster name>
clusterUser: <your user name>
kubernetesVersion: <your kubernetesversion>
kubeVipEnabled: false
virtualIP: <your master1 ip>
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: <your kubeopsroot path>
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true
zones:
# IMPORTANT: The following part has to be adapted so that every one of your masternodes and workernodes is included
# This file only includes the minimum requirements for the amount of masters and workers and an example usage of zones
# You should adapt this part to your amount of masters and workers and cluster them into as many zones as you like
- name: zone1
  nodes:
  - name: <your master1 hostname>
    iPAddress: <your master1 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker1 hostname>
    iPAddress: <your worker1 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone2
  nodes:
  - name: <your master2 hostname>
    iPAddress: <your master2 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker2 hostname>
    iPAddress: <your worker2 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>
- name: zone3
  nodes:
  - name: <your master3 hostname>
    iPAddress: <your master3 ip>
    type: controlplane
    kubeVersion: <kubernetesversion from above>
  - name: <your worker3 hostname>
    iPAddress: <your worker3 ip>
    type: worker
    kubeVersion: <kubernetesversion from above>

Full Example

# file cluster-values.yaml
apiVersion: kubeops/kubeopsctl/cluster/beta/v1
imagePullRegistry: registry.kubeops.net/kubeops/kubeops
airgap: false
clusterName: myCluster
clusterUser: myuser
kubernetesVersion: 1.32.2
kubeVipEnabled: false
virtualIP: 10.2.10.110
firewall: nftables
pluginNetwork: calico
containerRuntime: containerd
kubeOpsRoot: /home/myuser/kubeops
serviceSubnet: 192.168.128.0/17
podSubnet: 192.168.0.0/17
debug: true
systemCpu: 250m
systemMemory: 256Mi
packageRepository: https://packagerepo.kubeops.net/
changeCluster: true
zones:
- name: zone1
  nodes:
  - name: dev07-master1-ubuntu2404
    iPAddress: 10.2.10.110
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker1-ubuntu2404
    iPAddress: 10.2.10.210
    type: worker
    kubeVersion: 1.32.2
- name: zone2
  nodes:
  - name: dev07-master2-ubuntu2404
    iPAddress: 10.2.10.120
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker2-ubuntu2404
    iPAddress: 10.2.10.220
    type: worker
    kubeVersion: 1.32.2
- name: zone3
  nodes:
  - name: dev07-master3-ubuntu2404
    iPAddress: 10.2.10.130
    type: controlplane
    kubeVersion: 1.32.2
  - name: dev07-worker3-ubuntu2404
    iPAddress: 10.2.10.230
    type: worker
    kubeVersion: 1.32.2

7.1 Using KubeVip in your Cluster (optional)

If you want to use KubeVip to setup your Cluster, you need a virtual ip for that. Also you have to set kubeVipEnabled to true and set your virtualIP. If you dont want to use KubeVip you have to set kubeVipEnabled to false and set your first controlplane as your virtualIP in your cluster-values.yaml in the Setup. Refer to the official KubeVip-documentation for details here.

Examples:

kubeVipEnabled: true
virtualIP: <IP in your cluster ip range which is not given yet>

kubeVipEnabled: false
virtualIP: <master1 ip>

Note The virtual IP must be in the same network as the machines and must not already be in use.

8. Pull required KOSI packages

⚠ Warning This command should be adapted to your value before usage.

If you do not specify a parameter, the current Kubernetes version 1.32.2 will be pulled.
With parameter --kubernetesVersion <your wanted Kubernetesversion> you can pull an older Kubernetes version.
Available Kubernetes versions are 1.32.2 , 1.32.3 , 1.32.9 . 1.32.10 . 1.33.3 . 1.33.5 . 1.34.1 .

kubeopsctl pull

kubeopsctl pull --kubernetesVersion <x.xx.x>

9. Install podman

kosi install -p $KUBEOPSROOT/lima/podman_5.2.2.tgz -f cluster-values.yaml

⚠ Warning Updating podman, e.g, by using apt update or dnf update, might break your cluster.
On RHEL9, to update your applications but exclude podman you can run the following command: sudo dnf update --exclude=podman
On Ubuntu, you can hold the package with the command sudo apt-mark hold podman which makes apt skip podman on updates.

10. Install helm

kosi install -p $KUBEOPSROOT/lima/helm_v3.16.4.tgz

11. Install kubernetes tools (kubectl)

⚠ Warning This command has to be adapted to your values before usage.

Make sure the kubernetes version matches the one you pulled before.

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_<your kubernetes version>.tgz -f cluster-values.yaml

This command also installs kubelet and kubeadm. You can either mask or delete them on your admin as they are not necessary for the cluster creation process.

Full Example

kosi install -p $KUBEOPSROOT/lima/kubernetes-tools_1.32.2.tgz -f cluster-values.yaml

12. Cluster Setup

⚠ Warning This command has to be adapted to your values before usage.

Make sure that you are logged in on hub and registry.

kosi login -u <your username>

Now the login for hub and registry should be successful!

Make sure that you changed the kosi config.yaml.

cat $KUBEOPSROOT/kosi/config.yaml

Make sure that you pulled all required packages.

ls -1 $KUBEOPSROOT/lima

Install Kubernetes Cluster with kubeopsctl. Cluster setup takes about 10 to 15 minutes.

kubeopsctl apply -f cluster-values.yaml

Setup Cluster

1. Install KOSI

2. Set the KUBEOPSROOT env var

3. Adjust KOSI Configuration

4. Install KOSI enterprise plugins

5. Login with your user

6. Install kubeopsctl

7. Create a cluster-values.yaml configuration file

7.1 Using KubeVip in your Cluster (optional)

8. Pull required KOSI packages

9. Install podman

10. Install helm

11. Install kubernetes tools (kubectl)

12. Cluster Setup