KubeOpsDoku – kubeopctl

Kubeopsctl: Set up a Basic Cluster

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

kubeopctl plattform requirements
best practices for machine setup
setup secure headless environment for communication between admin and all masters / workers
how to install required software
how to use the official KubeOps Website to download kubeopsctl
how to create a basic cluster

After the installation, kubeopctl is available as command line interface.

Notice The intent of this guide is to setup an example cluster with 7 machines. We do not recommend this guide for production environments. For more information about kubeopsctl production environments, see LINK.

Prerequisites

To get the most out of this guide, the following requirements should be met:

basic understanding of Linux environments, bash / shell
basic understanding of text editors, vi / nano
administrator privileges (root) are granted

A total of 7 machines (virtual or physical) are required and need to be set up:

one admin - control plane, this machine will manage all tasks on the cluster and integrated machines
three masters
three workers

The final cluster will have the following structure. Masters and workers are added to two clusters zones.

Important The platform setup using this guide can take up to 3 hours. Depending on the resources available.

Step 1 - Minimal Platform Requirements

kubeopsctl is designed to work with the latest versions of the following operating systems.

Notice Make sure all machines use the same operating system.

Supported Operating Systems


Operating system	Red Hat Enterprise Linux (Version 8.2 or newer)

System requirements Admin Nodes


CPU	2x
Memory	2 GB
Diskspace	50 GB
Internet Access	Yes to use default `KubeOps Registry`

Important: Choose the correct KubeOps Registry based on your version:
"registry1.kubernative.net/lima"
"registry.preprod.kubeops.net"

System requirements Master Nodes


CPU	4x
Memory	8 GB
Diskspace	50 GB

System requirements Worker Nodes


CPU	8x
Memory	16 GB
Diskspace	50 GB 50 GB unformatted non-partitioned disk storage for `ceph`

Important

By default Red Hat Enterprise Linux restricts package management such as installing or updating. Before installing any software, you may need to register and subscribe with the Red Hat Customer Portal.

To register and subscribe, run the command subscription-manager register.

For more information about rook-ceph, see the prerequisites in its official documentation.

If a node fails, the minimum requirements may not be sufficient to guarantee trouble-free operation.

Step 2 - Set up your Machines

You can setup the admin, master and worker nodes as virtual or as physical machines.

During the setup of the machines, make sure that you meet the following requirements:

heed the platform requirements as mentioned above
all machines need to be synchronized with the current time
all machines need to be within the same network environment

Important

During the setup, note the IP addresses of all machines.

Assign unique hostnames for each machine (lowercase letters and numbers only).

We recommended using self-explanatory hostnames (see example table below).

To get the most out of this guide, use the following hostnames for your basic cluster:

Hostnames

Machine / Purpose	Hostnames
Admin	admin
Masters	master1 master2 master3
Workers	worker1 worker2 worker3

Assigning Hostnames manually

If you need to assign hostnames manually, login to the machine and use the hostnamectl set-hostname command.

hostnamectl set-hostname master1

Repeat this process for all machines where necessary.

Remove firewalld on Red Hat Enterprise Linux 8

Important Repeat this process for all master and worker machines where necessary.

If you are using Red Hat Enterprise Linux 8, you must remove firewalld. Kubeopsctl installs nftables by default.
You can use the following commands to remove firewalld:

systemctl disable --now firewalld
systemctl mask firewalld
dnf remove -y firewalld
reboot

Step 3 - Set up Access for the Admin Machine

The admin machine needs secure and headless access to all other machines.

Set up IP Addresses for DNS

It is recommended, that a DNS service is running. If you do not have a DNS service, you need to edit the /etc/hosts file on the admin machine.

Add the following lines at the end of the /etc/hosts file. Replace the IP addresses with the actual addresses you noted during the setup of all machines. Replace the hostnames with the actual hostnames you assigned during the setup of all machines:

10.2.10.10 admin
10.2.10.11 master1
10.2.10.12 master2
10.2.10.13 master3
10.2.10.14 worker1
10.2.10.15 worker2
10.2.10.16 worker3

Set up Secure Access

To securely access the master and worker machines, you need to create a ssh-key-pair (private and public key) on the admin machine. Afterwards copy the public key onto each machine.

To learn more about ssh and key-pairs see our guide on How to set up SSH keys.

Step 4 - Install Podman on the Admin Machine

To ensure compatibility across different containerized environments, kubeopsctl requires the installation of Podman (latest version).

Notice There can be an issue with conflicts with containerd, so it is recommended that containerd.io is removed before installing the podman package.

Install Podman on the admin machine using the inbuilt package manager.

sudo dnf install -y podman

you need to install jq for rook-ceph:

dnf install -y jq

also you need to login to systemd with your os user:

loginctl enable-linger myuser

Notice

By default, kubeopsctl only supports secure registries.

To setup kubeopsctl with insecure registries, see here.

Step 5 - Install kubeopsctl on the Admin Machine

With everything prepared, the next step is to download and install kubeopsctl on the admin machine.

Downloading KOSI

Download your desired version of the kubeopsctl package file (.rpm) from the official download page onto the admin machine.

Installing KOSI
```
sudo dnf install -y <path>/<kosi_rpm>
```
Login to the kubeops hub using kosi

After you input the password, you will gain access to the kubeops hub. Use the following command to begin the login process:
```
  kosi login -u <user>
```

Installing kubeopsctl

Install kubeopsctl using the inbuilt package manager. Replace <path> and <kubeopsctl_rpm> with the respective path and file name of the kubeopsctl package file.

Notice

On installation, kubeopsctl will perform all necessary steps to setup the environment in dependence of your used operating system. This includes the automatic installation / deinstallation of additional software or packages.

Removes software:

runc

Installs Software:

tc, libnftnl
iproute2, libnftnl

To install kubeopsctl use the following command.

sudo dnf install -y <path>/<kubeopsctl_rpm>

Create Work Folders and Setup Environment Variables

After the setup, you need to create work folders where kubeopsctl can save and manage configurations and other settings.

mkdir -p ~/kubeops
mkdir -p ~/kubeops/lima

To work comfortably, you need to assign these folders to the predefined environment variables KUBEOPSROOT and LIMAROOT.

echo 'export KUBEOPSROOT="${HOME}/kubeops"' >> $HOME/.bashrc
echo 'export LIMAROOT="${HOME}/kubeops/lima"' >> $HOME/.bashrc
source $HOME/.bashrc

Verify your Installation

To verify the installation of kubeopsctl on your system, use the command kubeopsctl version.

kubeopsctl version

Step 6 - Configure the Basic Cluster

With everything ready to start, the next step is to configure the cluster.

For configurations, kubeopsctl uses the YAML format.

Use an editor to create and edit the configuration file:

nano ~/basicCluster.yml

Copy and paste all lines into the file. You need to edit specific parameters according to the assigned IP addresses, hostnames etc.:

master/name - set all master hostnames
master/ipAdress - set all master IP addresses
worker/name - set all worker hostnames
worker/ipAdress - set all worker IP addresses

apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
imagePullRegistry: "registry1.kubernative.net/lima"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11
systemCpu: "200m"
systemMemory: "200Mi"

zones:
  - name: zone1
    nodes:
      master:
        - name: master1
          ipAdress: 10.2.10.11
          status: active
          kubeversion: 1.30.0
        - name: master2
          ipAdress: 10.2.10.12
          status: active
          kubeversion: 1.30.0
      worker:
        - name: worker1
          ipAdress: 10.2.10.14
          status: active
          kubeversion: 1.30.0
        - name: worker2
          ipAdress: 10.2.10.15
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: master3
          ipAdress: 10.2.10.13
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: worker3
          ipAdress: 10.2.10.16
          status: active
          kubeversion: 1.30.0


# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

harborValues: 
  harborpass: "password" # change to your desired password
  databasePassword: "Postgres_Password" # change to your desired password
  redisPassword: "Redis_Password" 
  externalURL: http://10.2.10.11:30002 # change to ip adress of master1

prometheusValues:
  grafanaUsername: "user"
  grafanaPassword: "password"

ingressValues:
  externalIPs: []

keycloakValues:
  keycloak:
    auth:
      adminUser: admin
      adminPassword: admin
  postgresql:
    auth:
      postgresPassword: ""
      username: bn_keycloak
      password: ""
      database: bitnami_keycloak
      existingSecret: ""

veleroValues:
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"

apiVersion: kubeops/kubeopsctl/alpha/v5 # mandatory
imagePullRegistry: "registry.preprod.kubeops.net"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11
systemCpu: "200m"
systemMemory: "200Mi"

zones:
  - name: zone1
    nodes:
      master:
        - name: master1
          ipAdress: 10.2.10.11
          status: active
          kubeversion: 1.30.0
        - name: master2
          ipAdress: 10.2.10.12
          status: active
          kubeversion: 1.30.0
      worker:
        - name: worker1
          ipAdress: 10.2.10.14
          status: active
          kubeversion: 1.30.0
        - name: worker2
          ipAdress: 10.2.10.15
          status: active
          kubeversion: 1.30.0
  - name: zone2
    nodes:
      master:
        - name: master3
          ipAdress: 10.2.10.13
          status: active
          kubeversion: 1.30.0  
      worker:
        - name: worker3
          ipAdress: 10.2.10.16
          status: active
          kubeversion: 1.30.0


# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
kubeops-dashboard: true
certman: true
ingress: true 
keycloak: true
velero: true

harborValues: 
  harborpass: "password" # change to your desired password
  databasePassword: "Postgres_Password" # change to your desired password
  redisPassword: "Redis_Password" 
  externalURL: http://10.2.10.11:30002 # change to ip adress of master1

prometheusValues:
  grafanaUsername: "user"
  grafanaPassword: "password"

ingressValues:
  externalIPs: []

keycloakValues:
  keycloak:
    auth:
      adminUser: admin
      adminPassword: admin
  postgresql:
    auth:
      postgresUserPassword: ""
      username: bn_keycloak
      password: ""
      database: bitnami_keycloak
    volumeSize: 8Gi

veleroValues:
  accessKeyId: "your_s3_storage_username"
  secretAccessKey: "your_s3_storage_password"

Step 7 - Start the Basic Cluster

After the configuration is setup correctly, you can start your basic cluster for the first time:

kubeopsctl apply -f ~/basicCluster.yml

Notice This automatic process may take a while. Do not turn off any machine.

Kubeopsctl: Commands, Cluster Health and Modifications

Mon, 01 Jan 0001 00:00:00 +0000

In this quickstart you will learn about:

basic kubeopsctl command line operations
apply changes to cluster, e.g. add new worker

Prerequisites

To get the most out of this guide, the following requirements should be met:

kubeopsctl is installed, see kubeopsctl Installation Guide
a cluster is installed and running
basic understanding of Linux environments, bash / shell
basic understanding of text editors, vi / nano
administrator privileges (root) are granted

Overview of the kubeopsctl CLI

kubeopsctl provides a set of command line operations. For more information, see here.

The main kubeopsctl commands are:

Command	Description
`--version`	Shows the current version of kubeopsctl.
`--help`	Shows an overview of all available commands:
`apply`	Sets up the kubeops platform with a configuration file.
`change registry`	Changes the currently used registry to a different one with a given configuration file.
`drain`	Drains a cluster, zone or node.
`uncordon`	Uncordons a cluster, zone or node. This ensures that while existing pods will remain running on the node, no new pods can be scheduled.
`upgrade`	Upgrades the Kubernetes version of a cluster, zone or node.
`status`	Prints the state of a cluster.

Print kubeopsctl Version

kubeopsctl --version

Show kubeopsctl Help

kubeopsctl --help

Setup Kubernetes Cluster or Apply Changes

Sets up the kubeops platform with a configuration file. Use the flag -f and pass the configuration file, e.g. kubeopsctl.yaml.

kubeopsctl apply -f kubeopsctl.yaml

You can also set the log level to a specific value. Available log levels are:

kubeopsctl apply -f kubeopsctl.yaml -l Debug3

The default log level is Info.

Error
Warning
Info (default log level)
Debug1
Debug2
Debug3

The command kubeopsctl apply can also be used to modify a cluster. For more information see Apply Changes to Cluster within this document.

Change the Registry

Changes the currently used registry to a different one with a given configuration file. For example:

kubeopsctl change registry -f kubeopsctl.yaml -r 10.2.10.11/library -t localhost/library

The -f parameter is used to use yaml parameter file.
The parameter -r is used to pull the docker images which are included in the package to a given local docker registry.
The -t parameter is used to tag the images with localhost. For the szenario that the registry of the cluster is exposed to the admin via a network internal domain name, but this name cant be resolved by the nodes, the flag -t can be used, to use the cluster internal hostname of the registry.

Draining

For draining clusters, zones or nodes use kubeopsctl drain <type>/<name>.

To drain a cluster use and replace <clustername> with the desired cluster:

kubeopsctl drain cluster/<clustername>

To drain a zone use and replace <zonename> with the desired zone:

kubeopsctl drain zone/<zonename>

To drain a node use and replace <nodename> with the desired node:

kubeopsctl drain node/<nodename>

Uncordoning

Uncordoning is similar to draining but draining means remove all running tasks/pods so the node can be fixed or taken offline safely. Uncordoning is just opening it up again for new tasks/pods.

For uncordoning clusters, zones or nodes use kubeopsctl uncordon TYPE/NAME.

To uncordon a cluster use and replace <clustername> with the desired cluster:

kubeopsctl uncordon cluster/<clustername>

To uncordon a zone use and replace <zonename> with the desired zone:

kubeopsctl uncordon zone/<zonename>

To uncordon a node use and replace <nodename> with the desired node:

kubeopsctl uncordon node/<nodename>

Upgrading

Upgrade clusters, zones or nodes by using kubeopsctl upgrade -v <version>.

To upgrade a cluster use and replace <clustername> with the desired cluster:

kubeopsctl upgrade cluster/<clustername> -v 1.26.6

To upgrade a zone use and replace <zonename> with the desired zone:

kubeopsctl upgrade zone/<zonename> -v 1.26.6

To upgrade a node use and replace <nodename> with the desired node:

kubeopsctl upgrade node/<nodename> -v 1.26.6

Check on Health and State of Clusters

To check on health and state of a cluster use the command kubeopsctl status cluster/<clustername>.

For example:

kubeopsctl status cluster/basiccluster

Apply Changes to Cluster

If the cluster is already running, it may be necessary to make additional settings or carry out updates.

For example:

when IP addresses or host names of nodes change
if you want to add or remove nodes

In any case, you need to modify the base configuration and apply the necessary changes to the configuration file. Changes will be applied state wise - short hand: only the difference will be applied.

For more information, see How to Guides.

Example: Add new Worker

If you want to add a new worker to your cluster, edit the configuration file.

In this example we reuse the basic configuration basiccluster.yml from the previous chapter (see Set up a Basic Cluster). Use an editor to edit the configuration file:

nano ~/basiccluster.yml

Add the lines to the configuration file for an additional worker4 at zone2. Also edit the desired ipAdress for the new worker.

apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
imagePullRegistry: "registry1.kubernative.net/lima"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11

zones:
	- name: zone1
		nodes:
			master:
				- name: master1
					ipAdress: 10.2.10.11
					status: active
					kubeversion: 1.30.0
				- name: master2
					ipAdress: 10.2.10.12
					status: active
					kubeversion: 1.30.0
			worker:
				- name: worker1
					ipAdress: 10.2.10.14
					status: active
					kubeversion: 1.30.0
				- name: worker2
					ipAdress: 10.2.10.15
					status: active
					kubeversion: 1.26.2
	- name: zone2
		nodes:
			master:
				- name: master3
					ipAdress: 10.2.10.13
					status: active
					kubeversion: 1.30.0	
			worker:
				- name: worker3
					ipAdress: 10.2.10.16
					status: active
					kubeversion: 1.30.0

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true
keycloak: true
velero: true

rookValues:
	cluster:
		resources:
			mgr:
				requests:
					cpu: 500m
					memory: 1Gi
			mon:
				requests:
					cpu: 500m
					memory: 1Gi
			osd:
				requests:
					cpu: 500m
					memory: 1Gi

harborValues:
	harborpass: "password" # change to your desired password
	databasePassword: "Postgres_Password" # change to your desired password
	redisPassword: "Redis_Password"
	externalURL: http://10.2.10.11:30002 # change to ip adress of master1
	nodePort: 30002
	harborPersistence:
		persistentVolumeClaim:
			registry:
				size: 40Gi
			jobservice:
				jobLog:
					size: 1Gi
			database:
				size: 1Gi
			redis:
				size: 1Gi
			trivy:
				size: 5Gi

prometheusValues:
	grafanaUsername: "user"
	grafanaPassword: "password"

logstashValues:
	volumeClaimTemplate:
		resources:
			requests:
				storage: 1Gi

openSearchValues:
	persistence:
		size: 4Gi

keycloakValues:
	keycloak:
		auth:
			adminUser: admin
			adminPassword: admin
	postgresql:
		auth:
			postgresPassword: ""
			username: bn_keycloak
			password: ""
			database: bitnami_keycloak
			existingSecret: ""

veleroValues:
	namespace: "velero"
	accessKeyId: "your_s3_storage_username"
	secretAccessKey: "your_s3_storage_password"
	useNodeAgent: false
	defaultVolumesToFsBackup: false
	provider: "aws"
	bucket: "velero"
	useVolumeSnapshots: false
	backupLocationConfig:
		region: "minio"
		s3ForcePathStyle: true
		s3Url: "http://minio.velero.svc:9000"

apiVersion: kubeops/kubeopsctl/alpha/v3 # mandatory
imagePullRegistry: "registry.preprod.kubeops.net"
localRegistry: true
clusterName: "example"
kubernetesVersion: "1.30.0"
masterIP: 10.2.10.11

zones:
	- name: zone1
		nodes:
			master:
				- name: master1
					ipAdress: 10.2.10.11
					status: active
					kubeversion: 1.30.0
				- name: master2
					ipAdress: 10.2.10.12
					status: active
					kubeversion: 1.30.0
			worker:
				- name: worker1
					ipAdress: 10.2.10.14
					status: active
					kubeversion: 1.30.0
				- name: worker2
					ipAdress: 10.2.10.15
					status: active
					kubeversion: 1.26.2
	- name: zone2
		nodes:
			master:
				- name: master3
					ipAdress: 10.2.10.13
					status: active
					kubeversion: 1.30.0	
			worker:
				- name: worker3
					ipAdress: 10.2.10.16
					status: active
					kubeversion: 1.30.0

# mandatory, set to true if you want to install it into your cluster
rook-ceph: true
harbor: true
opensearch: true
opensearch-dashboards: true
logstash: true
filebeat: true
prometheus: true
opa: true
headlamp: true
certman: true
ingress: true
keycloak: true
velero: true

rookValues:
	cluster:
		resources:
			mgr:
				requests:
					cpu: 500m
					memory: 1Gi
			mon:
				requests:
					cpu: 500m
					memory: 1Gi
			osd:
				requests:
					cpu: 500m
					memory: 1Gi

harborValues:
	harborpass: "password" # change to your desired password
	databasePassword: "Postgres_Password" # change to your desired password
	redisPassword: "Redis_Password"
	externalURL: http://10.2.10.11:30002 # change to ip adress of master1
	nodePort: 30002
	harborPersistence:
		persistentVolumeClaim:
			registry:
				size: 40Gi
			jobservice:
				jobLog:
					size: 1Gi
			database:
				size: 1Gi
			redis:
				size: 1Gi
			trivy:
				size: 5Gi

prometheusValues:
	grafanaUsername: "user"
	grafanaPassword: "password"

logstashValues:
	volumeClaimTemplate:
		resources:
			requests:
				storage: 1Gi

openSearchValues:
	persistence:
		size: 4Gi

keycloakValues:
	keycloak:
		auth:
			adminUser: admin
			adminPassword: admin
	postgresql:
		auth:
			postgresUserPassword: ""
			username: bn_keycloak
			password: ""
			database: bitnami_keycloak
		volumeSize: "8Gi"

veleroValues:
	namespace: "velero"
	accessKeyId: "your_s3_storage_username"
	secretAccessKey: "your_s3_storage_password"
	useNodeAgent: false
	defaultVolumesToFsBackup: false
	provider: "aws"
	bucket: "velero"
	useVolumeSnapshots: false
	backupLocationConfig:
		region: "minio"
		s3ForcePathStyle: true
		s3Url: "http://minio.velero.svc:9000"

After the modification is set up corretly, you can apply it to your cluster:

kubeopsctl apply -f ~/basiccluster.yml

Best Practices for Changes and Modifications

When configuring the first setup for a cluster, keep your base configuration file, e.g. basiccluster.yml.

Since the modifications are carried out per status / change, we recommend naming the new configuration files and adding a timestamp, if necessary.

For Example:

basiccluster.yml
20240101-add-worker4.yml
20240101-drain-worker2.yml
20240101-update-kubeversion.yml