Fileformats in kubeopsctl

cluster-values.yaml

The cluster-values.yaml defines every aspect of the cluster itself. It has no influence over which applications get installed.

apiVersion: kubeops/kubeopsctl/cluster/beta/v1 # required
imagePullRegistry: registry.kubeops.net/kubeops/kubeops # required
airgap: true # optional, default: true
clusterName: myCluster # required 
clusterUser: root # optional, default: root
kubernetesVersion: 1.32.2 # required
kubeVipEnabled: false # optional, default: true
virtualIP: 10.2.10.110 # required
firewall: nftables # optional, default: nftables
pluginNetwork: calico # optional, default: calico | possible alternative: cilium
containerRuntime: containerd # optional, default: containerd
kubeOpsRoot: /var/kubeops # optional, default: /var/kubeops
serviceSubnet: 192.168.128.0/17 # optional, default: 192.168.128.0/17
podSubnet: 192.168.0.0/17 # optional, default: 192.168.0.0/17
debug: false # optional, default: false
systemCpu: 250m # optional, default: 250m
systemMemory: 256Mi # optional, default: 256Mi
packageRepository: local # optional, default: local
changeCluster: true # optional, default: true

advanced:
  localRegistry:
    mountPath: /mnt/registry

zones: # required
- name: zone1 # required
  nodes: # required
  - name: master1 # required
    iPAddress: 10.2.10.110 # required
    type: controlplane # required
    kubeVersion: 1.32.2 # required

Detailed Parameter Information

Key	Possible Values	Additional Info
pluginNetwork	Calico, Cilium

enterprise-values.yaml

The enterprise-values.yaml defines all enterprise applications currently available for you to install in your cluster via kubeopsctl.
You can append multiple of them into a single enterprise-values.yaml as shown in the first example.

For each application you have 2 ways to change its values:

the standard values
the advanced values

While the standard values only cover predefined keys, the advanced values let you change every key available in the helm chart. Keep in mind, that the standard values overwrite the advanced values if both are set.

Each as optional marked line can be skipped unless otherwise stated. If a optional line is skipped, its default value will be used instead. If there is no default value, it can just be omitted and won’t affect the cluster and/or the application

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
  - name: opa-gatekeeper
    enabled: true
    values:
      standard:
        namespace: gatekeeper-system # optional, default is gatekeeper-system
      advanced:
  - name: filebeat-os
    enabled: false
    values:
      standard:
        namespace: logging # optional, default is logging
      advanced:

  ### Values for Rook-Ceph ###
  ### For detailed explanation for each key see: https://artifacthub.io/packages/helm/rook/rook-ceph?modal=values ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: rook-ceph
    enabled: true
    values:
      standard:
        namespace: rook-ceph # optional, default is rook-ceph
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route the ceph dashboard via Traefik
        cluster:
          spec:
            dataDirHostPath: "/var/lib/rook" # optional, default is /var/lib/rook
          resources:
            mgr:
              requests:
                cpu: "500m" # optional, default is 500m, limit: 1000m
                memory: "512Mi" # optional, default is 1Gi, limit: 1Gi
            mon:
              requests:
                cpu: "1" # optional, default is 1, limit: 2000m
                memory: "1Gi" # optional, default is 1Gi, limit: 2Gi
            osd:
              requests:
                cpu: "1" # optional, default is 1, limit: 2
                memory: "1Gi" # optional, default is 4Gi, limit: 4Gi
        operator:
          data:
            rookLogLevel: "DEBUG" # optional, default is DEBUG
      advanced: 
        cluster: # All values from https://artifacthub.io/packages/helm/rook/rook-ceph-cluster?modal=values are overwritable
        operator: # All values from https://artifacthub.io/packages/helm/rook/rook-ceph?modal=values are overwritable

  ### Values for Harbor deployment ###
  ### For detailed explanation for each key see: https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: harbor
    enabled: true
    values:
      standard:
        namespace: harbor # optional, default is harbor
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        harborpass: "password" # required: set password for harbor access
        databasePassword: "Postgres_Password" # required: set password for database access
        redisPassword: "Redis_Password" # required: set password for redis access
        externalURL: http://10.2.10.11:30002 # required, the ip address and port, from which harbor is accessable outside of the cluster
        nodePort: 30002 # required
        hostname: harbor.local # required
        harborPersistence:
          persistentVolumeClaim:
            registry:
              size: 40Gi # optional, default is 40Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            jobservice:
              jobLog:
                size: 1Gi # optional, default is 1Gi
                storageClass: "rook-cephfs" #optional, default is rook-cephfs
            database:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            redis:
              size: 1Gi # optional, default is 1Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
            trivy: 
              size: 5Gi # optional, default is 5Gi
              storageClass: "rook-cephfs" #optional, default is rook-cephfs
      advanced: #  All values from https://artifacthub.io/packages/helm/harbor/harbor/1.8.1#configuration are overwritable

  ### Values for cert-manager deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: cert-manager
    enabled: true
    values:
      standard:
        namespace: cert-manager # optional, default is cert-manager
        replicaCount: 3
        logLevel: 2
        secretName: root-secret
      advanced: # All values from https://artifacthub.io/packages/helm/cert-manager/cert-manager?modal=values are overwritable
      ## add helm values here
      ca: # LetsEncrypt ClusterIssuer / ACME HTTP01 solver settings
        # emailLetsEncrypt: <your_email@domain.com> # default: example@example.com --> must configure
        # ingressName: <ingress_name> # default: nginx; set to "traefik" to solve ACME HTTP01 via Traefik

  ### Values for ingress-nginx deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: ingress-nginx
    enabled: true
    values:
      standard:
        namespace: ingress-nginx # optional, default is ingress
      advanced: # All values from https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx?modal=values are overwritable

### Values for Traefik deployment ###
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false 
localRegistry: false
packages:
- name: traefik
  enabled: true
  values:
    standard:
      namespace: traefik # optional default is traefik
    advanced: # All values from https://artifacthub.io/packages/helm/traefik/traefik/39.0.0?modal=values are overwritable

### Values for keycloak deployment ###
apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: keycloak
  enabled: true
  values:
    standard:
      namespace: keycloak # Optional, default is "keycloak"
      ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
      storageClass: "rook-cephfs" # Optional, default is "rook-cephfs"
      keycloak:
        auth:
          adminUser: admin # Optional, default is admin
          adminPassword: admin # Optional, default is admin, only used during initial installation
          existingSecret: "" # Optional, default is ""
      postgresql:
        auth:
          postgresPassword: "" # Optional, default is ""
          username: bn_keycloak # Optional, default is "bn_keycloak"
          password: "" # Optional, default is ""
          database: bitnami_keycloak # Optional, default is "bitnami_keycloak"
          existingSecret: "" # Optional, default is ""
    advanced: # All values from https://artifacthub.io/packages/helm/bitnami/keycloak?modal=values are overwritable, except values that are only applied during the installation, such as the Keycloak admin password

keycloak.auth.adminPassword is only applied during the installation. After Keycloak has been installed, changing this value in enterprise-values.yaml does not change the Keycloak password. To change the Keycloak password after installation, change it directly in Keycloak.

  ### Values for filebeat deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: filebeat-os
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging   
      advanced: # All values from https://artifacthub.io/packages/helm/elastic/filebeat?modal=values are overwritable

  ### Values for Logstash deployment ###
  ### For detailed explanation for each key see: https://github.com/elastic/helm-charts/releases/tag/v7.16.3 ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: logstash-os
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging
        volumeClaimTemplate:
          accessModes: 
            - ReadWriteMany #optional, default is [ReadWriteMany]
          resources:
            requests:
              storage: 1Gi # required, depending on storage capacity
          storageClass: "rook-cephfs" #optional, default is rook-cephfs
      advanced: # All values from https://artifacthub.io/packages/helm/elastic/logstash?modal=values are overwritable

  ### Values for OpenSearch-Dashboards deployment ###
  ### For detailed explanation for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch-dashboards ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: opensearch-dashboards
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        nodePort: 30050
        hostname: opensearch-dashboard.local # optional, default is opensearch-dashboard.local; external ingress hostname, determines the Keycloak SSO URL
        adminPassword: "Password@@123456" # optional, default is Password@@123456; must match opensearch-os adminPassword
      advanced: # All values from https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch-dashboards?modal=values are overwritable

  ### Values for OpenSearch deployment ###
  ### For detailed explanation for each key see: https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: opensearch-os
    enabled: true
    values:
      standard:
        namespace: logging # optional, default is logging
        hostname: opensearch-dashboard.local # optional, default is opensearch-dashboard.local; Keycloak host for SSO, must match opensearch-dashboards
        adminPassword: "Password@@123456" # optional, default is Password@@123456; OpenSearch admin password, enforced after deploy
        opensearchJavaOpts: "-Xmx512M -Xms512M" # optional, default is -Xmx512M -Xms512M
        resources:
          requests:
            cpu: "250m" # optional, default is 250m
            memory: "1024Mi" # optional, default is 1024Mi
          limits:
            cpu: "300m" # optional, default is 300m
            memory: "3072Mi" # optional, default is 3072Mi
        persistence:
          size: 4Gi # required
          enabled: "true" # optional, default is true
          enableInitChown: "false" # optional, default is false
          labels:
            enabled: "false" # optional, default is false
          storageClass: "rook-cephfs" # optional, default is rook-cephfs
          accessModes:
            - "ReadWriteMany" # optional, default is {ReadWriteMany}
        securityConfig:
          enabled: false # optional, default value: false
          ### Additional values can be set, if securityConfig is enabled:
          # path: "/usr/share/opensearch/plugins/opensearch-security/securityconfig"
          # actionGroupsSecret:
          # configSecret:
          # internalUsersSecret: internal-users-config-secret
          # rolesSecret:
          # rolesMappingSecret:
          # tenantsSecret:
          # config:
          #   securityConfigSecret: ""
          #   dataComplete: true
          #   data: {}
        replicas: "3" # optional, default is 3
      advanced: # All values from https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch?modal=values are overwritable

  ### Values for Prometheus deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: kube-prometheus-stack
    enabled: true
    values:
      standard:
        namespace: monitoring # optional, default is monitoring
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        privateRegistry: false # optional, default is false
        grafanaUsername: "user" # optional, default is user
        grafanaPassword: "password" # optional, default is password
        grafanaResources:
          storageClass: "rook-cephfs" # optional, default is rook-cephfs
          storage: 5Gi # optional, default is 5Gi
          nodePort: 30211 # optional, default is 30211

        prometheusResources:
          storageClass: "rook-cephfs" # optional, default is rook-cephfs
          storage: 25Gi # optional, default is 25Gi
          retention: 10d # optional, default is 10d
          retentionSize: "24GB" # optional, default is 24GB
          nodePort: 32090
      advanced: # All values from https://artifacthub.io/packages/helm/prometheus-community/prometheus?modal=values-schema are overwritable

  ### Values for OPA deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: opa-gatekeeper
    enabled: true
    values:
      standard:
        namespace: gatekeeper-system # optional, default is gatekeeper-system
      advanced: # All values from https://artifacthub.io/packages/helm/gatekeeper/gatekeeper/3.1.1?modal=values are overwritable

  ### Values for KubeOps-Dashboard (Headlamp) deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: kubeops-dashboard
    enabled: true
    values:
      standard:
        namespace: monitoring # optional, default is monitoring
        ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
        service:
          nodePort: 30007
      advanced: # All values from https://artifacthub.io/packages/helm/headlamp/headlamp?modal=values are overwritable

  ### Values for velero deployment ###
  apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
  deleteNs: false
  localRegistry: false
  packages:
  - name: velero
    enabled: true
    values:
      standard:
        namespace: velero # Optional, default is "velero"
        accessKeyId: "your_s3_storage_username"
        secretAccessKey: "your_s3_storage_password"
        useNodeAgent: false
        defaultVolumesToFsBackup: false
        provider: "aws"
        bucket: "velero"
        useVolumeSnapshots: false
        backupLocationConfig:
          region: "minio"
          s3ForcePathStyle: true
          s3Url: "http://minio.velero.svc:9000"
      advanced: # All values from https://artifacthub.io/packages/helm/vmware-tanzu/velero?modal=values are overwritable

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
# used in deleteLogstash.sh
deleteNs: false # TODO undocumented values! dangerous if other charts are installed in the namespace :-o
# used in some packages
localRegistry: false # create pull secret for registry 
packages:
- name: kubevirt-manager
  enabled: true
  values:
    standard:
      namespace: kubevirt # optional default is kubevirt
      hostname: kubeops-vms.local
      nodeport: 30733 # optional default is 30733
      ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
    advanced: {}

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
# used in deleteLogstash.sh
deleteNs: false # TODO undocumented values! dangerous if other charts are installed in the namespace :-o
# used in some packages
localRegistry: false # create pull secret for registry 
packages:
- name: kubevirt-manager
  enabled: true
  values:
    standard:
      namespace: kubevirt # optional default is kubevirt
      hostname: kubeops-vms.local
      nodeport: 30733 # optional default is 30733
      ingressClassName: nginx # optional, default is nginx; set to "traefik" to route via Traefik
    advanced: {}
- name: kubevirt
  enabled: true
  hub: kubeops-vm
  values:
    standard:  
      namespace: kubevirt
      imagePullSecretCred:
        registry: registry.preprod.kubeops.net
        user: kubeops
        password: changeme # must be CLI Token from harbor 
      kubevirtobject:
        spec:
          configuration:
            developerConfiguration:
              logVerbosity:
                virtLauncher: 2
                virtHandler: 3
                virtController: 4
                virtAPI: 5
                virtOperator: 6
              featureGates:
              - ExpandDisks
              - CPUManager
              - ExperimentalIgnitionSupport
              - HypervStrictCheck
              - Sidecar
              - HostDevices
              - Snapshot
              - VMExport
              - HotplugVolumes
              - HostDisk
              - ExperimentalVirtiofsSupport
              - DownwardMetricsFeatureGate
              - Root
              - ClusterProfiler
              - WorkloadEncryptionSEV
              - VSOCK
              - DisableCustomSELinuxPolicy
              - KubevirtSeccompProfile
              - DisableMDEVConfiguration
              - PersistentReservation
              - VMPersistentState
              - MultiArchitecture
              - AutoResourceLimitsGate
              - AlignCPUs
              - VolumesUpdateStrategy
              - VolumeMigration
              #- NodeRestriction # Forbidden setting: Do not use!
              - InstancetypeReferencePolicy
              useEmulation: false
      cdiobject:
        spec:
          config:
            insecureRegistries:
            - registry.preprod.kubeops.net # Need to test if this is really necessary
            scratchSpaceStorageClass: rook-cephfs # Optional; Default: rook-cephfs

apiVersion: kubeops/kubeopsctl/enterprise/beta/v1
deleteNs: false
localRegistry: false
packages:
- name: rook-ceph
 enabled: true
 values:
   standard:
     namespace: rook-ceph
     cluster:
       resources:
         mgr:
           requests:
             cpu: "500m"
             memory: "512Mi"
         mon:
           requests:
             cpu: "1"
             memory: "1Gi"
         osd:
           requests:
             cpu: "1"
             memory: "1Gi"
       dashboard:
         enabled: true
     operator:
       data:
         rookLogLevel: "DEBUG"
- name: harbor
 enabled: true
 values:
   standard:
     namespace: harbor
     harborpass: "password"
     databasePassword: "password"
     redisPassword: "password"
     externalURL: http://10.2.10.11:30002
     nodePort: 30002
     hostname: harbor.local
     harborPersistence:
       persistentVolumeClaim:
         registry:
           size: 40Gi
           storageClass: "rook-cephfs"
         jobservice:
           jobLog:
             size: 1Gi
             storageClass: "rook-cephfs"
         database:
           size: 1Gi
           storageClass: "rook-cephfs"
         redis:
           size: 1Gi
           storageClass: "rook-cephfs"
         trivy: 
           size: 5Gi
           storageClass: "rook-cephfs"
   advanced:
- name: cert-manager
 enabled: true
 values:
   standard:
     namespace: cert-manager
     replicaCount: 3
     logLevel: 2
     secretName: root-secret
   advanced:
- name: ingress-nginx
 enabled: true
 values:
   standard:
     namespace: ingress-nginx
   advanced:
- name: keycloak
 enabled: true
 values:
   standard:
     namespace: keycloak
     storageClass: "rook-cephfs"
     keycloak:
       auth:
         adminUser: admin
         adminPassword: admin
         existingSecret: ""
     postgresql:
       auth:
         postgresPassword: ""
         username: bn_keycloak
         password: ""
         database: bitnami_keycloak
         existingSecret: ""
   advanced:
- name: filebeat-os
 enabled: true
 values:
   standard:
     namespace: logging
   advanced:
- name: logstash-os
 enabled: true
 values:
   standard:
     namespace: logging
     volumeClaimTemplate:
       accessModes: 
         - ReadWriteMany
       resources:
         requests:
           storage: 1Gi
       storageClass: "rook-cephfs"
   advanced:
- name: opensearch-dashboards
 enabled: true
 values:
   standard:
     namespace: logging
     nodePort: 30050
     hostname: opensearch-dashboard.local
     adminPassword: "Password@@123456"
   advanced:
- name: opensearch-os
 enabled: true
 values:
   standard:
     namespace: logging
     hostname: opensearch-dashboard.local
     adminPassword: "Password@@123456"
     opensearchJavaOpts: "-Xmx512M -Xms512M"
     resources:
       requests:
         cpu: "250m"
         memory: "1024Mi"
       limits:
         cpu: "300m"
         memory: "3072Mi"
     persistence:
       size: 4Gi
       enabled: "true"
       enableInitChown: "false"
       labels:
         enabled: "false"
       storageClass: "rook-cephfs"
       accessModes:
         - "ReadWriteMany"
     securityConfig:
       enabled: false
     replicas: "3"
   advanced:
- name: kube-prometheus-stack
 enabled: true
 values:
   standard:
     namespace: monitoring
     privateRegistry: false
     grafanaUsername: "user"
     grafanaPassword: "password"
     grafanaResources:
       storageClass: "rook-cephfs"
       storage: 5Gi
       nodePort: 30211
     prometheusResources:
       storageClass: "rook-cephfs"
       storage: 25Gi
       retention: 10d
       retentionSize: "24GB"
       nodePort: 32090
   advanced:
- name: opa-gatekeeper
 enabled: true
 values:
   standard:
     namespace: opa-gatekeeper
   advanced:
- name: kubeops-dashboard
 enabled: true
 values:
   standard:
     namespace: monitoring
     service:
       nodePort: 30007
   advanced:
- name: velero
 enabled: true
 values:
   standard:
     namespace: velero
     accessKeyId: "your_s3_storage_username"
     secretAccessKey: "your_s3_storage_password"
     useNodeAgent: false
     defaultVolumesToFsBackup: false
     provider: "aws"
     bucket: "velero"
     useVolumeSnapshots: false
     backupLocationConfig:
       region: "minio"
       s3ForcePathStyle: true
       s3Url: "http://minio.velero.svc:9000"
   advanced:
- name: kubevirt-manager
 enabled: true
 values:
   standard:
     namespace: kubevirt
     hostname: kubeops-vms.local
     nodeport: 30733
     ingressClassName: nginx
   advanced: {}
- name: kubevirt
 enabled: true
 hub: kubeops-vm
 values:
   standard:  
     namespace: kubevirt
     imagePullSecretCred:
       registry: registry.preprod.kubeops.net
       user: kubeops
       password: changeme
     kubevirtobject:
       spec:
         configuration:
           developerConfiguration:
             logVerbosity:
               virtLauncher: 2
               virtHandler: 3
               virtController: 4
               virtAPI: 5
               virtOperator: 6
             featureGates:
             - ExpandDisks
             - CPUManager
             - ExperimentalIgnitionSupport
             - HypervStrictCheck
             - Sidecar
             - HostDevices
             - Snapshot
             - VMExport
             - HotplugVolumes
             - HostDisk
             - ExperimentalVirtiofsSupport
             - DownwardMetricsFeatureGate
             - Root
             - ClusterProfiler
             - WorkloadEncryptionSEV
             - VSOCK
             - DisableCustomSELinuxPolicy
             - KubevirtSeccompProfile
             - DisableMDEVConfiguration
             - PersistentReservation
             - VMPersistentState
             - MultiArchitecture
             - AutoResourceLimitsGate
             - AlignCPUs
             - VolumesUpdateStrategy
             - VolumeMigration
             - InstancetypeReferencePolicy
             useEmulation: false
     cdiobject:
       spec:
         config:
           insecureRegistries:
           - registry.preprod.kubeops.net
           scratchSpaceStorageClass: rook-cephfs