Measures

Enable Comprehensive Audit Logging for the Kubernetes API

Configure the Kubernetes API server with an audit policy that captures key events, such as the creation, update, and deletion of resources, particularly focusing on sensitive resources like secrets.

Benefit: Capturing detailed logs ensures visibility into critical API requests, enabling better monitoring, detection, and forensic analysis of activities within the cluster.

Example Audit Policy File:

apiVersion: audit.k8s.io/v1

kind: Policy

rules:

- level: Metadata

  resources:

  - group: ""

    resources: ["pods", "services", "namespaces"]

- level: Request

  users: ["admin"]

  verbs: ["create", "update", "patch", "delete"]

- level: None

  users: ["system:serviceaccount:kube-system:default"]

  verbs: ["get", "list", "watch"]

- level: RequestResponse

  resources:

  - group: ""

    resources: ["secrets"]

  omitStages:

  - "RequestReceived"

Define Clear Log Retention and Rotation Policies

Set up retention policies to ensure audit logs are kept for the required duration based on organizational or regulatory needs. Implement log rotation to manage log file sizes and prevent disk exhaustion.

Benefit: Proper retention ensures logs are available when needed for audits or investigations without consuming excessive resources.

Retention Configuration:

--audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100

Secure Log Storage and Access

Store audit logs in a secure location with limited access. Implement RBAC (Role-Based Access Control) policies to ensure that only authorized personnel can access and manage these logs.

Benefit: Securing logs prevents tampering, unauthorized access, or accidental deletion, ensuring data integrity and compliance.

RBAC for Audit Log Access:

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

  namespace: kube-system

  name: audit-log-reader

rules:

- apiGroups: [""]

  resources: ["pods/log"]

  verbs: ["get", "list"]

Implement Monitoring and Alerting for Critical Events

Integrate audit logs with monitoring systems like Prometheus, ELK Stack, or Splunk to track critical events. Set up alerts for high-privilege actions, such as changes to secrets or service accounts.

Benefit: Real-time monitoring and alerting help detect and respond to suspicious activity promptly, reducing the risk of undetected security breaches.

Prometheus Alert Example:

groups:

- name: KubernetesAuditLogs

  rules:

  - alert: HighPrivilegeAction

    expr: rate(kube_audit_event_count{verb="create", resource="secrets"}[1m]) > 1

    for: 5m

    labels:

      severity: critical

    annotations:

      summary: "High privilege action detected"

      description: "High privilege action ({{ $labels.verb }}) detected on resource {{ $labels.resource }} by user {{ $labels.user }}."

A well-implemented audit log policy in Kubernetes ensures that all critical actions performed through the API are monitored and recorded. This enhances security by providing visibility into cluster activities, facilitating compliance with regulatory requirements, and enabling the timely detection of security incidents. By defining clear retention policies, securing log access, and setting up effective monitoring and alerting systems, organizations can maintain robust operational security in their Kubernetes environments