Blogs

As the world of container orchestration continues to evolve, Kubernetes 1.28 emerges as a testament to relentless progress and innovation. With features ranging from the revamped kubectl events command to sophisticated dynamic resource allocation, this version promises to redefine the way we perceive and manage container ecosystems.

Whether you're a cluster admin interested in monitoring metrics or a developer eyeing performance enhancements, Kubernetes 1.28 holds a plethora of upgrades designed to improve your orchestration experience.

Dive in with us as we unpack the remarkable enhancements that will shape the next generation of container management.

 

Four Intriguing Enhancements in Kubernetes 1.28

1. Sidecar Container Integration
   
   The sidecar pattern has long been recognized as a pivotal practice in Kubernetes. With the 1.28 release, sidecar containers are officially supported, streamlining functions like metrics collection, secret management, and service mesh integration. This development promises more robust Kubernetes workloads, much to the delight of developers, admins, and end-users.
   
   
2. Revamping Kubernetes Jobs
   
   This release introduces several notable improvements to Kubernetes jobs, especially beneficial for Machine Learning tasks. From ensuring sidecars don't hinder job completion to advanced features like retry mechanisms for pod failures and more sophisticated job control, Kubernetes is clearly poised to be a significant player in the ML space.
   
   
3. Enhancements for Rolling Upgrades
   
   Upgrading system components without significant downtimes has always been a challenge. Kubernetes 1.28 introduces features to make these rolling upgrades smoother and more reliable. Notably, the new functionalities promise reduced connection drops during upgrades, ensuring minimal service disruptions and a more seamless transition.
   
   
4. Community-Driven Package Management
   
   In a bid to foster greater community involvement, Kubernetes is shifting away from Google-centric infrastructures, opting for community-owned repositories for package distributions. This evolution underscores the project's maturity and hints at a future driven by diverse contributions.

Advanced API Features

• Consistent Reads From Cache: Kubernetes 1.28 brings an efficiency overhaul to API requests, particularly GET and LIST. The introduction of WatchProgressRequest optimizes performance by fetching information from etcd's watch cache.
  
  
• Data Streaming with Informers: Building on the previous feature, the API server can now stream data changes in real-time, reducing the memory overhead of repetitive LIST requests.
  
  
• Mixed Version Proxy: Addressing the challenges of cluster upgrades, the kube-apiserver now has the capability to delegate resource requests to the appropriate peer servers, ensuring seamless operations across different server versions.
  
  
• Improved Custom Resource Definitions (CRD) Handling: Kubernetes 1.28 simplifies the task of managing CRD schema changes. For instance, edits that don't interfere with new validation rules won't trigger unnecessary validation errors, making it more user-friendly.
  
  
• CRD Validation via Expression Language: The new release also allows for the definition of validation rules directly within custom resources, eliminating the need for additional webhooks. This enhancement has undergone improvements since its Beta phase in Kubernetes 1.25.
  
  
• Enhanced Admission Control with CEL: Kubernetes 1.28 adopts the Common Expression Language (CEL) for its admission controller, offering more precise validation mechanisms. For instance, admins can set up rules to deny object creations that breach certain criteria.

Application-Based Features

• Refined Job Handling: Jobs in Kubernetes can now be replaced only once they're fully terminated, catering to specific workload requirements. Admins can easily set this behavior using the podReplacementPolicy field.
  
  
• Fine-Tuned Job Retries: The release also introduces granular control over job retries. Admins can now define retry limits for specific indices within a job, ensuring more robust and fault-tolerant operations.

Job Timestamp Annotations Added in CronJobs

SIG: sig-apps

Status: Beta

Feature: CronJobCreationAnnotation (Enabled by default)

Now, the CronJob controller includes the expected timestamp for when a job will run:

batch.kubernetes.io/cronjob-scheduled-timestamp: "2016-05-19T03:00:00-07:00"

 

Adjustable Job Pod Failure Management

SIG: sig-apps

Status: Beta

Feature: JobPodFailurePolicy (Enabled by default)

Manage pod failures in jobs with the new podFailurePolicy. It lets you control retries without impacting the backoffLimit.

 

Pod Index Label Introduced for StatefulSets & Indexed Jobs

SIG: sig-apps

Status: Beta

Feature: PodIndexLabel (Enabled by default)

Jobs & StatefulSet controllers now support the pod index as a label.

 

KMS v2 Enhancements

SIG: sig-auth

Status: Beta

Feature: KMSv2 (Enabled by default)

KMS v2 aims to solve several challenges of its predecessor, like performance and manual key rotation. The updated version brings various improvements, including a revamped key hierarchy and better observability.

 

Reduction of Old Service Account Tokens

SIG: sig-auth

Status: Beta

Features: LegacyServiceAccountTokenTracking & LegacyServiceAccountTokenCleanUp (Both enabled by default)

From Kubernetes 1.22, service account tokens have transitioned from being auto-generated to being sourced from the TokenRequest API. The new changes aim to further minimize the older style tokens.

 

Know Your Authenticated Self with the Auth API

SIG: sig-auth

Status: Stable

Feature: APISelfSubjectReview (Enabled by default)

The new API reveals the authenticated identity, be it a user or a ServiceAccount. The info can be accessed with a CLI command: kubectl alpha auth whoami, assisting in diagnosing authentication issues.

 

Kubectl Delete Gets Interactive with New Flag

SIG: sig-cli

Status: Alpha

Using the -i or --interactive flag during kubectl delete provides a preview of objects slated for deletion, allowing user confirmation.

 

Kubectl Events Command Upgrade

SIG: sig-cli

Status: Stable

The revamped kubectl events command is designed to overcome the limitations of kubectl get events, offering improved watch controls and filtering. It's stable from Kubernetes 1.28.

 

Metrics Stability Enhancement

SIG: sig-instrumentation

Status: Beta

The new update introduces two stability levels, 'Internal' and 'Beta', for metrics. The 'Beta' metrics provide more reliability and forward compatibility.

 

Kube-proxy Boosts Ingress Connection Health Management

SIG: sig-network

Status: Alpha

Feature: KubeProxyDrainingTerminatingNodes (Disabled by default)

Kube-proxy now ensures connections terminate gracefully when a node is ending. There's also a new /livez path for more precise health checks.

 

Enhanced NodePort Ranges Allocation

SIG: sig-network

Status: Beta

Feature: ServiceNodePortStaticSubrange (Enabled by default)

This update allows better control over NodePort service, reserving the initial ports in service-node-port-range for static allocation.

 

Load Balancer Node Management Improved

SIG: sig-network

Status: Beta

Feature: StableLoadBalancerNodeSet (Enabled by default)

The way nodes are removed from the load balancer's node list is being refined. This minimizes undesirable chain reactions like immediate connection terminations and load balancer resyncs.

 

EndpointSlice Reconciler Relocated

SIG: sig-network

Status: Stable

The EndpointSlice reconciler logic is being exposed and moved to a new module for use in custom Endpoint controllers.

 

Optimized iptables-restore Functioning

SIG: sig-network

Status: Stable

Feature: MinimizeIPTablesRestore (Enabled by default)

Kube-proxy will see improved performance in iptables mode by modifying how iptables-restore functions, focusing only on changed rules.

 

Cleanup of IPTables Chain Ownership

SIG: sig-network

Status: Stable

Feature: IPTablesOwnershipCleanup (Enabled by default)

As a cleanup measure, unused iptables chains, previously created by components like kubelet and kube-proxy, are being removed.

 

DNS Configuration Broadened

SIG: sig-network

Status: Stable

Feature: ExpandedDNSConfig (Enabled by default)

The DNS configuration is now more flexible, accommodating more search paths and longer lists.

 

Enhanced Traffic Direction for Terminating Endpoints

SIG: sig-network

Status: Stable

Feature: ProxyTerminatingEndpoints (Enabled by default)

Kube-proxy will better handle traffic to endpoints that are ending. This is especially useful during rolling updates when endpoints might be in terminating states.

For a comprehensive understanding, visit the Kubernetes.io article titled 'Advancements in Kubernetes Traffic Engineering'.

 

Condition for Sandbox Initiation (KEP-3085)

SIG: sig-node

Status: Introduction to Alpha

Feature: PodReadyToInitiateContainersCondition (Default: false)

Kubernetes 1.28 has rolled out a new pod status called PodReadyToInitiateContainersCondition. This status illuminates when a Pod is initiating its sandbox and configuring its networking, assisting cluster admins and Kubernetes controllers.

 

Determining cgroup Driver via CRI

SIG: sig-node

Status: Introduction to Alpha

Feature: KubeletCgroupDriverDetectionFromCRI (Default: false)

The Kubernetes 1.28 update lets the kubelet intuitively determine the cgroup driver through the container runtime. This aims to mitigate diagnostic complications from configuration discrepancies.

 

Modular kubelet Config Directory

SIG: sig-node

Status: Introduction to Alpha

Feature: ModularIndexedJob (Default: false)

With the 1.28 release, Kubernetes facilitates the configuration of a kubelet directory. Using the new --config-dir argument, users can now input multiple .conf configuration files, streamlining their setup processes.

 

Sidecar Container Management

SIG: sig-node

Status: Introduction to Alpha

Feature: ManagedSidecarContainers (Default: false)

The new update advances the management of sidecar containers. Kubernetes now provides an improved method to identify and manage a sidecar container's lifecycle in correlation with primary containers.

 

CDI Device Inclusion in Device Plugin API

SIG: sig-node

Status: Introduction to Alpha

Feature: DevicePluginCDIDevicesIntegration (Default: false)

Kubernetes 1.28 unveils an enhancement allowing device plugin developers to seamlessly convey requests to the container runtimes, elevating device integration capabilities.

 

Evolving Dynamic Resource Allocation

SIG: sig-node

Status: Progression to Alpha

Feature: ProgressiveDynamicResourceAssignment (Default: false)

This feature, continuing its journey since its introduction in Kubernetes 1.26, enables pods to request specific resources beyond conventional CPU and memory needs, catering especially to environments requiring specialized resources.

 

User Namespace Integration in Pods

SIG: sig-node

Status: Progression to Alpha

Feature: StatelessUserNamespaceIntegrationInPods (Default: false)

The integration of user namespaces with Kubernetes 1.28 augments security by allowing pod processes to run with varied user IDs than those on the host, primarily available for Linux systems.

 

Enhanced Multi-Numa Alignment in Topology Manager

SIG: sig-node

Status: Progression to Beta

Feature: TopologyManagerAdvancedPolicyOptions (Default: true)

Kubernetes now provides a heightened awareness of system topology, focusing on allocating tasks to proximate CPU cores for heightened efficiency.

 

Memory Swap Support for Nodes

SIG: sig-node

Status: Progression to Beta

Feature: NodeMemorySwap (Default: true)

Supporting swap on Linux nodes, this feature in Kubernetes 1.28 is poised to prevent out-of-memory terminations, though with a cautionary note on potential latency.

 

Summary:

Kubernetes 1.28 marks another milestone in the pursuit of impeccable container orchestration. From significant improvements in kubectl commands and metrics stability to advanced networking features like refined ingress connection health management, this update is brimming with features that cater to both the experienced and the newcomers.

Enhanced node management, optimization of iptables, and features focused on user security demonstrate Kubernetes' commitment to addressing community feedback and ever-evolving challenges.

As we look forward to what the future holds, Kubernetes 1.28 undoubtedly sets a high bar for future iterations, pushing the boundaries of what's possible in the realm of container orchestration.