How to create L3 networking for VMs

This guide helps to create L3 networking for VMs
Tags:
Categories:

  7 minute read  

Setting up L3 networking with Multus for KubeVirt VMs enables efficient cross-node communication and network segmentation, enhancing flexibility and performance in a Kubernetes environment.

Prerequisites

  • A running Kubernetes cluster
  • kubectl command-line tool configured to connect to your cluster
  • Calico and Multus installed and configured
  • KOSI is installed and successfully authenticated

Configuring Node Networking using Network Manager State (NMState)

Network Manager State (nmstate) is a declarative network configuration tool for Linux. It provides an API and CLI to manage network settings in a structured, state-driven manner.

In Kubernetes, nmstate is often used with NodeNetworkConfigurationPolicy (NNCP) to configure node networking dynamically, allowing for changes such as setting up VLANs, bridges, bonds, and other networking features.

1. Deploying nmstate on a Kubernetes Cluster

First, create the nmstate-values.yaml file with the following content:

# nmstate-values.yaml
nmstateValues:
  namespace: nmstate

Use this values file to deploy the nmstate operator along with its CRDs:

kosi install --hub kubeops kubeops/nmstate:2.0.1 -f nmstate-values.yaml

2. Configuring Node Networking with NNCP

Create an nncp-values.yaml file to define network configurations dynamically, including bridges and routes:

# nncp-values.yaml
namespace: "nmstate"
nodes:
  - nodeName: cluster1master1
    nodeIP: 10.2.10.11
    bridgeIP: 10.10.1.1 # bridge IP for cluster1master1.
    bridgeSubnet: 10.10.1.0 # `/24` subnet for bridge
    nodeInterface: ens192 # use `ip a` or `ifconfig` to identify interface

  - nodeName: cluster1master2
    nodeIP: 10.2.10.12
    bridgeIP: 10.10.2.1
    bridgeSubnet: 10.10.2.0
    nodeInterface: ens192

  - nodeName: cluster1worker1
    nodeIP: 10.2.10.14
    bridgeIP: 10.10.3.1
    bridgeSubnet: 10.10.3.0
    nodeInterface: ens192

  - nodeName: cluster1worker2
    nodeIP: 10.2.10.15
    bridgeIP: 10.10.4.1
    bridgeSubnet: 10.10.4.0
    nodeInterface: ens192

Note:

If you’re adding a new node, you should assign a unique bridgeIP and bridgeSubnet, following the existing pattern.

For example, if your last node used:

bridgeIP: 10.10.4.1

bridgeSubnet: 10.10.4.0/24

  - nodeName: <new_node>
    nodeIP: <node_ip>
    bridgeIP: 10.10.5.1
    bridgeSubnet: 10.10.5.0
    nodeInterface: ens192

Use this values file to deploy the node network configuration policies on all nodes:

kosi install --hub kubeops kubeops/nncp:2.0.1 -f nncp-values.yaml

3. Allow Traffic betwwen the Interfaces

iptables -A FORWARD -i br-secondary -j ACCEPT
iptables -A FORWARD -o br-secondary -j ACCEPT

4. Check for Bridge Network and Evaluate the connection between the nodes via the BRidge network

  • Verify Bridge Creation on Each Node

Run the following command on each node to confirm that the bridge exists and has the correct IP:

ip a show br-secondary

  • Check Bridge Interface and Attached Interfaces

List all bridges and verify attached interfaces:

ip link show type bridge

  • Verify Routes and Connectivity

Check if the expected routes are in place:

ip route show

Then, test connectivity between nodes using ping:

ping -c 4 <another_node_bridgeIP>

Install KubeVirt

For installation instructions, please refer to the KubeVirt Installation Guide

Networks and Interfaces

L3 Networking with Multus

To set up L3 networking with Multus, each worker node requires a separate NetworkAttachmentDefinition (NAD) with a unique IP address and subnet matching the node’s network configuration. This allows isolated network environments for each pod, enabling multi-interface support on pods and communication between nodes in the cluster.

Network Planning

Before creating the NADs, plan the IP ranges and subnets for each worker node to avoid conflicts and ensure proper routing between nodes.

Example IP range mapping for worker nodes:

Node Subnet Example IP for Pod
cluster1worker1 10.10.3.0/24 10.10.3.10
cluster1worker2 10.10.4.0/24 10.10.4.10

Example NAD Configuration for Worker Node 1

This example shows how to configure a NetworkAttachmentDefinition (NAD) for worker node 1 (cluster1worker1), with a static IP range assigned to its pods:

# nad_cluster1worker1.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: secondary-network-node1 # Assign unique names to each node's NAD
  annotations:
    k8s.v1.cni.cncf.io/nodeSelector: '{"kubernetes.io/hostname": "cluster1worker1"}'
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "secondary-network",
      "type": "bridge",
      "bridge": "br-secondary",
      "ipam": {
        "type": "static",
        "addresses": [
          {
            "address": "10.10.3.10/24", # use correct worker node's subnet and mention the appropriate IP for pod's network
            "gateway": "10.10.3.1"
          }
        ],
        "routes": [
          {
            "dst": "10.10.4.0/24",
            "gw": "10.10.3.1"
          }
        ]
      }
    }

Apply the K8s Manifest Definition files

To deploy the NetworkAttachmentDefinition (NAD) objects for your nodes, apply the following Kubernetes manifest files:

kubectl apply -f nad_cluster1worker1.yaml
kubectl apply -f nad_cluster1worker2.yaml

Verify the created Network Attachment Definition Objects

To ensure that the NetworkAttachmentDefinition (NAD) objects have been successfully created, run the following command:

kubectl get net-attach-def -A

Deploy Virtual Machines (VMs) and Attach the Secondary Networks

Once the NADs are created, you can deploy Virtual Machines (VMs) and attach them to the secondary networks. The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached.

Network Selection from NADs

Each VM must be connected to the correct network defined by the NetworkAttachmentDefinition (NAD) for its corresponding node. For example, if you have created a NAD for cluster1worker1 named secondary-network-node1, ensure that the VM manifest refers to this network in the annotations section and correctly links to the networkName under the multus configuration.

Ensure that the network name in the manifest matches the NAD definition.

Create Image Pull Secret

Please create a secret for pulling image from registry.kubeops.net.

kubectl -n "l3-networking" create secret docker-registry kubeops-vm \
  --docker-server=registry.kubeops.net \
  --docker-username=<kubeops-user> \
  --docker-password=<harbor_token>

Example VM Manifest for Secondary Network

The following is an example Kubernetes manifest for deploying a VM with a secondary network interface attached to a specific NAD:

# vm1.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: "ubuntu-vm"
  namespace: "l3-networking"
  annotations:
    k8s.v1.cni.cncf.io/networks: secondary-network-node1 # Ensure this matches the NAD network name
spec:
  nodeSelector:
    kubernetes.io/hostname: cluster1worker1 # edit for scheduling node on other node
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/size: small
    spec:
      domain:
        cpu:
          cores: 1
        devices:
          disks:
            - name: containervolume
              disk:
                bus: virtio
            - name: cloudinitvolume
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: secondary-network
            bridge: {}
        resources:
          requests:
            memory: 2048M
      networks:
      - name: default
        pod: {}
      - name: secondary-network
        multus:
          networkName: secondary-network-node1 # Ensure this matches the NAD network name
      volumes:
        - name: containervolume
          containerDisk:
            image: registry.kubeops.net/kubeops-vm/kubeops/ubuntu-cloudimg-focal:2.0.1
            imagePullSecret: kubeops-vm
            imagePullPolicy: Always
        - name: cloudinitvolume
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              chpasswd:
                list: |
                  ubuntu:ubuntu
                  root:toor
                expire: False
              ssh_pwauth: True
              disable_root: false

Apply the VM Manifest

Once the NetworkAttachmentDefinition (NAD) objects have been created, you can deploy the Virtual Machines (VMs) and apply the manifests. The following commands apply the VM manifests to your Kubernetes cluster.

kubectl apply -f vm1.yaml
kubectl apply -f vm2.yaml

Verify the Networking

Upon successful VM startup, verify network connectivity using the following procedure:

On each VMs

1. Check the IP Address

First, check the IP address assigned to the pod (VM). This is crucial to ensure that the secondary network interface has been correctly assigned an IP.

kubectl exec -n <namespace> <pod-1> -- ip a

2. Assign IP Address if Not Automatically Assigned (Optional)

kubectl exec -n <namespace> <pod-1> -- sudo dhclient enp2s0

3. Check the Routes

Verify the routing configuration on the VM to ensure that traffic can flow through the secondary network. Use the following command:

kubectl exec -n <namespace> <pod-1> -- ip r

This will display the routing table for the VM and confirm if the secondary network is correctly set up.

Verify VM Communication

After ensuring the network settings are correct, test the communication between VMs by pinging the secondary network IP of another pod.

Test Connectivity Between VMs

Use the ping command to check if the VMs can communicate over the secondary network. Replace <pod2_secondary_network_IP> with the actual IP address of the secondary network interface on the second VM.

kubectl exec -n <namespace> <pod-1> -- ping -c 2 <pod2_secondary_network_IP>

If the ping is successful, it confirms that the VMs are able to communicate over the secondary network.

Additional Notes:

  • Ensure that the correct namespace and pod names are used when executing commands.

  • Make sure the secondary network interface is correctly configured on each VM, and the IP address and routes are properly assigned.

  • If the VMs cannot communicate, check for issues in the NetworkAttachmentDefinition (NAD) configuration or verify that the correct CNI plugins are running.