How to Install a Kubernetes Cluster on Ubuntu with a Custom Pod Network CIDR

Deploying a Kubernetes cluster requires careful configuration, especially when customizing network settings like the Pod Network CIDR. In this guide, we’ll walk through setting up a Kubernetes cluster on Ubuntu, addressing common pitfalls (like preflight errors) and configuring a custom Pod CIDR for your network plugin.

---

Prerequisites

1. Ubuntu Machines:

• Control Plane Node: 2+ CPUs, 2+ GB RAM.
• Worker Node(s): 1+ CPU, 1+ GB RAM.
• All nodes must have unique hostnames and MAC addresses.

1. System Configuration:

• Disable Swap:
  bash sudo swapoff -a sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
• Update Packages:
  bash sudo apt update && sudo apt upgrade -y

---

Step 1: Configure Kernel Modules and Networking

Kubernetes requires kernel modules and IP forwarding to manage container networking.

# Load the bridge-netfilter module
sudo modprobe br_netfilter

# Make it persistent
echo 'br_netfilter' | sudo tee /etc/modules-load.d/br_netfilter.conf

# Enable IP forwarding and bridge settings
echo 'net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1' | sudo tee /etc/sysctl.d/99-kubernetes.conf

# Apply changes
sudo sysctl --system

Verify the settings:

sudo sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables

Output should show:

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1

---

Step 2: Install Container Runtime (containerd)

Kubernetes needs a container runtime. We’ll use containerd:

# Install dependencies
sudo apt install -y curl gnupg2 software-properties-common

# Add Docker repository (for containerd)
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install containerd
sudo apt update
sudo apt install -y containerd.io

# Configure containerd to use systemd cgroups
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

---

Step 3: Install Kubernetes Tools

Add the official Kubernetes repository and install kubeadm, kubelet, and kubectl:

# Add Kubernetes repo
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Install tools
sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl  # Prevent auto-updates

---

Step 4: Initialize the Control Plane

Specify your Pod Network CIDR during initialization. For example, to use 10.244.0.0/16 (Flannel’s default):

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Save the kubeadm join command for worker nodes (it will look like):

kubeadm join <IP>:6443 --token <token> --discovery-token-ca-cert-hash <hash>

---

Step 5: Configure kubectl

Set up kubectl for your user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

---

Step 6: Install a CNI Plugin

Choose a network plugin that matches your Pod CIDR:

Option 1: Flannel (for 10.244.0.0/16)

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Option 2: Calico (for custom CIDR)

1. Edit the Calico manifest to match your CIDR:

   curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml -O
   sed -i 's/192.168.0.0/10.244.0.0/g' calico.yaml  # Replace with your CIDR
   kubectl apply -f calico.yaml

---

Step 7: Join Worker Nodes

On each worker node, run the kubeadm join command saved earlier.

---

Step 8: Verify the Cluster

kubectl get nodes

After a few minutes, nodes should show Ready status.

---

Troubleshooting

1. Pods Stuck in Pending:

• Ensure the CNI plugin is installed.
• Check firewall rules for Pod CIDR traffic.

1. Control Plane Not Ready:

• Verify kubelet is running: systemctl status kubelet.
• Check logs: journalctl -u kubelet -f.

1. Version Mismatch:

• Use the same Kubernetes version on all nodes.

---

Conclusion

You’ve now set up a Kubernetes cluster with a custom Pod Network CIDR! By addressing kernel settings, choosing the right CNI plugin, and configuring kubeadm properly, you can tailor the network to your needs. For production, consider additional steps like hardening security or adding monitoring tools like Prometheus.

Further Reading:

• Kubernetes Official Documentation
• Flannel Networking
• Calico Custom CIDR Guide