If you’ve ever managed Kubernetes clusters on bare-metal or on-premise infrastructure, you know the pain: installing, upgrading, securing, and scaling clusters is a lot of manual work. Two tools from the Kubernetes ecosystem tackle these challenges head-on — RKE2 and Cluster API (CAPI).

In this post, we’ll get our hands dirty with both. But first, let’s understand why they exist and what problems they solve. RKE2 (also known as RKE Government) is a Kubernetes distribution developed by Rancher (now part of SUSE).

Here’s how simple the installation looks — A single script installs the rke2-server (or rke2-agent) systemd service, the binary, and all required container images. Compare that to the dozens of steps needed for a traditional kubeadm setup, the difference is night and day.

# Download and run the installer — that's it!
curl -sfL https://get.rke2.io --output install.sh
chmod +x install.sh
INSTALL_RKE2_CHANNEL=v1.33 ./install.sh

# Verify
rke2 --version
# rke2 version v1.33.7+rke2r3 (...)

RKE2 Architecture at a Glance

RKE2 follows a familiar server + agent model.

📖 For the official architecture diagram, see docs.rke2.io/architecture.

Boot sequence:

1. The agent (kubelet) starts up
2. Static pod manifests are written to disk
3. Control-plane pods (etcd, apiserver, etc.) are launched by kubelet

┌──────────────────────────────────────────────┐
│              RKE2 Server Node                │
│           (= Control Plane Node)             │
│                                              │
│  ┌─────────────────────────────────────────┐ │
│  │         RKE2 Supervisor Process         │ │
│  │  ┌──────────┐  ┌────────────────────┐   │ │
│  │  │ kubelet  │  │    containerd      │   │ │
│  │  └────┬─────┘  └────────────────────┘   │ │
│  │       │                                 │ │
│  │       ▼  watches pod-manifests/         │ │
│  │  ┌──────────────────────────────────┐   │ │
│  │  │  Static Pods:                    │   │ │
│  │  │  • etcd                          │   │ │
│  │  │  • kube-apiserver                │   │ │
│  │  │  • kube-controller-manager       │   │ │
│  │  │  • kube-scheduler                │   │ │
│  │  └──────────────────────────────────┘   │ │
│  └─────────────────────────────────────────┘ │
└──────────────────────────────────────────────┘

┌──────────────────────────────────────────────┐
│              RKE2 Agent Node                 │
│            (= Worker Node)                   │
│                                              │
│  ┌─────────────────────────────────────────┐ │
│  │         RKE2 Agent Process              │ │
│  │  ┌──────────┐  ┌────────────────────┐   │ │
│  │  │ kubelet  │  │    containerd      │   │ │
│  │  └──────────┘  └────────────────────┘   │ │
│  └─────────────────────────────────────────┘ │
└──────────────────────────────────────────────┘

What is Cluster API?

Now imagine you don’t just have one cluster to manage — you have tens or hundreds. Installing each one manually (even with RKE2) becomes a bottleneck. This is where Cluster API (CAPI) comes in.

Cluster API is a Kubernetes sub-project that lets you manage Kubernetes clusters themselves as Kubernetes resources. In other words: you use Kubernetes to create, configure, and upgrade Kubernetes clusters.

It introduces custom resources like below.

# These are just regular Kubernetes objects!
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster          # Represents an entire K8s cluster

apiVersion: cluster.x-k8s.io/v1beta2
kind: Machine          # Represents a single node (VM or bare-metal)

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment  # Like a Deployment, but for Machines

The key concept is the separation between two types of clusters.

• Management Cluster: The cluster where Cluster API is deployed and its controllers run. Think of it as the “control tower.”
• Workload Cluster: A cluster that was provisioned through Cluster API. This is where your applications actually run.

┌─────────────────────────────────┐      ┌─────────────────────────────────┐
│      Management Cluster         │      │       Workload Cluster(s)       │
│                                 │      │                                 │
│  • Runs Cluster API controllers │─────▶│  • Provisioned BY Cluster API   │
│  • Stores cluster definitions   │      │  • Runs your actual workloads   │
│  • The "brain" of operations    │      │  • Can be upgraded declaratively│
│                                 │      │                                 │
└─────────────────────────────────┘      └─────────────────────────────────┘

So, here is the bottom line.

• RKE2 solves the problem of installing and running a single, secure Kubernetes cluster with minimal effort.
• Cluster API solves the problem of managing the lifecycle of many clusters at scale — creation, upgrades, scaling — all declaratively.

In this hands-on guide, we’ll walk through:

• RKE2: Install a server + agent cluster from scratch, deploy a sample app, manage certificates, and perform both manual and automated upgrades.
• Cluster API: Set up a management cluster, provision a full workload cluster (3 control-plane + 3 worker nodes), deploy apps, upgrade the cluster version with a single kubectl patch, and tear it all down.

RKE2 Overview — Architecture, Security-First Design, and How It Works

RKE2 is a Kubernetes distribution developed by Rancher (now part of SUSE). If you have ever worked with K3s, you will immediately notice a familiar philosophy here — keep things simple, ship a single binary, and get out of the operator’s way.

But RKE2 takes a slightly different path. While K3s was designed to be ultra-lightweight and perfect for edge and IoT scenarios, RKE2 was built with security and compliance as the top priority from day one. The name itself sometimes appears as “RKE Government” in older documentation, which gives you a hint about its original target audience — environments where passing security audits is not optional.

So what exactly makes RKE2 special? There are three pillars that define it.

The first pillar is security by default. Out of the box, with zero additional configuration, RKE2 is designed to pass the CIS (Center for Internet Security) Kubernetes Benchmark. If you have ever gone through a CIS hardening exercise on a vanilla kubeadm cluster, you know how painful that can be — dozens of flags to set, file permissions to tighten, audit policies to write. RKE2 handles most of this automatically. It ships with sane defaults for etcd encryption, Pod Security Standards, audit logging, and network policies. You still have the freedom to customize everything, but the starting point is already hardened.

The second pillar is operational simplicity. Just like K3s, RKE2 is distributed as a single binary. You do not need to separately install kubelet, kube-proxy, etcd, or any other control plane component by hand. A single install script pulls everything you need, registers a systemd service, and you are ready to go. Upgrades follow the same pattern — run the installer again with a newer channel, restart the service, and the cluster rolls forward. This dramatically reduces the surface area for human error during both initial deployment and ongoing maintenance.

The third pillar is a container-first architecture. RKE2 minimizes its dependency on the host operating system. Instead of running control plane components as bare processes directly on the host (the way kubeadm does by default), RKE2 runs everything on top of containerd. The kubelet, which is managed by the RKE2 supervisor process, watches a static pod manifest directory and launches etcd, kube-apiserver, kube-controller-manager, and kube-scheduler as static pods. This means the host OS only needs to provide a Linux kernel and containerd — everything else lives inside containers.

Let us talk about the architecture in more detail. If you look at the official architecture diagram at https://docs.rke2.io/architecture, you will see that an RKE2 cluster is composed of two types of nodes: RKE2 Server Nodes and RKE2 Agent Nodes.

A Server Node is what you would normally call a control plane node — it runs etcd, the API server, the controller manager, and the scheduler, in addition to a kubelet and kube-proxy. An Agent Node is simply a worker node — it runs a kubelet and kube-proxy, and its sole job is to run your application workloads.

The interesting part is what happens inside each node. There is a single RKE2 process (called the RKE2 Supervisor) that orchestrates everything. On a server node, this supervisor process manages containerd and the kubelet internally. It writes static pod manifests into a well-known directory, and the kubelet picks them up and starts the corresponding pods. So the boot sequence looks like this — the RKE2 supervisor starts, it brings up the internal kubelet (which also starts containerd), the kubelet detects the static pod manifests that the supervisor has written, and then the control plane pods — etcd, kube-apiserver, kube-controller-manager, kube-scheduler — come to life one by one. On an agent node the process is simpler — the supervisor starts, brings up kubelet and kube-proxy, and the node registers itself with the API server running on the server nodes.

This design has a subtle but important consequence. Because the control plane components run as static pods managed by the kubelet, you can inspect them with standard Kubernetes tooling. You can run kubectl get pods -n kube-system and see etcd-k8s-node1, kube-apiserver-k8s-node1, and so on, just like you would on a kubeadm cluster. But unlike kubeadm, you did not have to manually bootstrap any of this — the RKE2 supervisor took care of it.

Now let us see what this looks like in practice. When you install RKE2 on a fresh server, the install script is remarkably straightforward. You download it, make it executable, and run it with a channel flag to pin your desired Kubernetes minor version.

curl -sfL https://get.rke2.io --output install.sh
chmod +x install.sh
INSTALL_RKE2_CHANNEL=v1.33 ./install.sh

After the script finishes, you will see output confirming that several RPM packages have been installed (on RHEL/Rocky-based systems). For example, on a Rocky Linux 9 machine you might see something like below.

Installed:
  rke2-common-1.33.7~rke2r3-0.el9.aarch64
  rke2-selinux-0.22-1.el9.noarch
  rke2-server-1.33.7~rke2r3-0.el9.aarch64

You can verify the installed version immediately:

rke2 --version

rke2 version v1.33.7+rke2r3 (7e4fd1a82edf497cab91c220144619bbad659cf4)
go version go1.24.11 X:boringcrypto

Notice the “boringcrypto” tag in the Go version. This is not accidental. RKE2 is compiled with BoringCrypto, Google’s FIPS 140–2 validated cryptographic module. This means that all TLS operations within the RKE2 binary use FIPS-compliant cryptography — another nod to the security-first philosophy.

The install script also sets up YUM/DNF repositories so that future updates can be pulled in cleanly. You can confirm this by checking the repo list:

dnf repolist

rancher-rke2-1.33-stable       Rancher RKE2 1.33 (v1.33)
rancher-rke2-common-stable     Rancher RKE2 Common (v1.33)

And if you peek inside the repo configuration file, you will see that it points to Rancher’s official RPM mirror:

# /etc/yum.repos.d/rancher-rke2.repo

[rancher-rke2-common-stable]
name=Rancher RKE2 Common (v1.33)
baseurl=https://rpm.rancher.io/rke2/stable/common/centos/9/noarch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key

[rancher-rke2-1.33-stable]
name=Rancher RKE2 1.33 (v1.33)
baseurl=https://rpm.rancher.io/rke2/stable/1.33/centos/9/aarch64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key

Both GPG signature checking and repository-level GPG checking are enabled by default. This ensures that the packages you pull have not been tampered with in transit — yet another security detail that RKE2 gets right without you having to think about it.

At this point, RKE2 is installed but not yet running. The binary supports two primary subcommands, which directly correspond to the two node roles we discussed earlier:

rke2 --help

server    Run management server
   agent     Run node agent

The “server” subcommand starts the full control plane (API server, etcd, scheduler, controller manager) along with a kubelet and kube-proxy. The “agent” subcommand starts only a kubelet and kube-proxy, which is all a worker node needs.

Before starting the server, you typically write a configuration file at /etc/rancher/rke2/config.yaml. This is where you customize the behavior of RKE2 — things like which CNI plugin to use, which IP addresses to bind to, and which built-in add-ons to disable. Here is an example from the hands-on lab:

# /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"
debug: true
cni: canal
bind-address: 192.168.10.11
advertise-address: 192.168.10.11
node-ip: 192.168.10.11
disable-cloud-controller: true
disable:
  - servicelb
  - rke2-coredns-autoscaler
  - rke2-ingress-nginx
  - rke2-snapshot-controller
  - rke2-snapshot-controller-crd
  - rke2-snapshot-validation-webhook

A few things worth calling out here. The “write-kubeconfig-mode” setting controls the file permissions of the generated kubeconfig file at /etc/rancher/rke2/rke2.yaml. Setting it to 0644 makes it world-readable, which is convenient in a lab but something you would lock down in production.

The “cni” field lets you choose between canal (the default, which combines Calico for network policy and Flannel for overlay networking), cilium, calico, or none. The “disable” list is particularly useful — RKE2 ships with several built-in Helm charts (ingress-nginx, metrics-server, CoreDNS autoscaler, etc.), and you can selectively disable any of them if you plan to bring your own.

RKE2 also supports per-chart customization through a mechanism called HelmChartConfig. If you want to override the default values of a built-in Helm chart, you simply drop a HelmChartConfig manifest into /var/lib/rancher/rke2/server/manifests/ before starting the server. For example, to tell the Canal CNI to use a specific network interface:

# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      iface: "enp0s9"

This is a very elegant pattern. Instead of requiring you to manually run helm install with custom values after the cluster is up, RKE2 lets you declare your desired configuration before the cluster even boots. The RKE2 Helm controller will pick up these manifests and apply them automatically during the initial startup sequence.

Once everything is configured, starting the server is a single systemctl command:

systemctl enable --now rke2-server.service

This typically takes about two minutes. During that time, the RKE2 supervisor starts containerd, generates TLS certificates for all control plane components, writes static pod manifests, and waits for the kubelet to bring up etcd and the API server. You can watch the progress in real time with:

journalctl -u rke2-server -f

After the server is up, you will find all the necessary binaries tucked inside /var/lib/rancher/rke2/bin/:

tree /var/lib/rancher/rke2/bin/
├── containerd
├── containerd-shim-runc-v2
├── crictl
├── ctr
├── kubectl
├── kubelet
└── runc

These are not installed into your system PATH by default. RKE2 keeps them isolated to avoid conflicts with any existing binaries on the host. If you want to use kubectl and other tools directly, the recommended approach is to create symbolic links:

ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
ln -s /var/lib/rancher/rke2/bin/crictl /usr/local/bin/crictl
ln -s /var/lib/rancher/rke2/bin/containerd /usr/local/bin/containerd
ln -s /var/lib/rancher/rke2/bin/runc /usr/local/bin/runc
ln -s /var/lib/rancher/rke2/bin/ctr /usr/local/bin/ctr
ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml

Then you copy the generated kubeconfig to your home directory:

mkdir ~/.kube
cp /etc/rancher/rke2/rke2.yaml ~/.kube/config

And you are ready to interact with your cluster:

kubectl cluster-info

Kubernetes control plane is running at https://192.168.10.11:6443

kubectl get node -owide

NAME        STATUS   ROLES                AGE   VERSION          INTERNAL-IP     OS-IMAGE                      CONTAINER-RUNTIME
k8s-node1   Ready    control-plane,etcd   15m   v1.34.3+rke2r3   192.168.10.11   Rocky Linux 9.6 (Blue Onyx)   containerd://2.1.5-k3s1

If you check the running pods, you will see the control plane components we discussed — all running as pods in the kube-system namespace, exactly as the architecture diagram predicts:

kubectl get pod -A

NAMESPACE     NAME                                         READY   STATUS      AGE
kube-system   etcd-k8s-node1                               1/1     Running     2m
kube-system   kube-apiserver-k8s-node1                     1/1     Running     2m
kube-system   kube-controller-manager-k8s-node1            1/1     Running     2m
kube-system   kube-scheduler-k8s-node1                     1/1     Running     2m
kube-system   kube-proxy-k8s-node1                         1/1     Running     2m
kube-system   rke2-canal-dkw2n                             2/2     Running     2m
kube-system   rke2-coredns-rke2-coredns-784bcb7f4d-tpt2d   1/1     Running     2m
kube-system   rke2-metrics-server-7b59bd8854-m5w2c         1/1     Running     1m

You will also notice that RKE2 deployed several add-ons as Helm releases — Canal for networking, CoreDNS for DNS resolution, metrics-server for resource metrics, and a set of runtime classes:

helm list -A

NAME                NAMESPACE    STATUS    CHART                              APP VERSION
rke2-canal          kube-system  deployed  rke2-canal-v3.31.3-build2026011900 v3.31.3
rke2-coredns        kube-system  deployed  rke2-coredns-1.45.008              1.13.1
rke2-metrics-server kube-system  deployed  rke2-metrics-server-3.13.006       0.8.0
rke2-runtimeclasses kube-system  deployed  rke2-runtimeclasses-0.1.000        0.1.0

This Helm-based add-on management is one of the things that makes RKE2 so pleasant to operate. Every built-in component is a standard Helm chart, which means you can inspect its values, override them with HelmChartConfig, or disable them entirely and replace them with your own preferred solution. There is no magic, no proprietary packaging — just Helm charts running on your cluster.

To summarize the architecture: RKE2 gives you a production-grade, CIS-hardened Kubernetes cluster through a single binary and a single configuration file. The RKE2 supervisor process manages containerd and kubelet internally, launches control plane components as static pods, and deploys networking and DNS through Helm charts. The result is a cluster that is easy to install, easy to understand, and secure by default — without sacrificing any of the flexibility that Kubernetes operators expect.

RKE2 Hands-On

Before we get into the actual installation, let’s talk about what we’re building. Our lab environment consists of two nodes: k8s-node1 (which will serve as the control plane) and k8s-node2 (which will be our worker node). Both are running Rocky Linux 9.6, and we’re provisioning them with Vagrant. If you want to follow along, you can grab the Vagrantfile and init script from the repository and spin up the environment like this:

mkdir k8s-rke2
cd k8s-rke2

curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-rke2/Vagrantfile
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-rke2/init_cfg.sh

vagrant up
vagrant status

Once both VMs are up and running, we’re ready to begin.

Installing the Server (Control Plane)

Let’s SSH into our first node, which will become the RKE2 server, also known as the control plane node.

vagrant ssh k8s-node1

RKE2 provides a convenient installation script that sets everything up as a systemd service. We’re going to download this script, make it executable, and run it while specifying the v1.33 release channel. This tells the installer to pull the latest stable version from the 1.33 line.

curl -sfL https://get.rke2.io --output install.sh
chmod +x install.sh
INSTALL_RKE2_CHANNEL=v1.33 ./install.sh

When the installation completes, you’ll see output confirming that the rke2-common, rke2-selinux, and rke2-server packages have been installed. At this point, nothing is running yet. The installer simply placed the binary and the systemd unit files on your system.

You can verify the installed version by running the following command. You should see something like v1.33.7+rke2r3 along with the Go version used to compile it.

rke2 --version

The installer also added a couple of Rancher RPM repositories to your system. You can confirm this by checking the repo list and the repo configuration file. This is how future upgrades will be pulled.

dnf repolist
cat /etc/yum.repos.d/rancher-rke2.repo

Now comes the important part: configuring RKE2 before we start the service. RKE2 reads its configuration from /etc/rancher/rke2/config.yaml.

We’re going to create this file and set a few key options. We’ll make the kubeconfig file world-readable for convenience, enable debug logging to get more visibility, choose Canal as our CNI plugin, bind the API server and advertise address to our node’s static IP, disable the cloud controller since we’re on-prem, and also disable a handful of built-in add-ons that we either don’t need or want to manage ourselves (the built-in service load balancer, CoreDNS autoscaler, ingress-nginx, and the snapshot-related components).

cat << EOF > /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"

debug: true

cni: canal

bind-address: 192.168.10.11
advertise-address: 192.168.10.11
node-ip: 192.168.10.11

disable-cloud-controller: true

disable:
  - servicelb
  - rke2-coredns-autoscaler
  - rke2-ingress-nginx
  - rke2-snapshot-controller
  - rke2-snapshot-controller-crd
  - rke2-snapshot-validation-webhook
EOF

Since we’re using Canal as the CNI, and our Vagrant VMs have multiple network interfaces, we need to tell Flannel (which is the networking backend inside Canal) which interface to use. RKE2 manages its built-in add-ons through Helm charts, and we can customize them by placing HelmChartConfig manifests in the server manifests directory.

Let’s create that directory first and then drop in our Canal configuration.

mkdir -p /var/lib/rancher/rke2/server/manifests/

cat << EOF > /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    flannel:
      iface: "enp0s9"
EOF

We’ll also create a config for CoreDNS to explicitly disable its autoscaler, since we already disabled the autoscaler add-on in the main config and we want to make sure both sides agree.

cat << EOF > /var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    autoscaler:
      enabled: false
EOF

Before starting the service, it’s a great idea to open a second terminal and set up some monitoring so you can watch the boot process in real time. In that second terminal, run the following two commands. The first watches the process tree so you can see RKE2’s child processes spawn, and the second tails the journal for the rke2-server unit.

watch -d pstree -a
journalctl -u rke2-server -f

Now, back in your main terminal, let’s enable and start the RKE2 server service. This single command both enables the service to start on boot and starts it immediately. The initial startup takes roughly two minutes as it downloads container images and bootstraps the control plane. After the core components are up, you’ll need to wait another minute or two for CoreDNS pods to become healthy.

systemctl enable --now rke2-server.service
systemctl status rke2-server --no-pager

If you flip over to your monitoring terminal, you’ll see the process tree come alive. The rke2 process starts containerd and kubelet, and then kubelet picks up the static pod manifests and launches the etcd, kube-apiserver, kube-controller-manager, and kube-scheduler pods.

This is one of the elegant things about RKE2: the entire control plane boots up through the same static-pod mechanism that kubeadm uses, but it’s all orchestrated by a single RKE2 binary.

Once the service is running, we need to set up our kubeconfig so we can use kubectl. RKE2 writes its kubeconfig to /etc/rancher/rke2/rke2.yaml. Let’s copy it to the standard location.

mkdir ~/.kube
cp /etc/rancher/rke2/rke2.yaml ~/.kube/config

RKE2 installs its own set of binaries under /var/lib/rancher/rke2/bin/, including kubectl, crictl, containerd, ctr, and runc. Rather than modifying the PATH, a clean approach is to create symbolic links in /usr/local/bin/ so that these tools are available system-wide.

ln -s /var/lib/rancher/rke2/bin/containerd /usr/local/bin/containerd
ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
ln -s /var/lib/rancher/rke2/bin/crictl /usr/local/bin/crictl
ln -s /var/lib/rancher/rke2/bin/runc /usr/local/bin/runc
ln -s /var/lib/rancher/rke2/bin/ctr /usr/local/bin/ctr
ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml

Let’s also set up shell completion and an alias for kubectl. This might seem minor, but trust me, it saves a lot of typing over the course of a long lab session.

source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'alias k=kubectl' >> /etc/profile
echo 'complete -F __start_kubectl k' >> /etc/profile

Now let’s verify everything is working. First, check that the cluster is reachable.

kubectl cluster-info -v=6

You should see “Kubernetes control plane is running at https://192.168.10.11:6443". Then check the node status and all the pods in the kube-system namespace.

kubectl get node -owide

You’ll see a single node, k8s-node1, in the Ready state with the roles control-plane and etcd, running the version v1.33.7+rke2r3 on Rocky Linux 9.6 with containerd as the container runtime.

If you check the Helm releases, you’ll see that RKE2 automatically deployed Canal, CoreDNS, the metrics server, and runtime classes as Helm charts. This is another nice feature of RKE2: it uses Helm under the hood to manage its add-ons, which gives you a familiar and powerful way to customize them.

helm list -A

And finally, let’s look at all the pods running in the cluster.

kubectl get pod -A

You should see etcd, kube-apiserver, kube-controller-manager, kube-scheduler, and kube-proxy running as static pods on k8s-node1, along with the Canal DaemonSet pod, the CoreDNS deployment pod, and the metrics server. There will also be a few completed Helm install jobs.

Everything should be in a Running or Completed state. At this point, your single-node RKE2 control plane is fully operational.

Joining an Agent (Worker Node)

Now that our server node is up, let’s add a worker. The first thing we need is the join token. RKE2 generates this token during the server bootstrap and stores it on disk. On k8s-node1, retrieve it like this:

cat /var/lib/rancher/rke2/server/node-token

You’ll get a long string that looks something like K10cfbf1f601080e27248e795b54de68ea18961910d639be08257095a7109e0dbf0::server:5add6b365458d11cc8a0164c005fc749. Copy this value because we’ll need it on the worker node.

You can also verify that the supervisor port is listening. This is port 9345, and it’s the dedicated management and bootstrap API that new nodes use to join the cluster.

ss -tnlp | grep 9345

Before switching to the worker node, let’s set up a monitoring loop on k8s-node1 so we can watch the new node appear in real time.

watch -d 'kubectl get node; echo; kubectl get pod -n kube-system'

Now SSH into k8s-node2 and let’s get the agent installed. The process is very similar to the server installation, but we pass INSTALL_RKE2_TYPE=”agent” to tell the installer we want the agent role, not the server role.

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_CHANNEL=v1.33 sh -

Next, we create the agent configuration file. This is much simpler than the server config. It only needs two things: the URL of the server’s supervisor API (on port 9345) and the join token we copied earlier.

TOKEN=K10cfbf1f601080e27248e795b54de68ea18961910d639be08257095a7109e0dbf0::server:5add6b365458d11cc8a0164c005fc749

mkdir -p /etc/rancher/rke2/
cat << EOF > /etc/rancher/rke2/config.yaml
server: https://192.168.10.11:9345
token: $TOKEN
EOF

Then, just like the server, we enable and start the agent service. You can also tail the journal to watch the join process.

systemctl enable --now rke2-agent.service
journalctl -u rke2-agent -f

Flip back to your monitoring terminal on k8s-node1 and within a few seconds you’ll see k8s-node2 appear as a new node. Let’s verify the cluster now has two nodes.

kubectl get node -owide

You should see both k8s-node1 (with roles control-plane, etcd, and master) and k8s-node2 (with no specific role label, meaning it’s a pure worker). Both should be in the Ready state. If you check the pods in kube-system filtered by k8s-node2, you’ll see that only kube-proxy and the Canal DaemonSet pod were automatically deployed there, which is exactly what you’d expect for a worker node.

kubectl get pod -n kube-system -owide | grep k8s-node2

And that’s it. Two commands on the server (install + start), two commands on the agent (install + start), and a tiny config file on each side. The cluster is fully operational with a control plane and a worker node.

Deploying a Sample App

With our two-node cluster ready, let’s deploy a simple application to prove that everything is working end to end, including pod scheduling, networking, and service exposure.

We’re going to deploy a simple HTTP echo server (traefik/whoami) as a Deployment with two replicas, fronted by a NodePort Service on port 30000. We also add a pod anti-affinity rule to ensure the two replicas land on different nodes, which lets us verify that cross-node networking works.

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30000
  type: NodePort
EOF

Let’s check on the deployed resources to make sure everything is healthy.

kubectl get deploy,pod,svc,ep -owide

You should see the Deployment with 2/2 replicas available, two pods in the Running state (one on k8s-node1 and one on k8s-node2), the Service of type NodePort with port 30000 mapped, and the Endpoints object listing both pod IPs.

Now let’s hit the service from the host machine to verify connectivity. We’ll use a loop that calls the NodePort on k8s-node2 and prints the hostname returned by each response.

while true; do curl -s http://192.168.10.12:30000 | grep Hostname; date; sleep 1; done

If everything is configured correctly, you’ll see the responses alternate between the two pod hostnames. This confirms that the NodePort service is load-balancing traffic across both pods, and that the Canal overlay network is successfully routing packets between nodes.

One interesting thing to notice here is that pods got scheduled on both nodes, including the control plane node. If you check the taints on k8s-node1, you’ll find that there are none.

kubectl describe node k8s-node1 | grep -i taints

This is a deliberate RKE2 design choice. Unlike kubeadm, which applies a NoSchedule taint to control plane nodes by default, RKE2 leaves them untainted so that workloads can run on all nodes out of the box. This is especially useful for smaller clusters or edge deployments where you want to maximize resource utilization. Of course, in a production environment with dedicated control plane nodes, you’d want to add taints yourself to keep workloads off the control plane.

At this point, we have a fully functional two-node RKE2 cluster running a sample application with verified cross-node networking. The installation was remarkably straightforward: a single binary, a simple config file, one systemd command on each node, and we were up and running.

RKE2 Day-2 Operations

Once your RKE2 cluster is up and running with a server node and an agent node joined, the real work begins.

Day-2 operations are about keeping your cluster healthy, secure, and up to date over time. In this section, we will walk through three essential tasks that every RKE2 operator needs to know: managing and rotating TLS certificates, performing a manual version upgrade from v1.33 to v1.34, and setting up fully automated upgrades using the Rancher System Upgrade Controller to take the cluster all the way to v1.35. We will go step by step, show every command, and explain what is happening behind the scenes so you can confidently apply these procedures to your own clusters.

Certificate Management and Rotation

Every component in your RKE2 cluster communicates over TLS. The API server, etcd, the kubelet, the scheduler, the controller manager — they all rely on certificates to authenticate and encrypt traffic between each other. RKE2 issues these certificates automatically during installation, and each client or server certificate is valid for 365 days from the date it was issued. The Certificate Authority (CA) certificates that sign them have a much longer lifetime of 10 years.

RKE2 has a built-in mechanism for certificate renewal. Every time the rke2-server or rke2-agent service starts, it checks the expiration dates of all certificates. If any certificate is within 120 days of expiring, RKE2 will automatically renew it during startup. This renewal process reuses the existing private keys, so the certificate identity stays the same — only the validity period gets extended. Additionally, when a certificate’s expiration date falls within 120 days, Kubernetes will emit a warning event of type CertificateExpirationWarning so you can be aware of upcoming expirations even if you have not manually checked.

Let us start by inspecting the certificates on both the server and agent nodes to understand what is installed and when everything expires.

On the server node (k8s-node1), run the following command to display all certificates in a table format:

rke2 certificate check --output table

You will see output like this, showing every certificate file, its subject, usage type, expiration date, residual time, and current status:

FILENAME                           SUBJECT                             USAGES                  EXPIRES                  RESIDUAL TIME   STATUS
--------                           -------                             ------                  -------                  -------------   ------
client-kube-apiserver.crt          system:apiserver                    ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-kube-apiserver.crt          rke2-client-ca@1770647369           CertSign                Feb 07, 2036 14:29 UTC   10 years        OK
serving-kube-apiserver.crt         kube-apiserver                      ServerAuth              Feb 09, 2027 14:29 UTC   1 year          OK
serving-kube-apiserver.crt         rke2-server-ca@1770647369           CertSign                Feb 07, 2036 14:29 UTC   10 years        OK
client-rke2-cloud-controller.crt   rke2-cloud-controller-manager       ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-scheduler.crt               system:kube-scheduler               ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
kube-scheduler.crt                 kube-scheduler                      ServerAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-kube-proxy.crt              system:kube-proxy                   ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-kubelet.crt                 system:node:k8s-node1               ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
serving-kubelet.crt                k8s-node1                           ServerAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-rke2-controller.crt         system:rke2-controller              ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-admin.crt                   system:admin                        ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-auth-proxy.crt              system:auth-proxy                   ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client-controller.crt              system:kube-controller-manager      ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
kube-controller-manager.crt        kube-controller-manager             ServerAuth              Feb 09, 2027 14:29 UTC   1 year          OK
client.crt                         etcd-client                         ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK
server-client.crt                  etcd-server                         ServerAuth,ClientAuth   Feb 09, 2027 14:29 UTC   1 year          OK
peer-server-client.crt             etcd-peer                           ServerAuth,ClientAuth   Feb 09, 2027 14:29 UTC   1 year          OK
client-supervisor.crt              system:rke2-supervisor              ClientAuth              Feb 09, 2027 14:29 UTC   1 year          OK

Notice how the server node has a large number of certificates. You have client authentication certificates for every major component (API server, scheduler, controller manager, kubelet, kube-proxy, etcd, and the RKE2 supervisor), as well as server authentication certificates for the API server, scheduler, controller manager, kubelet, and etcd.

Each leaf certificate expires in 1 year, while the CA certificates that signed them expire in 10 years.

On the agent node (k8s-node2), the certificate inventory is much smaller because the agent only needs certificates for its own kubelet, kube-proxy, and the RKE2 controller:

rke2 certificate check --output table

FILENAME                     SUBJECT                     USAGES       EXPIRES                  RESIDUAL TIME   STATUS
--------                     -------                     ------       -------                  -------------   ------
client-kubelet.crt           system:node:k8s-node2       ClientAuth   Feb 14, 2027 14:21 UTC   1 year          OK
serving-kubelet.crt          k8s-node2                   ServerAuth   Feb 14, 2027 14:21 UTC   1 year          OK
client-rke2-controller.crt   system:rke2-controller      ClientAuth   Feb 14, 2027 14:21 UTC   1 year          OK
client-kube-proxy.crt        system:kube-proxy           ClientAuth   Feb 14, 2027 14:21 UTC   1 year          OK

Now, there are situations where you might want to rotate certificates manually rather than waiting for the automatic renewal window. Perhaps you suspect a certificate has been compromised, or you are performing a planned security rotation as part of your organization’s compliance requirements. RKE2 provides a straightforward command for this.

The process requires three steps: stop the RKE2 server, rotate the certificates, and then start the server again. Here is the full procedure on the server node (k8s-node1):

First, stop the RKE2 server service:

systemctl stop rke2-server

Next, run the certificate rotation command:

rke2 certificate rotate

You will see output confirming that every certificate is being rotated:

INFO[0000] Server detected, rotating agent and server certificates
INFO[0000] Rotating dynamic listener certificate
INFO[0000] Rotating certificates for admin
INFO[0000] Rotating certificates for controller-manager
INFO[0000] Rotating certificates for kube-proxy
INFO[0000] Rotating certificates for kubelet
INFO[0000] Rotating certificates for rke2-controller
INFO[0000] Rotating certificates for api-server
INFO[0000] Rotating certificates for auth-proxy
INFO[0000] Rotating certificates for cloud-controller
INFO[0000] Rotating certificates for etcd
INFO[0000] Rotating certificates for scheduler
INFO[0000] Rotating certificates for supervisor
INFO[0000] Successfully backed up certificates to /var/lib/rancher/rke2/server/tls-1770651290, please restart rke2 server or agent to rotate certificates

Notice that RKE2 automatically backs up the old certificates to a timestamped directory under /var/lib/rancher/rke2/server/. This is a nice safety net — if anything goes wrong, you can restore the previous certificates.

You can verify the new certificate expiration dates before restarting:

rke2 certificate check --output table

Now start the server back up:

systemctl start rke2-server

After the server is running, verify everything looks good once more:

rke2 certificate check --output table

There is one important thing to remember after certificate rotation: the kubeconfig file at /etc/rancher/rke2/rke2.yaml gets regenerated with the new certificate data.

If you previously copied this file to ~/.kube/config, you need to update your copy. You can check if they differ and then copy the updated file:

diff /etc/rancher/rke2/rke2.yaml ~/.kube/config
yes | cp /etc/rancher/rke2/rke2.yaml ~/.kube/config ; echo
kubectl cluster-info

A great thing about RKE2’s certificate rotation is that you do not need to do anything special on the worker nodes. When the rke2-agent service on a worker node detects that the connection to the server has been interrupted (because the server’s certificates changed), it will automatically reconnect and receive new certificates from the server. This behavior is very similar to how kubeadm-based clusters handle certificate renewal on worker nodes — the worker simply re-bootstraps its credentials upon reconnection.

Manual Version Upgrade (v1.33 to v1.34)

Upgrading RKE2 manually is refreshingly simple compared to many other Kubernetes distributions. The process involves running the same installation script you used initially, but pointing it to a newer version channel. RKE2 will upgrade the binaries and restart the control plane components, often with minimal disruption to running workloads.

Before starting, let us set up some monitoring so we can observe what happens during the upgrade. Open several terminal windows on the server node and run these commands:

In one terminal, continuously curl your sample application to verify it stays accessible throughout the upgrade:

while true; do curl -s http://192.168.10.12:30000 | grep Hostname; date; sleep 1; done

In another terminal, watch the kube-system pods to see which ones get recreated:

watch -d "kubectl get pod -n kube-system -owide --sort-by=.metadata.creationTimestamp | tac"

In a third terminal, watch the node status:

watch -d "kubectl get node"

And optionally, if you want to monitor etcd cluster health during the upgrade:

watch -d etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/rke2/server/tls/etcd/client.key \
  member list --write-out=table

Now confirm your current version:

rke2 --version

rke2 version v1.33.7+rke2r3 (7e4fd1a82edf497cab91c220144619bbad659cf4)
go version go1.24.11 X:boringcrypto

You can also check what versions are available across different release channels by querying the RKE2 update API:

curl -s https://update.rke2.io/v1-release/channels | jq .data

This returns a JSON array with all available channels including stable, latest, and version-specific channels like v1.34 and v1.35. Each entry shows the latest version available in that channel.

Now let us upgrade the server node. On k8s-node1, run the RKE2 installer with the v1.34 channel specified:

curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.34 sh -

The installer will download and install the new RPM packages. You will see the transaction output showing the old v1.33 packages being replaced by v1.34:

Running transaction
  Preparing        :                                                        1/1
  Upgrading        : rke2-common-1.34.3~rke2r3-0.el9.aarch64               1/4
  Upgrading        : rke2-server-1.34.3~rke2r3-0.el9.aarch64               2/4
  Running scriptlet: rke2-server-1.34.3~rke2r3-0.el9.aarch64               2/4
  Running scriptlet: rke2-server-1.33.7~rke2r3-0.el9.aarch64               3/4
  Cleanup          : rke2-server-1.33.7~rke2r3-0.el9.aarch64               3/4
  Running scriptlet: rke2-server-1.33.7~rke2r3-0.el9.aarch64               3/4
  Running scriptlet: rke2-common-1.33.7~rke2r3-0.el9.aarch64               4/4
  Cleanup          : rke2-common-1.33.7~rke2r3-0.el9.aarch64               4/4
  Running scriptlet: rke2-common-1.33.7~rke2r3-0.el9.aarch64               4/4
  Verifying        : rke2-common-1.34.3~rke2r3-0.el9.aarch64               1/4
  Verifying        : rke2-common-1.33.7~rke2r3-0.el9.aarch64               2/4
  Verifying        : rke2-server-1.34.3~rke2r3-0.el9.aarch64               3/4
  Verifying        : rke2-server-1.33.7~rke2r3-0.el9.aarch64               4/4

What is remarkable here is that just running the installer script is enough to trigger the upgrade of the control plane components. During the scriptlet phase, the RKE2 process detects the new binaries and recreates the static pod manifests, which causes the kubelet to restart the control plane pods.

If you are watching your monitoring terminals, you will see the etcd, kube-apiserver, and kube-proxy pods get recreated first, followed by the scheduler and controller manager. And throughout this entire process, if you are watching your application curl loop, you should see that application traffic continues to flow without interruption.

Verify the new version:

rke2 --version

rke2 version v1.34.3+rke2r3 (7598946e0086a9131564ccbb3c142b3fa54516ad)
go version go1.24.11 X:boringcrypto

You can also confirm the yum repository was updated to point to the v1.34 channel:

dnf repolist

rancher-rke2-1.34-stable       Rancher RKE2 1.34 (v1.34)
rancher-rke2-common-stable     Rancher RKE2 Common (v1.34)

To see which container images each kube-system pod is running after the upgrade:

kubectl get pods -n kube-system \
  -o custom-columns=\
POD:.metadata.name,\
CONTAINERS:.spec.containers[*].name,\
IMAGES:.spec.containers[*].image

Now restart the RKE2 server service to ensure everything is fully reconciled:

systemctl restart rke2-server

Check the node versions to see the server is upgraded but the worker is still on the old version:

kubectl get node -owide

NAME        STATUS   ROLES                       AGE     VERSION          INTERNAL-IP     ...
k8s-node1   Ready    control-plane,etcd,master   18m     v1.34.3+rke2r3   192.168.10.11   ...
k8s-node2   Ready    <none>                      7m23s   v1.33.7+rke2r3   192.168.10.12   ...

This is the expected intermediate state. Kubernetes supports a version skew of up to two minor versions between the control plane and worker nodes, so having the server on v1.34 and the worker on v1.33 is perfectly fine temporarily.

Now let us upgrade the worker node. Switch to k8s-node2 and run the installer, this time specifying the agent type:

rke2 --version
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent INSTALL_RKE2_CHANNEL=v1.34 sh -

Verify the new version on the worker:

rke2 --version

rke2 version v1.34.3+rke2r3 (7598946e0086a9131564ccbb3c142b3fa54516ad)
go version go1.24.11 X:boringcrypto

Confirm the repository was updated:

dnf repolist

rancher-rke2-1.34-stable       Rancher RKE2 1.34 (v1.34)
rancher-rke2-common-stable     Rancher RKE2 Common (v1.34)

Restart the agent service:

systemctl restart rke2-agent

Now go back to the server node (k8s-node1) and verify that both nodes are running the same version:

kubectl get node -owide

NAME        STATUS   ROLES                       AGE   VERSION          INTERNAL-IP     ...
k8s-node1   Ready    control-plane,etcd,master   22m   v1.34.3+rke2r3   192.168.10.11   ...
k8s-node2   Ready    <none>                      11m   v1.34.3+rke2r3   192.168.10.12   ...

If you check the kube-system pods sorted by creation time, you will notice that only the kube-proxy pod on the worker node was recreated during the worker upgrade. The rest of the worker node pods (like canal) continued running unchanged:

kubectl get pod -n kube-system --sort-by=.metadata.creationTimestamp | tac

The entire manual upgrade process is remarkably smooth. The key takeaway is the order of operations: always upgrade the server (control plane) first, then upgrade the agent (worker) nodes one by one. This ensures the control plane is always at the same or newer version than the workers, which is a fundamental Kubernetes requirement.

Automated Upgrade with System Upgrade Controller

Manual upgrades work well for small clusters, but when you are managing many nodes or want to ensure upgrades happen reliably without human intervention, the Rancher System Upgrade Controller is the way to go.

This controller watches for custom Plan resources that declaratively specify which nodes should be upgraded, to which version, and in what order. It then creates Jobs that run on each target node to perform the actual upgrade.

Before we begin, let us set up our monitoring terminals again. On the server node (k8s-node1):

Watch the upgrade plans and their associated jobs and pods:

watch -d 'kubectl -n system-upgrade get plans -o wide; echo ; kubectl -n system-upgrade get jobs,pods'

Watch the node versions:

watch -d "kubectl get node"

Watch the kube-system pods:

watch -d "kubectl get pod -n kube-system -owide --sort-by=.metadata.creationTimestamp | tac"

And keep your application health check running:

while true; do curl -s http://192.168.10.12:30000 | grep Hostname; date; sleep 1; done

The first step is to install the System Upgrade Controller itself. It consists of a CRD (the Plan resource definition) and a controller deployment that watches for Plan resources:

kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml \
  -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml

This creates several resources: a system-upgrade namespace, a ServiceAccount, RBAC roles and bindings (including a special “drainer” ClusterRole that allows the controller to cordon and drain nodes), a ConfigMap with default environment variables, and the controller Deployment itself.

Verify the installation:

kubectl get deploy,pod,cm -n system-upgrade

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/system-upgrade-controller   1/1     1            1           12m

NAME                                             READY   STATUS    RESTARTS   AGE
pod/system-upgrade-controller-6f9f9b8cf4-46n82   1/1     Running   0          12m

NAME                               DATA   AGE
configmap/default-controller-env   10     12m

Also confirm the CRD was registered:

kubectl get crd | grep upgrade

$ plans.upgrade.cattle.io   2026-02-09T18:08:08Z

You can tail the controller logs to watch what happens in real time:

kubectl logs -n system-upgrade -l app.kubernetes.io/name=system-upgrade-controller -f

Now comes the interesting part — defining the upgrade plans. We will create two Plan resources: one for server nodes (control plane) and one for agent nodes (workers). The key design here is that the agent plan has a prepare step that references the server plan. This ensures that all server nodes are upgraded before any agent node begins its upgrade, which respects the Kubernetes version skew policy.

cat << EOF | kubectl apply -f -
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/rke2-upgrade
  channel: https://update.rke2.io/v1-release/channels/latest
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/rke2-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/rke2-upgrade
  channel: https://update.rke2.io/v1-release/channels/latest
EOF

Let us break down what each field does. The concurrency field is set to 1, meaning only one node will be upgraded at a time. This is the safest approach — you can increase it if you have a large cluster and want faster rollouts, but for most on-premise environments, upgrading one node at a time is the prudent choice. The cordon field is set to true, which means each node will be cordoned (marked as unschedulable) before the upgrade begins, preventing new pods from being scheduled on it during the process.

The nodeSelector is what differentiates the two plans. The server plan targets nodes that have the node-role.kubernetes.io/control-plane label set to “true”, while the agent plan targets nodes where that label does not exist at all. This cleanly separates control plane nodes from worker nodes.

The channel field points to the RKE2 update API, specifically the “latest” channel. The controller will query this URL to determine what the latest available version is. You could alternatively use a version field with a specific version string like “v1.35.0+rke2r3” if you want to pin to an exact version rather than following the latest channel.

The prepare section in the agent plan is crucial for ordering. It tells the controller to run a preparation step using the rancher/rke2-upgrade image with the argument “prepare server-plan”. This step will wait until the server-plan has completed successfully on all matching nodes before proceeding with the agent upgrade. This is how the controller enforces the correct upgrade order.

As soon as you apply these Plan resources, the controller springs into action. It resolves the latest version from the channel URL, creates Jobs for each target node, and begins the upgrade process. You can watch the progress by checking the plan status:

kubectl -n system-upgrade get plans -o wide

NAME          IMAGE                  CHANNEL                                             VERSION   COMPLETE   MESSAGE   APPLYING
agent-plan    rancher/rke2-upgrade   https://update.rke2.io/v1-release/channels/latest             True
server-plan   rancher/rke2-upgrade   https://update.rke2.io/v1-release/channels/latest             True

Check the jobs that were created:

kubectl -n system-upgrade get jobs

NAME                                                              STATUS     COMPLETIONS   DURATION   AGE
apply-agent-plan-on-k8s-node2-with-db1bffd09b601fca4c7c06-7dc30   Complete   1/1           2m5s       4m16s
apply-server-plan-on-k8s-node1-with-db1bffd09b601fca4c7c0-28ad1   Complete   1/1           55s        4m16s

And the pods that performed the actual upgrades:

kubectl get pod -n system-upgrade -owide

NAME                                                              READY   STATUS      RESTARTS   AGE     IP              NODE        ...
apply-agent-plan-on-k8s-node2-with-db1bffd09b601fca4c7c06-54frb   0/1     Unknown     0          4m30s   192.168.10.12   k8s-node2   ...
apply-agent-plan-on-k8s-node2-with-db1bffd09b601fca4c7c06-jrsnq   0/1     Completed   0          2m37s   192.168.10.12   k8s-node2   ...
apply-server-plan-on-k8s-node1-with-db1bffd09b601fca4c7c0-xg4jl   0/1     Unknown     0          4m30s   192.168.10.11   k8s-node1   ...
apply-server-plan-on-k8s-node1-with-db1bffd09b601fca4c7c0-msrt2   0/1     Completed   0          3m43s   192.168.10.11   k8s-node1   ...
system-upgrade-controller-6f9f9b8cf4-zksgq                        1/1     Running     0          8m20s   10.42.0.8       k8s-node1   ...

You might notice some pods with an “Unknown” status. These are the initial pods that were running on the node when it was being restarted during the upgrade — they lost connection and could not report their final status. The “Completed” pods are the ones that actually finished the upgrade successfully on a second attempt.

One very important thing to understand about these upgrade pods is the level of access they require. Because they need to modify the host operating system (installing new RKE2 binaries, restarting systemd services), they run with significant privileges. You can inspect this:

kubectl describe pod -n system-upgrade | grep ^Volumes: -A4

Volumes:
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory

The upgrade pods mount the entire host root filesystem at /host with read and write permissions. They also use the host IPC, NET, and PID namespaces, and have the CAP_SYS_BOOT capability. This level of access is necessary for the upgrade to work, but it is something you should be aware of from a security perspective, especially in production environments. Make sure you review and understand these permissions before deploying the System Upgrade Controller in a sensitive environment.

You can also check the controller logs to see the full sequence of events:

kubectl logs -n system-upgrade -l app.kubernetes.io/name=system-upgrade-controller

object="system-upgrade/server-plan" ... type="Normal" reason="Resolved" message="Resolved latest version from Spec.Channel: v1.35.0-rke2r3"
object="system-upgrade/server-plan" ... type="Normal" reason="SyncJob" message="Jobs synced for version v1.35.0-rke2r3 on Nodes k8s-node1. Hash: db1bffd09b601fca4c7c067d987c4d368f9237f8219289438d8678e8"
object="system-upgrade/server-plan" ... type="Normal" reason="JobComplete" message="Job completed on Node k8s-node1"
object="system-upgrade/server-plan" ... type="Normal" reason="Complete" message="Jobs complete for version v1.35.0-rke2r3. Hash: db1bffd09b601fca4c7c067d987c4d368f9237f8219289438d8678e8"

The logs clearly show the sequence: the controller resolved the latest version as v1.35.0-rke2r3, synced upgrade jobs to the server node first, waited for completion, and then proceeded with the agent nodes.

Finally, verify that both nodes are now running the latest version:

kubectl get node -owide

NAME        STATUS   ROLES                       AGE    VERSION          INTERNAL-IP     ...
k8s-node1   Ready    control-plane,etcd,master   108m   v1.35.0+rke2r3   192.168.10.11   ...
k8s-node2   Ready    <none>                      97m    v1.35.0+rke2r3   192.168.10.12   ...

Both nodes are now on v1.35.0. The automated upgrade completed successfully, upgrading the server first and then the agent, exactly as specified in the Plan resources.

The beauty of this approach is that it is entirely declarative. You define your desired state (which version, which nodes, in what order), and the controller makes it happen. If you add new nodes to the cluster later, the controller will automatically apply the upgrade plan to them as well, as long as they match the nodeSelector. This makes the System Upgrade Controller an excellent choice for clusters that need to stay current with minimal operational overhead.

Cluster API Overview — Management vs. Workload clusters

Before we continue the hands-on part, let’s take a step back and understand what Cluster API actually is and why it exists. If you’ve ever managed more than a handful of Kubernetes clusters, you know the pain. Every cluster needs to be provisioned, configured, upgraded, and eventually decommissioned. Doing all of this manually or with a patchwork of scripts quickly becomes a nightmare as the number of clusters grows. Cluster API, often abbreviated as CAPI, was born out of this exact problem.

At its core, Cluster API is a Kubernetes-native tool that lets you create, configure, and manage Kubernetes clusters using Kubernetes itself. Yes, you read that right. You use Kubernetes custom resources to describe what your cluster should look like, and a set of controllers running inside a Kubernetes cluster takes care of making that a reality. Think of it as applying the declarative, reconciliation-driven model that Kubernetes uses for Pods and Deployments, but now applied to entire clusters, nodes, and control planes.

This brings us to the two most important concepts you need to understand before going any further: the Management Cluster and the Workload Cluster.

The Management Cluster is the Kubernetes cluster where Cluster API components are installed and running. It hosts all the CAPI controllers, the custom resource definitions (CRDs), and the provider-specific controllers. This is the cluster that watches your declarative cluster definitions and does the heavy lifting of provisioning infrastructure, bootstrapping nodes, and orchestrating upgrades. You can think of it as the “brain” that controls everything. In our hands-on lab, we use a simple KinD (Kubernetes in Docker) cluster as the management cluster, but in a real production environment, this would typically be a dedicated, highly available cluster that you treat with extra care.

The Workload Cluster, on the other hand, is the cluster that gets created and managed by the Management Cluster. This is where your actual applications run. You don’t install Cluster API components on workload clusters. Instead, the management cluster provisions them from scratch, including spinning up the underlying infrastructure (VMs, containers, or bare metal), bootstrapping each node with kubeadm, forming the control plane, and joining worker nodes. From the workload cluster’s perspective, it’s just a normal Kubernetes cluster. It has no idea that it was created and is being managed by another cluster.

Now let’s talk about how this is structured internally. Cluster API is designed around a pluggable provider model, and there are four types of providers that work together to make everything happen.

The first is the Core Provider, which lives in the capi-system namespace. This is the main Cluster API controller that manages the fundamental CRDs like Cluster, Machine, MachineSet, and MachineDeployment. It orchestrates the overall reconciliation loop. When you check the installed providers in our lab, you'll see it like this:

kubectl get providers.clusterctl.cluster.x-k8s.io -A

NAMESPACE                           NAME                    TYPE                     PROVIDER      VERSION
capi-system                         cluster-api             CoreProvider             cluster-api   v1.12.2

The second is the Bootstrap Provider. In our case, this is the Kubeadm Bootstrap Provider, running in the capi-kubeadm-bootstrap-system namespace. Its job is to generate the cloud-init or user-data configuration that each node needs to bootstrap itself into a functioning Kubernetes node. It creates the kubeadm init and kubeadm join configurations so that control plane nodes and worker nodes know how to set themselves up. You can verify it's running:

kubectl get providers -n capi-kubeadm-bootstrap-system bootstrap-kubeadm -o yaml

providerName: kubeadm
type: BootstrapProvider
version: v1.12.2

The third is the Control Plane Provider. This one, running in capi-kubeadm-control-plane-system, is specifically responsible for managing the lifecycle of control plane nodes. It handles the KubeadmControlPlane resource, which means it takes care of scaling the control plane up or down, rolling out upgrades to control plane nodes one at a time, and making sure etcd membership is correctly managed throughout. This is critical for safe, zero-downtime upgrades:

kubectl get providers -n capi-kubeadm-control-plane-system control-plane-kubeadm -o yaml

providerName: kubeadm
type: ControlPlaneProvider
version: v1.12.2

The fourth and final piece is the Infrastructure Provider. This is where things get environment-specific. The infrastructure provider is responsible for actually creating the underlying resources, whether that’s VMs on vSphere, instances on AWS, or in our lab’s case, Docker containers that simulate nodes. Our lab uses the Docker provider (CAPD), which runs in the capd-system namespace. It's important to note that the Docker provider is designed strictly for development and testing purposes, not for production. But it's perfect for learning because it lets you see the full Cluster API workflow without needing access to a cloud or hypervisor:

kubectl get providers -n capd-system infrastructure-docker -o yaml

providerName: docker
type: InfrastructureProvider
version: v1.12.2

When you put all four providers together and look at the pods running across the management cluster, you get a clear picture of the architecture:

kubectl get pod -A

NAMESPACE                           NAME                                                            READY   STATUS
capd-system                         capd-controller-manager-7c9d67ffdf-7npsd                        1/1     Running
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-bd5f89bbd-9c9ng       1/1     Running
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-55c48d9b5-bckj5   1/1     Running
capi-system                         capi-controller-manager-6cc7b949c4-rmd7h                        1/1     Running
cert-manager                        cert-manager-598d877b78-9lkmd                                   1/1     Running
cert-manager                        cert-manager-cainjector-6b5777d564-7mfzz                        1/1     Running
cert-manager                        cert-manager-webhook-5d9fc6b4ff-slscg                           1/1     Running

You’ll also notice cert-manager is installed. This is a dependency that Cluster API installs automatically because the controllers use TLS certificates for webhook communication.

Another concept worth understanding is ClusterClass, which was introduced as an experimental feature and is what we use in our lab. ClusterClass lets you define a reusable “template” or “blueprint” for clusters. Instead of specifying every detail for each individual cluster, you define a ClusterClass once, covering things like what machine templates to use, how the control plane should be configured, and what the worker node pools look like. Then, when you create an actual Cluster resource, you simply reference the ClusterClass and provide a minimal set of overrides like the cluster name, Kubernetes version, and replica counts. This is enabled by setting the CLUSTER_TOPOLOGY feature gate to true before initializing the management cluster:

export CLUSTER_TOPOLOGY=true
clusterctl init --infrastructure docker

And when you look at the feature gates that the core controller is running with, you can confirm it:

kubectl describe -n capi-system deployment.apps/capi-controller-manager | grep feature-gates

--feature-gates=MachinePool=true,ClusterTopology=true,RuntimeSDK=false,...

With ClusterTopology enabled, the generated cluster manifest contains both a ClusterClass definition (the blueprint) and a Cluster resource that references it. If you inspect the generated YAML, the resource types make this clear:

cat capi-quickstart.yaml | grep -E '^apiVersion:|^kind:'

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerClusterTemplate
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlaneTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachinePoolTemplate
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster

The ClusterClass at the top is the blueprint. The Cluster at the bottom is the actual workload cluster that references it. Everything in between, the DockerClusterTemplate, KubeadmControlPlaneTemplate, DockerMachineTemplates, and KubeadmConfigTemplate, are the building blocks that the ClusterClass stitches together.

One of the most powerful benefits of this architecture is how upgrades work. Because the management cluster controls the workload cluster declaratively, upgrading the Kubernetes version of an entire workload cluster is as simple as patching a single field on the Cluster resource. The controllers handle the rest: they roll out new control plane nodes one by one, wait for each to become healthy, drain and remove old nodes, and then do the same for worker nodes. There’s no need to SSH into machines or run manual commands. You just change the desired version, and the system converges to match it.

To summarize the mental model: the Management Cluster is the operator, and the Workload Cluster is what gets operated on. The Management Cluster runs CAPI controllers that watch custom resources describing desired cluster state. When you create, modify, or delete these resources, the controllers spring into action, provisioning or tearing down real infrastructure to match your intent. This separation of concerns means you can manage dozens or even hundreds of workload clusters from a single management cluster, all using the same familiar Kubernetes API and tooling you already know.

Cluster API Hands-On

In this section, we will walk through every step of setting up a management cluster, provisioning a full workload cluster with three control plane nodes and three workers, deploying a sample application onto it, and finally upgrading that workload cluster to a newer Kubernetes version. Everything here uses the Docker infrastructure provider, which means each “machine” is actually a Docker container pretending to be a node. This makes it perfect for learning and experimentation on your local machine without needing any cloud accounts or bare-metal servers.

Setting Up the Management Cluster (KinD + Docker Provider)

Before we can use Cluster API to provision workload clusters, we need a management cluster. This is a regular Kubernetes cluster that runs the Cluster API controllers. These controllers watch for custom resources like Cluster, Machine, MachineDeployment, and so on, and then reconcile the desired state by creating actual infrastructure. In our case, the infrastructure provider is Docker, so the controllers will create Docker containers to act as Kubernetes nodes.

Let us start by creating a working directory. This keeps all our files organized in one place.

mkdir capi-docker && cd capi-docker

Before proceeding, it is a good idea to check your Docker context and make sure there are no leftover containers from previous experiments that might interfere.

docker context ls
docker ps -a

Now we create a KinD (Kubernetes in Docker) cluster. This will serve as our management cluster. Notice that we mount the Docker socket into the KinD node. This is critical because the Cluster API Docker provider needs access to the host’s Docker daemon in order to create containers that will become our workload cluster nodes. We also set up a couple of extra port mappings for convenience, one for a sample application and one for kube-ops-view, a nice visual dashboard.

kind create cluster --name myk8s --image kindest/node:v1.35.0 --config - <<EOF

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /var/run/docker.sock 
    containerPath: /var/run/docker.sock
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
EOF

Once the cluster is up, you can optionally install kube-ops-view on the management cluster. This gives you a nice visual representation of your cluster’s pods and nodes. It is not required, but it makes the experience much more enjoyable when you are watching things happen in real time.

helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 \
  --set service.main.type=NodePort,service.main.ports.http.nodePort=30001 \
  --set env.TZ="Asia/Seoul" --namespace kube-system

You can then open your browser and navigate to http://127.0.0.1:30001/#scale=1.5 to see the dashboard.

Next, we need the clusterctl command-line tool. This is the primary CLI for interacting with Cluster API. On macOS you can install it with Homebrew, and on Linux you can download the binary directly.

# macOS
brew install clusterctl  

# Linux amd64
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.2/clusterctl-linux-amd64 -o clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

Verify the installation by checking the version.

clusterctl version -o json | jq

Now comes the important part. We initialize the management cluster by telling clusterctl to install all the necessary Cluster API components with the Docker infrastructure provider. We also enable the ClusterTopology experimental feature, which lets us use ClusterClass resources. ClusterClass is essentially a reusable template that defines how clusters should be structured, making it much easier to stamp out multiple clusters with consistent configurations.

export CLUSTER_TOPOLOGY=true
clusterctl init --infrastructure docker

This command does a lot behind the scenes. It installs cert-manager (which Cluster API depends on for webhook certificates), and then it deploys four sets of controllers into separate namespaces. You will see output indicating that each provider is being installed. Once it finishes, let us verify that everything landed correctly.

kubectl get pod -A

You should see pods running in several new namespaces. The capd-system namespace contains the Docker infrastructure provider controller. The capi-system namespace has the core Cluster API controller. The capi-kubeadm-bootstrap-system namespace runs the bootstrap provider, which is responsible for generating the cloud-init or kubeadm configuration that nodes use when they first boot. And capi-kubeadm-control-plane-system runs the control plane provider, which manages the lifecycle of control plane nodes including scaling and upgrades.

NAMESPACE                           NAME                                                            READY   STATUS
capd-system                         capd-controller-manager-7c9d67ffdf-7npsd                        1/1     Running
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-bd5f89bbd-9c9ng       1/1     Running
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-55c48d9b5-bckj5   1/1     Running
capi-system                         capi-controller-manager-6cc7b949c4-rmd7h                        1/1     Running
cert-manager                        cert-manager-598d877b78-9lkmd                                   1/1     Running
cert-manager                        cert-manager-cainjector-6b5777d564-7mfzz                        1/1     Running
cert-manager                        cert-manager-webhook-5d9fc6b4ff-slscg                           1/1     Running

You can also inspect the installed providers using the providers custom resource. This is a handy way to confirm which providers are active and what version they are running.

kubectl get providers.clusterctl.cluster.x-k8s.io -A

The output tells you exactly what is installed: the core Cluster API provider, the kubeadm bootstrap provider, the kubeadm control plane provider, and the Docker infrastructure provider, all at version v1.12.2.

NAMESPACE                           NAME                    TYPE                     PROVIDER      VERSION
capd-system                         infrastructure-docker   InfrastructureProvider   docker        v1.12.2
capi-system                         cluster-api             CoreProvider             cluster-api   v1.12.2
capi-kubeadm-bootstrap-system       bootstrap-kubeadm       BootstrapProvider        kubeadm       v1.12.2
capi-kubeadm-control-plane-system   control-plane-kubeadm   ControlPlaneProvider     kubeadm       v1.12.2

A large number of Custom Resource Definitions (CRDs) have also been installed. These define all the resource types that Cluster API uses to represent clusters, machines, machine deployments, machine pools, and more. You can see them all by running the following command.

kubectl get crd | grep x-k8s

At this point, your management cluster is fully initialized and ready to provision workload clusters. The controllers are running, the CRDs are in place, and the Docker provider is standing by to create containers on demand.

Provisioning a Workload Cluster (3 CP + 3 Workers)

With the management cluster ready, we can now tell it to create a workload cluster. The first thing to do is set some environment variables that control the networking configuration of the new cluster. These values will be injected into the cluster manifest when we generate it.

export SERVICE_CIDR=["10.20.0.0/16"]
export POD_CIDR=["10.10.0.0/16"]
export SERVICE_DOMAIN="myk8s-1.local"
export POD_SECURITY_STANDARD_ENABLED="false"

The SERVICE_CIDR defines the IP range for Kubernetes Services. The POD_CIDR defines the IP range for Pods. The SERVICE_DOMAIN sets the internal DNS domain. And we disable Pod Security Standards for simplicity in this lab environment.

Now we use clusterctl to generate the cluster manifest. The development flavor is specifically designed for the Docker provider. We request Kubernetes v1.34.3 with three control plane nodes and three worker nodes.

clusterctl generate cluster capi-quickstart --flavor development \
  --kubernetes-version v1.34.3 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

This produces a YAML file containing all the Cluster API resources needed to describe our desired cluster. Let us peek at what kinds of resources are in there.

cat capi-quickstart.yaml | grep -E '^apiVersion:|^kind:'

The output reveals quite a few resource types, and understanding them helps demystify how Cluster API works.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerClusterTemplate
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlaneTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: DockerMachinePoolTemplate
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster

The ClusterClass is like a blueprint. It defines what a cluster should look like in terms of its infrastructure, control plane configuration, and worker node templates. The Cluster resource at the bottom is the actual instance that references this ClusterClass and specifies the concrete values like version and replica counts. The various template resources (DockerClusterTemplate, DockerMachineTemplate, KubeadmControlPlaneTemplate, KubeadmConfigTemplate) define how each piece of infrastructure and configuration should be created.

Now let us apply this manifest to the management cluster and watch the magic happen.

kubectl apply -f capi-quickstart.yaml

You should see output confirming that all the resources were created.

clusterclass.cluster.x-k8s.io/quick-start created
dockerclustertemplate.infrastructure.cluster.x-k8s.io/quick-start-cluster created
kubeadmcontrolplanetemplate.controlplane.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinetemplate created
dockermachinepooltemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinepooltemplate created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/quick-start-default-worker-bootstraptemplate created
cluster.cluster.x-k8s.io/capi-quickstart created

The controllers immediately begin reconciling. The Docker provider starts creating containers, the bootstrap provider generates kubeadm configurations, and the control plane provider orchestrates the bring-up of each control plane node one by one. You can watch the progress in several ways. Opening a few terminal windows with monitoring commands makes this much more fun to observe.

watch -d "docker ps ; echo ; clusterctl describe cluster capi-quickstart"
watch -d kubectl get cluster -o wide
watch -d kubectl get machines

After a few minutes, if you run docker ps, you will see new containers appearing. These are the workload cluster nodes, each running the kindest/node image.

docker ps

You will notice several types of containers. There are three containers with port 6443 exposed, which are the control plane nodes. Three containers without exposed ports are the worker nodes. One container running kindest/haproxy serves as a load balancer in front of the control plane API servers. And the original myk8s-control-plane container is your management cluster.

CONTAINER ID   IMAGE                                COMMAND                  NAMES
d7622e825f15   kindest/node:v1.34.3                 "/usr/local/bin/entr…"   capi-quickstart-hvt8l-plmp7
803b5917463f   kindest/node:v1.34.3                 "/usr/local/bin/entr…"   capi-quickstart-hvt8l-j6tpk
051bcc807657   kindest/node:v1.34.3                 "/usr/local/bin/entr…"   capi-quickstart-md-0-64l6n-9hjzs-lcjv4
46b27b6eb154   kindest/node:v1.34.3                 "/usr/local/bin/entr…"   capi-quickstart-md-0-64l6n-9hjzs-8rz7j
a5271457c0d6   kindest/node:v1.34.3                 "/usr/local/bin/entr…"   capi-quickstart-md-0-64l6n-9hjzs-qnd5z
1607db8af33f   kindest/node:v1.34.3                 "/usr/local/bin/entr…"   capi-quickstart-hvt8l-lpp75
6c1f5141492c   kindest/haproxy:v20230606-42a2262b   "haproxy -W -db -f /…"   capi-quickstart-lb
2f4d7bd4d2ad   kindest/node:v1.35.0                 "/usr/local/bin/entr…"   myk8s-control-plane

Now we need to get the kubeconfig for the workload cluster so we can interact with it directly.

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

There is an important gotcha here that is worth mentioning. If you are running KinD on your local machine, the kubeconfig file may contain an internal Docker network IP address (something like 172.18.0.3) as the API server endpoint. This address is not reachable from your host machine.

You need to replace it with the host-accessible address. Check docker ps for the capi-quickstart-lb container’s port mapping. If it shows something like 0.0.0.0:55000->6443/tcp, then edit the kubeconfig to point to 127.0.0.1:55000 instead.

# Check the LB port mapping
docker ps | grep capi-quickstart-lb

# Edit the kubeconfig
vim capi-quickstart.kubeconfig
# Change server: https://172.18.0.3:6443
# To:    server: https://127.0.0.1:55000

Try connecting to the workload cluster.

kubectl --kubeconfig=capi-quickstart.kubeconfig get nodes -owide

At this point, all six nodes will show up but they will be in NotReady status. This is expected and completely normal. The nodes are not ready because there is no CNI (Container Network Interface) plugin installed yet. Without a CNI plugin, pods cannot communicate with each other, and the kubelet reports the node as not ready. Let us fix that by installing Calico.

kubectl --kubeconfig=capi-quickstart.kubeconfig apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

Give it a minute or so, and then check the nodes again.

kubectl --kubeconfig=capi-quickstart.kubeconfig get nodes -owide

Now you should see all six nodes in Ready status, three control plane nodes and three workers, all running Kubernetes v1.34.3 on Debian Bookworm with containerd as the container runtime.

NAME                                     STATUS   ROLES           AGE     VERSION   INTERNAL-IP
capi-quickstart-md-0-lt794-jfjvx-9wrql   Ready    <none>          113s    v1.34.3   192.168.97.6
capi-quickstart-md-0-lt794-jfjvx-f4pr4   Ready    <none>          113s    v1.34.3   192.168.97.5
capi-quickstart-md-0-lt794-jfjvx-rjfz9   Ready    <none>          113s    v1.34.3   192.168.97.7
capi-quickstart-sn62k-6n9sd              Ready    control-plane   2m14s   v1.34.3   192.168.97.4
capi-quickstart-sn62k-pr4kp              Ready    control-plane   46s     v1.34.3   192.168.97.9
capi-quickstart-sn62k-xhmtw              Ready    control-plane   75s     v1.34.3   192.168.97.8

You can also use clusterctl describe to get a nice summary view of the entire cluster hierarchy.

clusterctl describe cluster capi-quickstart

The output shows the cluster, its infrastructure (DockerCluster), the control plane (KubeadmControlPlane) with its three machines, and the workers (MachineDeployment) with their three machines. Everything should show as Ready and Available.

NAME                                                           READY   STATUS
Cluster/capi-quickstart                                        True    Available
├─ClusterInfrastructure - DockerCluster/capi-quickstart-wcb2g  True    Ready
├─ControlPlane - KubeadmControlPlane/capi-quickstart-sn62k     True    Available
│ └─3 Machines...                                              True    Ready
└─Workers
  └─MachineDeployment/capi-quickstart-md-0-lt794               True    Available
    └─3 Machines...                                            True    Ready

Your workload cluster is now fully operational.

Deploying a Sample App

To make things more convenient for the rest of our work, let us set up a shell alias so we do not have to type the kubeconfig flag every time.

alias k8s1='kubectl --kubeconfig=capi-quickstart.kubeconfig'
k8s1 cluster-info

You should see the control plane endpoint and CoreDNS reported as running. Optionally, you can also install kube-ops-view on the workload cluster for visual monitoring. This time we use port-forward since we do not have a NodePort conveniently mapped.

helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 \
  --set env.TZ="Asia/Seoul" --namespace kube-system --kubeconfig=capi-quickstart.kubeconfig

k8s1 -n kube-system port-forward svc/kube-ops-view 8080:8080 &

Open http://127.0.0.1:8080/#scale=1.5 in your browser to see the workload cluster’s visual dashboard.

Now let us deploy a simple application to prove that the cluster works end to end. We will deploy a whoami web server using a Deployment with three replicas and expose it via a NodePort Service. The pod anti-affinity rule encourages the scheduler to spread the pods across different nodes.

cat << EOF | kubectl --kubeconfig=capi-quickstart.kubeconfig apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30003
  type: NodePort
EOF

Check that the deployment, pods, service, and endpoints are all in good shape.

k8s1 get deploy,pod,svc,ep -owide

You should see three pods running, spread across your worker nodes, with the service exposing NodePort 30003. To actually test the connectivity, we can use the management cluster’s KinD node to reach into one of the workload cluster nodes on that NodePort. First, pick one of the control plane container names from docker ps, then curl it.

docker ps
CT1=capi-quickstart-hvt8l-hb4hb

docker exec -it myk8s-control-plane curl -s $CT1:30003

You can even run a loop to see load balancing in action across the three pods.

while true; do docker exec -it myk8s-control-plane curl -s $CT1:30003 | grep Hostname; date; sleep 1; done

Each request should return a different Hostname value as it hits different pods behind the service.

Since the workload cluster has a HAProxy load balancer in front of the three control plane API servers, you can also inspect its stats page. Check the LB container’s port mappings first.

docker inspect capi-quickstart-lb | jq | grep -i hostport

If you see a HostPort like 55001 mapped to port 8404, you can open http://127.0.0.1:55001/stats in your browser to see the HAProxy statistics dashboard showing the health status of all three backend API servers.

Upgrading the Workload Cluster (v1.34 to v1.35)

One of the most powerful features of Cluster API is declarative cluster lifecycle management. Upgrading a cluster from one Kubernetes version to another is as simple as changing a single field in the Cluster resource. The controllers handle all the complexity of rolling out new nodes, draining old ones, and ensuring the cluster remains available throughout the process.

Before starting the upgrade, open several terminal windows for monitoring so you can watch the process unfold in real time.

watch -d kubectl --kubeconfig=capi-quickstart.kubeconfig get node
watch -d "docker ps ; echo ; clusterctl describe cluster capi-quickstart"
kubectl --kubeconfig=capi-quickstart.kubeconfig get node -w

If you still have kube-ops-view running, keep an eye on that browser tab as well. It is quite satisfying to watch nodes appear and disappear during the rolling upgrade.

Now, because we used ClusterClass and Topology when we created the cluster, the upgrade is done by patching the Cluster resource’s topology version field. You do not touch the KubeadmControlPlane or MachineDeployment directly; the topology controller takes care of propagating the version change to all the right places.

kubectl patch cluster capi-quickstart --type merge -p '{"spec":{"topology":{"version":"v1.35.0"}}}'

The moment you run this command, the controllers spring into action. The upgrade proceeds in a careful, ordered fashion. First, the control plane nodes are upgraded one at a time. A new control plane node running v1.35.0 is created, it joins the cluster and the etcd ring, and then one of the old v1.34.3 control plane nodes is cordoned, drained, and removed. This repeats until all three control plane nodes have been replaced. Only after the entire control plane is upgraded does the controller move on to the worker nodes, following the same rolling replacement strategy.

You can watch the nodes cycling through by repeatedly checking the node list.

k8s1 get node

During the upgrade, you will see a mix of old and new nodes. At some point, you might see output like this, where the last node is being drained and removed.

NAME                                      STATUS                     ROLES           VERSION
capi-quickstart-74kjs-ncktx               Ready                      control-plane   v1.35.0
capi-quickstart-74kjs-npnjr               Ready                      control-plane   v1.35.0
capi-quickstart-74kjs-tdd8d               Ready                      control-plane   v1.35.0
capi-quickstart-md-0-npmnz-x7b2z-892h8   Ready                      <none>          v1.35.0
capi-quickstart-md-0-npmnz-x7b2z-fszrs   Ready                      <none>          v1.35.0
capi-quickstart-md-0-npmnz-x7b2z-lksqj   Ready                      <none>          v1.35.0
capi-quickstart-md-0-npmnz-x9jx6-qx8rf   Ready,SchedulingDisabled   <none>          v1.34.3

Notice the last worker node shows SchedulingDisabled. It has been cordoned and is being drained before removal.

If you check docker ps during the upgrade, you will see new containers being created with the kindest/node:v1.35.0 image while the old v1.34.3 containers are gradually removed.

docker ps

CONTAINER ID   IMAGE                                NAMES
9a79e524c198   kindest/node:v1.35.0                 capi-quickstart-md-0-npmnz-x7b2z-lksqj
f5658e9d5385   kindest/node:v1.35.0                 capi-quickstart-md-0-npmnz-x7b2z-fszrs
1a71ef5e651a   kindest/node:v1.35.0                 capi-quickstart-md-0-npmnz-x7b2z-892h8
3baae4af04d4   kindest/node:v1.35.0                 capi-quickstart-74kjs-tdd8d
5e85d57bd948   kindest/node:v1.35.0                 capi-quickstart-74kjs-npnjr
e7742cd044b3   kindest/node:v1.35.0                 capi-quickstart-74kjs-ncktx
c125003ec5d7   kindest/haproxy:v20230606-42a2262b   capi-quickstart-lb
6419a7cf30ce   kindest/node:v1.35.0                 myk8s-control-plane

Every container is now running v1.35.0. The HAProxy load balancer container remains untouched since it only proxies traffic and does not run Kubernetes itself.

You can also verify that the load balancer’s backend configuration has been updated to point to the new control plane nodes by copying the HAProxy config out of the container.

docker cp capi-quickstart-lb:/usr/local/etc/haproxy/haproxy.cfg .
cat haproxy.cfg | grep capi-quickstart

The backend server entries should now reference the new control plane containers with their new internal IP addresses.

Once the upgrade completes, all nodes will be running v1.35.0 and the cluster is fully healthy again. The entire process was triggered by a single kubectl patch command. There was no manual SSH-ing into nodes, no running kubeadm upgrade on each machine, no worrying about the order of operations. Cluster API handled everything declaratively.

This is the real power of Cluster API. Whether you are managing two clusters or two hundred, the workflow is the same: define the desired state, apply it, and let the controllers do the work. Combined with GitOps tools, you can version-control your entire cluster fleet and roll out upgrades with a pull request.

Cleanup

When you’re done experimenting with everything, it’s important to tear down all the resources properly so nothing lingers on your machine. We have two separate environments to clean up here: the Cluster API lab with its KinD-based management and workload clusters, and the RKE2 lab with its Vagrant-based virtual machines. Let’s walk through each one carefully.

Starting with the Cluster API side, the first thing to remove is the workload cluster that was provisioned through Cluster API. This is one of the most elegant parts of the entire Cluster API experience. Because the workload cluster is represented as a single Cluster custom resource in the management cluster, deleting it triggers the full reconciliation loop in reverse. The controller will drain the nodes, delete the Machine objects, tear down the Docker containers that were acting as virtual machines, and remove the load balancer. You can chain the delete command with docker ps to immediately see the effect.

kubectl delete cluster capi-quickstart && docker ps

The output should look like this. Notice how all the workload cluster containers have disappeared, leaving only the original KinD management cluster container running.

cluster.cluster.x-k8s.io "capi-quickstart" deleted
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                                                             NAMES
037b1d199fb7   kindest/node:v1.35.0   "/usr/local/bin/entr…"   57 minutes ago   Up 57 minutes   0.0.0.0:30000-30001->30000-30001/tcp, 127.0.0.1:57103->6443/tcp   myk8s-control-plane

Before this command, you had quite a few containers running. Three containers for control plane nodes, three for worker nodes, one HAProxy load balancer container, and the management cluster container itself. That’s eight containers in total. After the delete, only the management cluster’s KinD container survives. Everything else was cleaned up automatically by the Cluster API controllers. This is one of the most powerful aspects of the declarative model. You don’t have to manually SSH into machines, run drain commands, or stop services one by one. The controller handles the entire teardown sequence for you.

You should also clean up the kubeconfig file and any other artifacts that were generated during the lab. The capi-quickstart.kubeconfig file and the capi-quickstart.yaml manifest file are no longer needed.

rm -f capi-quickstart.kubeconfig capi-quickstart.yaml

Now that the workload cluster is completely gone, you can safely remove the KinD management cluster itself. This will destroy the last remaining Docker container and also clean up the kubeconfig context that KinD registered on your host machine.

kind delete cluster --name myk8s

The output confirms the deletion.

Deleting cluster "myk8s" ...
Deleted nodes: ["myk8s-control-plane"]

At this point, if you run docker ps, you should see no containers at all. Your Docker environment is back to a completely clean state. If you also installed clusterctl via Homebrew on macOS and no longer need it, you can optionally remove it as well.

brew uninstall clusterctl

You can also clean up the working directory you created at the beginning of the Cluster API lab.

cd ~
rm -rf capi-docker

That takes care of the entire Cluster API environment. Now let’s move on to the RKE2 lab.

The RKE2 lab was built on top of Vagrant with two virtual machines, k8s-node1 acting as the server (control plane) and k8s-node2 acting as the agent (worker node). Since all the RKE2 components, the systemd services, the binaries, the etcd data, the certificates, and everything else live entirely inside those VMs, the cleanest way to remove everything is to simply destroy the VMs through Vagrant. Navigate back to the directory where your Vagrantfile lives and run the destroy command with the force flag so it doesn’t prompt you for confirmation on each VM.

cd ~/k8s-rke2
vagrant destroy -f

Vagrant will stop both VMs and delete their disk images. You can confirm they’re gone by checking the status.

vagrant status

Both k8s-node1 and k8s-node2 should show as “not created,” which means they have been fully removed from your system.

If you want to clean up the Vagrantfile and init script as well since you won’t need them anymore, you can remove the entire working directory.

cd ~
rm -rf k8s-rke2

Now, there is one more thing worth mentioning for situations where you might want to remove RKE2 from a node without destroying the VM itself. Perhaps you’re working in a non-Vagrant environment, or you want to repurpose the machine for something else. RKE2 actually ships with uninstall scripts that get placed on the system during installation. On a server node, you would run the server uninstall script, and on an agent node, you would run the agent version. These scripts are thorough. They stop the rke2 services, kill any remaining containerd processes, remove the binaries from the data directory, delete the systemd unit files, clean up the yum repository configurations that were added during install, and wipe the data directories under /var/lib/rancher and /etc/rancher.

For a server node, the command would be:

/usr/bin/rke2-uninstall.sh

And for an agent node:

/usr/bin/rke2-agent-uninstall.sh

After running the appropriate script, the node is returned to a pre-RKE2 state. No Kubernetes components, no containerd, no certificates, and no leftover configuration. This is handy in production scenarios where you need to decommission a node or reinstall RKE2 from scratch without rebuilding the entire OS.

You should also remember to clean up the symbolic links we created during the RKE2 setup if you went the uninstall script route instead of destroying the VM. During the lab, we created several symlinks in /usr/local/bin to expose the RKE2 bundled binaries like kubectl, crictl, containerd, runc, and ctr to the system PATH. The uninstall script may or may not remove these depending on the version, so it’s good practice to verify.

rm -f /usr/local/bin/kubectl
rm -f /usr/local/bin/crictl
rm -f /usr/local/bin/containerd
rm -f /usr/local/bin/runc
rm -f /usr/local/bin/ctr
rm -f /etc/crictl.yaml

And the shell profile customizations we added for kubectl completion and the k alias:

sed -i '/source <(kubectl completion bash)/d' /etc/profile
sed -i '/alias k=kubectl/d' /etc/profile
sed -i '/complete -F __start_kubectl k/d' /etc/profile

Again, none of this manual cleanup is necessary if you’re simply destroying the Vagrant VMs, which is by far the simplest and most complete approach for a lab environment. But it’s good to know these details for real-world scenarios where the underlying machines are long-lived and managed independently of the Kubernetes installation.

For the system upgrade controller that we installed during the automated upgrade section, that was deployed inside the RKE2 cluster itself, so it gets destroyed along with the cluster when the VMs go away. But if you were running this on a persistent cluster and wanted to remove just the upgrade controller, you would delete the resources in reverse order: first the Plan custom resources, then the controller deployment and its associated RBAC and namespace.

kubectl delete plan -n system-upgrade server-plan agent-plan
kubectl delete -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
kubectl delete -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml

This removes the system-upgrade namespace, the controller deployment, the ClusterRole and ClusterRoleBinding resources, the ServiceAccount, and the Plan CRD itself. Your cluster continues to run normally at whatever version it was last upgraded to, and no further automatic upgrades will be attempted.

At this point, both lab environments are fully cleaned up. Your host machine should have no leftover containers, no lingering VMs, and no stale configuration files. You’re ready to start fresh whenever you want to revisit these exercises or move on to new experiments.

Wrap Up & Key Takeaways

So let's wrap up everything we covered in this hands-on journey through RKE2 and Cluster API. This was a dense session, so it's worth taking a moment to reflect on what we actually did, what we learned, and why it matters.

We started with RKE2, Rancher's security-focused Kubernetes distribution. The very first thing you probably noticed is how simple the installation process is. A single curl command followed by running the install script is all it takes to get the binary on your machine.

curl -sfL https://get.rke2.io --output install.sh
chmod +x install.sh
INSTALL_RKE2_CHANNEL=v1.33 ./install.sh

From there, we wrote a configuration file and started the systemd service. That's it. No manually bootstrapping etcd, no setting up certificates by hand, no juggling multiple binaries. RKE2 handles all of that internally through its supervisor process. The control plane components like etcd, kube-apiserver, kube-controller-manager, and kube-scheduler all come up as static pods, managed by the kubelet that RKE2 starts internally. You can confirm this after startup by checking the running pods in the kube-system namespace.

kubectl get pod -A

You will see etcd, kube-apiserver, kube-controller-manager, kube-scheduler, and kube-proxy all running as pods, alongside the Helm-installed components like Canal (the default CNI), CoreDNS, and the metrics server. This is a key architectural point. RKE2 does not install these components using raw manifests or direct binary execution on the host. Instead, it uses a combination of static pod manifests and Helm charts, giving you a consistent and manageable deployment model.

Adding a worker node was equally straightforward. On the agent node, you install the RKE2 agent binary, point it at the server's registration endpoint on port 9345, and provide the node token that the server generated during its initial startup.

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_CHANNEL=v1.33 sh -

mkdir -p /etc/rancher/rke2/
cat << EOF > /etc/rancher/rke2/config.yaml
server: https://192.168.10.11:9345
token: K10cfbf1f601080e27248e795b54de68ea18961910d639be08257095a7109e0dbf0::server:5add6b365458d11cc8a0164c005fc749
EOF

systemctl enable --now rke2-agent.service

Once the agent joined, we verified it from the control plane and noticed something interesting. The control plane node had no taints applied by default. This means pods can be scheduled on the control plane node as well, which is different from what you might expect if you are coming from a kubeadm-based setup where control plane nodes typically have a NoSchedule taint. We confirmed this directly.

kubectl describe node k8s-node1 | grep -i taints
Taints:             <none>

This is a design choice in RKE2 that you should be aware of. In production, you may want to add taints to your control plane nodes manually if you want to keep workloads off them.

Moving on to Day-2 operations, we explored certificate management. RKE2 issues client and server certificates that are valid for 365 days. When a certificate is within 120 days of expiration, RKE2 automatically renews it on the next restart. You can inspect all certificates and their expiration dates at any time using the built-in certificate check command.

rke2 certificate check --output table

This gives you a clear table showing every certificate file, its subject, usage type, expiration date, residual time, and current status. For manual rotation, the process is stop the service, rotate, then start the service again.

Introduction

Most Kubernetes tutorials assume one thing: your nodes can reach the internet. Pull an image from Docker Hub, download a binary from GitHub, install a package from the upstream repo — it all just works. Until it doesn’t.

In regulated industries — finance, defense, healthcare, government — production clusters often sit behind an air gap. No outbound internet access, period. Every container image, every OS package, every Python dependency has to be staged internally before a single pod can run. If you’ve never dealt with this constraint, it might sound like a minor inconvenience. In practice, it changes almost everything about how you plan and execute a Kubernetes deployment.

I went through this process recently as part of the CloudNet@ community’s K8S Deploy study group (Week 6), and this post documents the full journey: standing up the internal infrastructure (DNS, NTP, package mirrors, a private container registry), using the kubespray-offline tooling to pre-download all required artifacts, and finally running Kubespray’s Ansible playbook to bring up a working cluster — all without a single packet leaving the private network.

The lab environment is modest — one admin server and two k8s nodes running Rocky Linux 10 on VirtualBox — but the workflow maps directly to real-world air-gapped deployments. By the end, you’ll have a clear picture of what it takes to go from a blank, isolated network to a functioning Kubernetes cluster with Flannel networking, a private image registry, Helm OCI support, and internal package repositories.

Architecture & Lab Setup

Network Topology Overview

The lab mimics a typical air-gapped deployment pattern. Three virtual machines sit on a private network (192.168.10.0/24), and only one of them — the admin server — has a path to the outside world.

┌─────────────────────────────────────────────────┐
│                 Private Network                  │
│               192.168.10.0/24                    │
│                                                  │
│   ┌───────────┐                                  │
│   │   admin    │──── enp0s8 ───► Internet        │
│   │  .10.10   │     (NAT GW)                    │
│   │           │                                  │
│   │  - DNS Server (BIND, :53)                   │
│   │  - NTP Server (chrony)                      │
│   │  - Container Registry (:35000)              │
│   │  - Nginx File Server (:80)                  │
│   │  - YUM/DNF Repo Mirror                      │
│   │  - PyPI Mirror                              │
│   │  - NAT Gateway                              │
│   └─────┬─────┘                                  │
│         │ enp0s9                                 │
│         │                                        │
│   ┌─────┴─────┐     ┌───────────┐               │
│   │ k8s-node1 │     │ k8s-node2 │               │
│   │  .10.11   │     │  .10.12   │               │
│   │  (master) │     │  (worker) │               │
│   └───────────┘     └───────────┘               │
│     enp0s8 DOWN       enp0s8 DOWN               │
│     (no internet)     (no internet)              │
└─────────────────────────────────────────────────┘

The admin server wears many hats. In a production air-gapped environment, these responsibilities would typically be split across dedicated infrastructure — a DNS team managing BIND or CoreDNS, a networking team handling the NAT gateway, a platform team running Harbor for container images.

For this lab, one server does it all:

• DNS Server (BIND) — resolves internal hostnames and forwards external domain queries
• NTP Server (chrony) — keeps cluster time synchronized across all nodes
• NAT Gateway — routes internal traffic to the internet through enp0s8 during the preparation phase, then gets disabled to simulate a true air gap
• Container Registry (Docker Registry) — serves all Kubernetes container images on port 35000
• Nginx File Server — hosts Kubernetes binaries, RPM packages, and Python packages over HTTP on port 80
• YUM/DNF Repository Mirror — a full mirror of Rocky Linux BaseOS, AppStream, and Extras repos
• PyPI Mirror — offline Python package index for Kubespray’s Ansible dependencies

The two k8s nodes have their external-facing enp0s8 interfaces disabled entirely. Their only network path is enp0s9, pointing at 192.168.10.10 as the default gateway. Even DNS queries go through the admin server. These nodes are, for all practical purposes, fully isolated from the internet.

Vagrant Environment

The whole setup runs on VirtualBox, managed by Vagrant. Each VM gets Rocky Linux 10, 4 vCPUs, and 2 GB of RAM. The admin server has one important difference: its primary disk is expanded to 120 GB, because it needs to hold all the mirrored packages, container images, and binaries — easily exceeding 20 GB in total.

BOX_IMAGE = "bento/rockylinux-10.0"
BOX_VERSION = "202510.26.0"
N = 2

Vagrant.configure("2") do |config|

  # k8s nodes
  (1..N).each do |i|
    config.vm.define "k8s-node#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/Kubespary-offline-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-node#{i}"
        vb.cpus = 4
        vb.memory = 2048
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-node#{i}"
      subconfig.vm.network "private_network", ip: "192.168.10.1#{i}"
      subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}",
                           auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "init_cfg.sh", args: [N]
    end
  end

  # Admin server
  config.vm.define "admin" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/Kubespary-offline-Lab"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "admin"
      vb.cpus = 4
      vb.memory = 2048
      vb.linked_clone = true
    end
    subconfig.vm.host_name = "admin"
    subconfig.vm.network "private_network", ip: "192.168.10.10"
    subconfig.vm.network "forwarded_port", guest: 22, host: "60000",
                         auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.disk :disk, size: "120GB", primary: true
    subconfig.vm.provision "shell", path: "admin.sh", args: [N]
  end
end

The linked_clone = true setting keeps disk usage manageable by sharing a base image across VMs rather than fully duplicating it. Each k8s node provisions itself through init_cfg.sh, while the admin server runs a separate admin.sh script with additional setup.

Provisioning Scripts

admin.sh handles the admin server’s initial configuration — timezone, firewall/SELinux, IP forwarding, SSH key distribution to all nodes, Helm installation, and disk expansion:

echo ">>>> Initial Config Start <<<<"

echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul

echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

echo "[TASK 3] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
echo "192.168.10.10 admin" >> /etc/hosts
for (( i=1; i<=$1; i++ )); do
  echo "192.168.10.1$i k8s-node$i" >> /etc/hosts
done

echo "[TASK 4] Delete default routing - enp0s9 NIC"
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9 >/dev/null 2>&1

echo "[TASK 5] Config net.ipv4.ip_forward"
cat << EOF > /etc/sysctl.d/99-ipforward.conf
net.ipv4.ip_forward = 1
EOF
sysctl --system >/dev/null 2>&1

echo "[TASK 6] Install packages"
dnf install -y python3-pip git sshpass cloud-utils-growpart >/dev/null 2>&1

echo "[TASK 7] Install Helm"
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \
  | DESIRED_VERSION=v3.20.0 bash >/dev/null 2>&1

echo "[TASK 8] Increase Disk Size"
growpart /dev/sda 3 >/dev/null 2>&1
xfs_growfs /dev/sda3 >/dev/null 2>&1

echo "[TASK 9] Setting SSHD"
echo "root:qwe123" | chpasswd
cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1

echo "[TASK 10] Setting SSH Key"
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa >/dev/null 2>&1
sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no \
  root@192.168.10.10 >/dev/null 2>&1
for (( i=1; i<=$1; i++ )); do
  sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no \
    root@192.168.10.1$i >/dev/null 2>&1
done
for (( i=1; i<=$1; i++ )); do
  ssh -o StrictHostKeyChecking=no root@k8s-node$i hostname >/dev/null 2>&1
done

echo "[TASK 11] Install K9s"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget -P /tmp https://github.com/derailed/k9s/releases/latest/download/k9s_linux_${CLI_ARCH}.tar.gz >/dev/null 2>&1
tar -xzf /tmp/k9s_linux_${CLI_ARCH}.tar.gz -C /tmp
chown root:root /tmp/k9s
mv /tmp/k9s /usr/local/bin/
chmod +x /usr/local/bin/k9s

echo "[TASK 12] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc

echo ">>>> Initial Config End <<<<"

Task 5 is particularly important — net.ipv4.ip_forward = 1 enables the admin server to function as a NAT gateway later, forwarding packets from the isolated k8s nodes to the internet. Task 8 uses growpart and xfs_growfs to expand the disk from the default size to the 120 GB specified in the Vagrantfile. Task 10 distributes SSH keys to all nodes, which is essential for Ansible (and therefore Kubespray) to work without password prompts.

init_cfg.sh runs on each k8s node and handles Kubernetes-specific prerequisites:

echo ">>>> Initial Config Start <<<<"

echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul

echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

echo "[TASK 3] Disable and turn off SWAP & Delete swap partitions"
swapoff -a
sed -i '/swap/d' /etc/fstab
sfdisk --delete /dev/sda 2 >/dev/null 2>&1
partprobe /dev/sda >/dev/null 2>&1

echo "[TASK 4] Config kernel & module"
cat << EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
vxlan
EOF
modprobe overlay >/dev/null 2>&1
modprobe br_netfilter >/dev/null 2>&1

cat << EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system >/dev/null 2>&1

echo "[TASK 5] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
echo "192.168.10.10 admin" >> /etc/hosts
for (( i=1; i<=$1; i++ )); do
  echo "192.168.10.1$i k8s-node$i" >> /etc/hosts
done

echo "[TASK 6] Delete default routing - enp0s9 NIC"
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9 >/dev/null 2>&1

echo "[TASK 7] Setting SSHD"
echo "root:qwe123" | chpasswd
cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1

echo "[TASK 8] Install packages"
dnf install -y python3-pip git >/dev/null 2>&1

echo "[TASK 9] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc

echo ">>>> Initial Config End <<<<"

Task 3 is a hard Kubernetes requirement — kubelet refuses to start if swap is enabled, so the script disables it, removes the fstab entry, and deletes the swap partition entirely. Task 4 loads three kernel modules that Kubernetes networking depends on: overlay for the container filesystem, br_netfilter so bridge traffic passes through iptables (required by kube-proxy and most CNI plugins), and vxlan for overlay network tunneling. The sysctl parameters in the same task ensure that bridged IPv4/IPv6 traffic gets processed by iptables and that IP forwarding is active.

Bringing It Up

mkdir k8s-offline && cd k8s-offline

curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-kubespary-offline/Vagrantfile
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-kubespary-offline/admin.sh
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-kubespary-offline/init_cfg.sh

vagrant up
vagrant status

Once all three VMs are running, SSH into each one:

ssh root@192.168.10.10   # admin  (password: qwe123)
ssh root@192.168.10.11   # k8s-node1
ssh root@192.168.10.12   # k8s-node2

At this point the VMs are up, the network is in place, and the k8s nodes still have internet access through their enp0s8 interfaces. The admin server has already distributed SSH keys to every node during provisioning, so Ansible can reach them without password prompts. The next step is to cut off that external access and start building the internal services that will replace it.

Core Infrastructure Services

With the VMs running, the next step is cutting off the k8s nodes from the internet and building the internal services they’ll depend on instead. Three pieces need to be in place before anything Kubernetes-related can happen: a network gateway with NAT, time synchronization via NTP, and DNS resolution.

Network Gateway & NAT

The goal here is simple: make it so the k8s nodes have zero direct internet access, and their only path to the outside world (when needed) runs through the admin server.

This involves two things — disabling the external interface on each node and pointing their default route at the admin server.

On each k8s node, bring down enp0s8 (the NAT-attached interface that Vagrant uses for internet access) and set enp0s9 (the private network interface) as the sole network path:

# Disable external interface immediately
nmcli connection down enp0s8
nmcli connection modify enp0s8 connection.autoconnect no

# Add default route through admin server (priority 200)
nmcli connection modify enp0s9 +ipv4.routes "0.0.0.0/0 192.168.10.10 200"
nmcli connection up enp0s9

You can verify the routing table looks correct:

ip route

On k8s-node1:

default via 192.168.10.10 dev enp0s9 proto static metric 200
192.168.10.0/24 dev enp0s9 proto kernel scope link src 192.168.10.11 metric 100

On k8s-node2:

default via 192.168.10.10 dev enp0s9 proto static metric 200
192.168.10.0/24 dev enp0s9 proto kernel scope link src 192.168.10.12 metric 100

The autoconnect no setting persists across reboots. You can confirm it by checking the connection file:

cat /etc/NetworkManager/system-connections/enp0s8.nmconnection

[connection]
id=enp0s8
uuid=7f94e839-e070-4bfe-9330-07090381d89f
type=ethernet
autoconnect=false
...

The enp0s9 connection file now carries the new route:

[ipv4]
address1=192.168.10.11/24
method=manual
never-default=true
route1=0.0.0.0/0,192.168.10.10

At this point, the nodes can’t reach anything outside the private network. A quick test confirms it:

ping -c 1 -w 1 -W 1 8.8.8.8
# 1 packets transmitted, 0 received, 100% packet loss

curl www.google.com
# curl: (6) Could not resolve host: www.google.com

Even manually setting DNS servers doesn’t help — the packets simply have nowhere to go without a NAT gateway:

cat << EOF > /etc/resolv.conf
nameserver 168.126.63.1
nameserver 8.8.8.8
EOF

curl www.google.com
# still fails — no route to the internet

On the admin server, enable NAT so internal traffic can reach the internet through enp0s8. IP forwarding was already enabled by admin.sh during provisioning, but you can verify and set it explicitly:

sysctl -w net.ipv4.ip_forward=1

cat <<EOF > /etc/sysctl.d/99-ipforward.conf
net.ipv4.ip_forward = 1
EOF
sysctl --system

Now add the MASQUERADE rule:

iptables -t nat -A POSTROUTING -o enp0s8 -j MASQUERADE

Verify it:

iptables -t nat -L -n -v

Chain POSTROUTING (policy ACCEPT 1 packets, 120 bytes)
 pkts bytes target     prot opt in     out     source               destination
    2   168 MASQUERADE  all  --  *      enp0s8  0.0.0.0/0            0.0.0.0/0

Go back to any k8s node and test — external connectivity should now work through the admin server’s NAT.

The important part comes next. Remove the NAT rule to simulate the actual air-gapped state:

iptables -t nat -D POSTROUTING -o enp0s8 -j MASQUERADE

Back on the k8s nodes, internet access is gone again. This toggle — NAT on during preparation, NAT off during installation — is how we prove the cluster deployment works entirely offline. For the rest of this guide, the NAT rule stays off unless explicitly noted otherwise.

NTP Server & Client (chrony)

Kubernetes components are sensitive to clock skew. Certificate validation, etcd leader election, log correlation — all of these break or behave unpredictably when nodes disagree about what time it is. In an air-gapped environment, the nodes can’t reach public NTP pools, so the admin server needs to act as the internal time source.

On the admin server, start by checking the current chrony status and configuration:

systemctl status chronyd.service --no-pager
grep "^[^#]" /etc/chrony.conf

The default Rocky Linux configuration looks like this:

pool 2.rocky.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony

A few of these settings are worth understanding. The iburst option sends a burst of packets right after startup to synchronize quickly instead of waiting for the normal polling interval.

The driftfile records how much the local hardware clock drifts from real time — chrony uses this to compensate even when the network is unavailable. The makestep 1.0 3 directive tells chrony to force an immediate time jump (rather than gradual slewing) if the offset exceeds 1 second, but only during the first 3 updates after startup.

And rtcsync periodically copies the system clock to the hardware RTC, so the time stays reasonably accurate across reboots.

You can check which NTP sources chrony is currently using:

chronyc sources -v
dig +short 2.rocky.pool.ntp.org

Now replace the configuration with one suited for our air-gapped setup:

cp /etc/chrony.conf /etc/chrony.bak

cat << EOF > /etc/chrony.conf
# External NTP servers (used during preparation phase)
server pool.ntp.org iburst
server kr.pool.ntp.org iburst

# Allow internal network clients to sync from this server
allow 192.168.10.0/24

# If external servers become unreachable, serve local time
local stratum 10

logdir /var/log/chrony
EOF

systemctl restart chronyd.service

The local stratum 10 line is critical for air-gapped operation. Without it, chrony would stop serving time to clients once it loses contact with upstream servers. With this setting, the admin server falls back to its own clock at stratum 10 — not perfectly accurate, but good enough to keep the cluster running.

Verify the configuration:

timedatectl status

Local time: Wed 2026-02-11 22:36:16 KST
           Universal time: Wed 2026-02-11 13:36:16 UTC
                 RTC time: Wed 2026-02-11 14:09:31
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

chronyc sources -v

On each k8s node, point chrony at the admin server:

cp /etc/chrony.conf /etc/chrony.bak

cat << EOF > /etc/chrony.conf
server 192.168.10.10 iburst
logdir /var/log/chrony
EOF

systemctl restart chronyd.service

Check that synchronization is working:

timedatectl status

Local time: Wed 2026-02-11 22:38:19 KST
           Universal time: Wed 2026-02-11 13:38:19 UTC
                 RTC time: Wed 2026-02-11 14:11:34
                Time zone: Asia/Seoul (KST, +0900)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

chronyc sources -v

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* admin                         0   7     0     -     +0ns[   +0ns] +/-    0ns

The ^* prefix indicates this source is currently selected and synchronized. Back on the admin server, you can confirm both nodes are connecting:

chronyc clients

Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
===============================================================================
k8s-node1                       3      0   1   -     1       0      0   -     -
k8s-node2                       2      0   1   -     0       0      0   -     -

Both nodes are syncing their clocks through the admin server. Even after the NAT rule is removed and the admin server itself can’t reach public NTP pools, the local stratum 10 fallback keeps time flowing to the cluster.

DNS Server & Client (BIND)

The k8s nodes need to resolve hostnames — both internal names like admin and k8s-node1, and external names like registry.k8s.io (which appear in container image references even if the actual pull happens from the private registry). BIND on the admin server handles both: it resolves internal queries directly and forwards everything else to upstream DNS servers (when the NAT is active) or returns cached results (when offline).

On the admin server, install BIND:

dnf install -y bind bind-utils

Write the full /etc/named.conf:

cp /etc/named.conf /etc/named.bak

cat <<EOF > /etc/named.conf
options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        secroots-file   "/var/named/data/named.secroots";
        recursing-file  "/var/named/data/named.recursing";
        allow-query     { 127.0.0.1; 192.168.10.0/24; };
        allow-recursion { 127.0.0.1; 192.168.10.0/24; };

        forwarders {
                168.126.63.1;
                8.8.8.8;
        };

        recursion yes;

        dnssec-validation auto;

        managed-keys-directory "/var/named/dynamic";
        geoip-directory "/usr/share/GeoIP";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";

        include "/etc/crypto-policies/back-ends/bind.config";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
EOF

The key options here: listen-on port 53 { any; } makes BIND listen on all interfaces, so the k8s nodes can reach it. allow-query and allow-recursion restrict access to localhost and the private network. The forwarders block points to KT's public DNS (168.126.63.1) and Google's (8.8.8.8) — these handle any external domain lookups when the NAT gateway is active.

The dnssec-validation auto setting enables DNSSEC validation; if you run into resolution failures related to DNSSEC, you might need to set this to no as a workaround.

Validate and start the service:

named-checkconf /etc/named.conf   # no output means no errors
systemctl enable --now named

Set the admin server itself to use its own DNS:

echo "nameserver 192.168.10.10" > /etc/resolv.conf

Test it:

dig +short google.com @192.168.10.10
# 142.250.183.110

dig +short google.com
# 142.250.183.110

On each k8s node, there’s an extra step needed. NetworkManager likes to overwrite /etc/resolv.conf on every connection change or reboot. To prevent that:

cat << EOF > /etc/NetworkManager/conf.d/99-dns-none.conf
[main]
dns=none
EOF

systemctl restart NetworkManager

Now set the DNS to point at the admin server:

echo "nameserver 192.168.10.10" > /etc/resolv.conf

Test DNS resolution — and this is the interesting part. Even though the k8s nodes have no internet access (the NAT rule is off), DNS queries still work because BIND on the admin server forwards them through its own enp0s8 interface:

dig +short google.com @192.168.10.10
# 142.250.183.110

dig +short google.com
# 142.250.183.110

The query reaches the admin server’s BIND instance, which forwards it to the upstream DNS servers through its internet-connected interface, and returns the answer to the k8s node. The k8s node itself never touches the internet directly. Once the admin server’s NAT is also removed (for the fully air-gapped installation phase), BIND serves cached results for any domains it has previously resolved. New external domains won’t resolve, but that’s fine — by that point, everything the cluster needs is already available locally.

With NAT, NTP, and DNS in place, the admin server is ready to take on its next set of responsibilities: hosting the package repositories, container images, and Python packages that Kubespray will need during the offline installation.

Offline Repositories

Before touching kubespray-offline, it helps to understand the three categories of dependencies that Kubernetes needs — and how to serve each of them internally. The admin server has internet access during the preparation phase, so we use that window to mirror everything into local repositories. Once the mirror is ready and validated, the external route can be removed entirely.

kubespray-offline will automate most of this in a later step (Section 5), but knowing how each piece works makes troubleshooting far easier when something inevitably goes wrong.

Local YUM/DNF Mirror

Kubernetes nodes need OS-level packages — container runtimes, SELinux policy modules, socat, conntrack, ipvsadm, and many others. In an air-gapped network, dnf install has nowhere to go unless you provide a local repository.

On the admin server, dnf reposync pulls the entire upstream repository tree down to a local directory:

dnf reposync --repoid=baseos --download-metadata -p /data/repos/rocky/10
dnf reposync --repoid=appstream --download-metadata -p /data/repos/rocky/10
dnf reposync --repoid=extras --download-metadata -p /data/repos/rocky/10

This took about 12 minutes in the lab. The baseos repo came in at 6.2 GB, appstream at 14 GB, and extras at a modest 67 MB — roughly 20 GB total. Once synced, nginx serves the directory over HTTP at http://192.168.10.10/rpms/rocky/10/.

On each k8s node, the default Rocky Linux repo files get replaced with a single internal-rocky.repo pointing all three channels to the admin server's URL. After that, dnf install and dnf update work exactly as they would on a connected system — the nodes just don't know (or care) that the packages are coming from a machine one hop away.

One thing to watch out for: reposync copies the RPMs, but you also need the repository metadata. The --download-metadata flag handles this, but if you ever re-sync and the metadata gets stale, run createrepo --update on the directory to regenerate it. Stale metadata leads to dependency resolution failures that look completely unrelated to the actual problem.

Private Container Registry

Every container image that the cluster needs — kube-apiserver, kube-proxy, coredns, flannel, pause, etcd, metrics-server, and more — must be available from an internal registry. In this lab, we run Docker Registry v3.0.0 on port 35000:

nerdctl run -d --name registry --network host \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:35000 \
  -v /var/lib/registry:/var/lib/registry \
  registry:3.0.0

The --network host flag avoids NAT overhead and keeps things simple for a single-node registry. The data directory at /var/lib/registry is mounted as a volume, so registry contents survive container restarts.

The workflow for populating it is straightforward: pull an image on a machine with internet access, tag it with the 192.168.10.10:35000 prefix, then push. For the 55 container images that Kubespray requires, this process gets automated by load-push-all-images.sh in the kubespray-offline toolchain — but the principle is the same whether you push one image or five hundred.

Since this is an HTTP registry (not HTTPS), every node’s containerd configuration must include it as an insecure registry. Miss this step and you’ll get TLS handshake errors that can be confusing if you’re not expecting them.

Private PyPI Mirror

This one catches people off guard. Kubespray is an Ansible project, and Ansible has Python dependencies — jinja2, PyYAML, netaddr, cryptography, and a handful of others. If the nodes can't reach pypi.org, pip install fails silently or throws cryptic SSL errors.

Two approaches work here. The first is devpi-server, which runs a full PyPI-compatible index at http://192.168.10.10:3141. It caches packages as they're requested, which is convenient but requires the server itself to have had internet access at least once to prime the cache. The second approach uses pypi-mirror to generate a static file-based index served directly by nginx at http://192.168.10.10/pypi/. This is simpler to reason about — it's just files on disk — and pairs naturally with the nginx instance that's already serving RPMs.

Either way, global pip configuration goes into /etc/pip.conf:

[global]
index-url = http://192.168.10.10/pypi/
trusted-host = 192.168.10.10

With this in place, every pip install on the network resolves against the internal mirror. The trusted-host line is necessary because we're serving over plain HTTP — without it, pip refuses the connection by default.

Worth noting: kubespray-offline’s pypi-mirror.sh script handles the package download during the preparation phase, and setup-offline.sh writes the pip configuration automatically during deployment. Understanding the manual setup here mostly pays off when you need to add Python packages later, after the initial deployment is done and you're operating the cluster day-to-day.

Kubespray-Offline: Download & Prepare

With the core infrastructure in place — NAT gateway, DNS, NTP, and local repos all running on the admin server — it’s time to tackle the big question: how do you get everything Kubernetes needs onto a machine that can’t reach the internet?

The answer is kubespray-offline. This project wraps around Kubespray (the popular Ansible-based Kubernetes installer) and adds a complete offline preparation layer. You run a single master script on an internet-connected machine, it pulls down every binary, container image, RPM package, and Python dependency that Kubespray will need, and packages them into a portable outputs/ directory. Copy that directory to your air-gapped admin server, run a few setup scripts, and you've got a fully self-contained deployment environment.

In my case, the entire download took about 17 minutes and produced roughly 3.3 GB of artifacts.

git clone https://github.com/kubespray-offline/kubespray-offline.git
cd kubespray-offline

How download-all.sh Works

The entry point is download-all.sh, and it's essentially a runner that calls 10 sub-scripts in sequence. Each one handles a specific category of dependencies. Here's what happens when you execute it:

cd /root/kubespray-offline
./download-all.sh

The first script to run is config.sh. This is where every version number lives — Kubespray itself, containerd, runc, the CNI plugins, nginx, the registry image, and so on. In my setup it looked like this:

# config.sh (key variables)
KUBESPRAY_VERSION=2.30.0
CONTAINERD_VERSION=2.2.1
RUNC_VERSION=1.3.4
CNI_VERSION=1.8.0
NGINX_VERSION=1.29.4
REGISTRY_VERSION=3.0.0

Getting these version numbers right matters more than you might expect. If config.sh says containerd 2.2.1 but you later tell Kubespray to use a different version in your inventory, the playbook will try to download a file that doesn't exist on your local nginx server — and in an air-gapped network, there's nowhere to fall back to.

After config.sh, the scripts run in this order:

precheck.sh validates that podman or docker is installed and checks SELinux status. Nothing fancy, but it catches the obvious "you forgot to install podman" mistake early.

prepare-pkgs.sh installs the system-level tools needed for the rest of the process: rsync, gcc, libffi-devel, createrepo, git, podman, and python3. These all come from the standard Rocky Linux repos, so this script needs internet access.

prepare-py.sh creates a Python virtual environment at ~/.venv/3.12 and installs the Ansible dependencies into it. Kubespray is fundamentally a collection of Ansible playbooks, so having the right Python environment is a prerequisite for everything that follows.

get-kubespray.sh downloads the kubespray-2.30.0.tar.gz release from GitHub and extracts it into the cache directory.

pypi-mirror.sh is where things get interesting for the offline story. It downloads every Python package that Kubespray's Ansible roles will need — pip, setuptools, ansible-core, PyYAML, jinja2, and dozens of others — into a local directory that will later be served as a static PyPI mirror.

download-kubespray-files.sh does the heaviest lifting. It runs Kubespray's own contrib/offline/generate_list.sh to produce two critical files: files.list and images.list. The files list contained 27 URLs pointing to binaries like kubectl, kubelet, kubeadm, etcd, containerd, crictl, and the CNI plugins. The images list had 55 container image references — everything from registry.k8s.io/kube-apiserver to docker.io/flannel/flannel to the CoreDNS and metrics-server images. The script then downloads every one of these.

# Generated lists (under outputs/)
outputs/files/files.list    # 27 binary URLs
outputs/images/images.list  # 55 container images

download-additional-containers.sh pulls two extra images that aren't part of Kubernetes itself but are needed for the offline infrastructure: nginx:1.29.4 (to serve files over HTTP) and registry:3.0.0 (the private container registry).

create-repo.sh builds a local RPM repository using createrepo and adds modulemd metadata for RHEL 8/9 compatibility. This repo gets bundled into the outputs directory so that target nodes can install OS packages without touching the internet.

Finally, copy-target-scripts.sh copies the deployment scripts into outputs/, making the whole directory self-contained and ready to transfer.

When download-all.sh finishes, the outputs/ directory has everything:

outputs/
├── files/          # Kubernetes binaries (kubectl, kubelet, etcd, containerd, etc.)
├── images/         # Container images as .tar.gz archives
├── rpms/           # RPM packages with repo metadata
├── pypi/           # Python packages for offline pip
├── kubespray-2.30.0/   # Kubespray source with playbooks
├── setup-container.sh
├── start-nginx.sh
├── start-registry.sh
├── load-push-all-images.sh
├── setup-offline.sh
└── setup-py.sh

At this point, you copy the entire outputs/ directory to your air-gapped admin server (via USB, SCP through a bastion, or whatever transfer method your security policy allows) and move on to deployment.

Deploying the Nginx File Server and Registry

On the air-gapped admin server, the deployment scripts in outputs/ are meant to be run in a specific order. Skipping a step or running them out of sequence will break things downstream.

The first script is setup-container.sh. It installs containerd, runc, nerdctl, and the CNI plugins from the local files — no package manager involved, just direct binary installation from the archives that were downloaded earlier. After this runs, you have a working container runtime on the admin server.

cd /root/kubespray-offline/outputs
./setup-container.sh

Next, start-nginx.sh launches an nginx container on port 80. This nginx instance serves the entire outputs/ directory over HTTP — the binaries in files/, the container image tarballs in images/, the Python packages in pypi/, and the RPM repository in rpms/. Any node on the 192.168.10.0/24 network can now fetch these artifacts from http://192.168.10.10/.

./start-nginx.sh
# nginx:1.29.4 now running on port 80
# Serves: files/, images/, pypi/, rpms/ over HTTP

Then start-registry.sh brings up the Docker Registry v3.0.0 container, listening on port 35000. At this point the registry is empty — it's just waiting for images to be pushed into it.

./start-registry.sh
# registry:3.0.0 now running on port 35000

One thing worth noting: the registry runs with --network host and binds directly to 0.0.0.0:35000, so there's no port mapping or container networking complexity to debug. If port 35000 is reachable, the registry is reachable.

Loading and Pushing Images

This is the step that takes the longest in the deployment phase, and it’s also the one most likely to reveal problems if something went wrong during the download phase.

load-push-all-images.sh iterates through every .tar.gz file in the images/ directory. For each image, it loads the archive into the local containerd image store, re-tags it with the 192.168.10.10:35000 prefix, and pushes it to the private registry.

./load-push-all-images.sh

Under the hood, each image goes through three operations:

# What happens for each image (conceptually):
nerdctl load -i images/kube-apiserver-v1.34.3.tar.gz
nerdctl tag registry.k8s.io/kube-apiserver:v1.34.3 192.168.10.10:35000/kube-apiserver:v1.34.3
nerdctl push 192.168.10.10:35000/kube-apiserver:v1.34.3 --insecure-registry

With 55 images to process, this took about 2–3 minutes in my lab. On slower hardware or with larger image sets, expect it to take longer.

Once the script finishes, you can verify everything landed correctly:

curl -s http://192.168.10.10:35000/v2/_catalog | python3 -m json.tool

This should return a JSON list of all 55+ repositories in the registry. If any image is missing here, Kubespray will fail when it tries to pull that image during cluster installation — and the error message won’t always make it obvious that the root cause is a missing image in your local registry.

After the images are pushed, two more setup scripts finalize the environment:

setup-offline.sh configures the admin server to use its own local repositories. It writes /etc/yum.repos.d/offline.repo pointing to the local RPM mirror, and creates ~/.config/pip/pip.conf pointing pip at the local PyPI mirror. From this point forward, even dnf install and pip install on the admin server itself go through the local copies.

setup-py.sh installs Python 3.12 from the offline RPM repository and prepares the Ansible environment. Kubespray's playbooks need a specific set of Python packages, and this script makes sure they're all available without any network calls.

./setup-offline.sh    # Configure local repos for dnf and pip
./setup-py.sh         # Install Python 3.12 + Ansible from offline packages

At this point, the admin server is fully self-sufficient. It has a file server, a container registry populated with every required image, local package repositories for both RPMs and Python packages, and a working Ansible installation. The cluster deployment can proceed entirely offline.

The whole deployment phase — from running setup-container.sh to having a populated registry and configured repos — took about 3 minutes in my lab environment. Most of that time was spent in load-push-all-images.sh. The other scripts finished in seconds.

Installing & Verifying the Cluster

At this point, the admin server is doing a lot of heavy lifting. It’s serving OS packages over HTTP, hosting a container registry on port 35000, mirroring PyPI, and running an nginx file server with all the Kubernetes binaries staged and ready. The k8s nodes can reach all of these services over the internal 192.168.10.0/24 network, and nothing else.

Now comes the part where all that preparation pays off.

Preparing the Nodes for Offline Installation

Before Kubespray touches the k8s nodes, each one needs a few things in place: a working container runtime, access to the offline package repo, and a pip configuration pointing at the internal mirror. The kubespray-offline project ships a set of scripts that handle this sequentially.

On the admin server, from the outputs directory:

cd /root/kubespray-offline/outputs

# Install containerd, runc, nerdctl, and CNI plugins from local files
./setup-container.sh

# Start nginx on port 80 — serves binaries, images, PyPI, and RPMs
./start-nginx.sh

# Start the private registry on port 35000
./start-registry.sh

# Load every .tar.gz image, re-tag with the 192.168.10.10:35000 prefix, and push
./load-push-all-images.sh

# Point yum and pip at the internal servers
./setup-offline.sh

# Install Python 3.12 from the offline repo
./setup-py.sh

The load-push-all-images.sh step takes the longest. It walks through every image archive in the outputs/images/ directory, loads each one into the local container runtime, re-tags it with the 192.168.10.10:35000 prefix, and pushes it to the private registry. For 55 images, expect this to take a couple of minutes.

After setup-offline.sh runs, you can verify that the nodes are pointed at the right repos:

# Check yum is configured for the internal mirror
cat /etc/yum.repos.d/offline.repo

# Check pip is configured for the internal PyPI
cat ~/.config/pip/pip.conf

Kubespray Configuration

Kubespray lives inside the outputs directory as a tarball that was extracted during the download phase. The inventory and group variables need a few edits before anything can run.

cd /root/kubespray-offline/outputs/kubespray-2.30.0

The Inventory File

The inventory defines which nodes play which roles. For this two-node setup, k8s-node1 serves as the control plane and etcd host, while k8s-node2 is a pure worker.

# inventory/mycluster/inventory.ini

[all]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12

[kube_control_plane]
k8s-node1

[etcd]
k8s-node1

[kube_node]
k8s-node1
k8s-node2

[k8s_cluster:children]
kube_control_plane
kube_node

Nothing unusual here — a minimal two-node topology. In production you’d want three etcd members and separate control plane nodes, but for validating the offline workflow, this is enough.

The offline.yml File

This is the file that makes the entire air-gap setup work. It tells Kubespray where to find every binary and container image, replacing all the default upstream URLs with internal equivalents.

# inventory/mycluster/group_vars/all/offline.yml

http_server: "http://192.168.10.10"
registry_host: "192.168.10.10:35000"

# Binaries
kubeadm_download_url: "{{ http_server }}/kubernetes/kubeadm-{{ kube_version }}-linux-{{ image_arch }}"
kubectl_download_url: "{{ http_server }}/kubernetes/kubectl-{{ kube_version }}-linux-{{ image_arch }}"
kubelet_download_url: "{{ http_server }}/kubernetes/kubelet-{{ kube_version }}-linux-{{ image_arch }}"

# This one bit me — the default has "linux-amd64" hardcoded.
# If you're on ARM64 or want portability, use the template variable.
etcd_download_url: "{{ http_server }}/kubernetes/etcd/etcd-v{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"

containerd_download_url: "{{ http_server }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
runc_download_url: "{{ http_server }}/runc.{{ image_arch }}"
cni_download_url: "{{ http_server }}/cni-plugins-linux-{{ image_arch }}-v{{ cni_version }}.tgz"

# Container images — all pulled from the private registry
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"

# Registry mirror configuration for containerd
containerd_registries_mirrors:
  - prefix: docker.io
    mirrors:
      - host: "http://192.168.10.10:35000"
  - prefix: quay.io
    mirrors:
      - host: "http://192.168.10.10:35000"
  - prefix: registry.k8s.io
    mirrors:
      - host: "http://192.168.10.10:35000"

A word on the etcd_download_url line. The default value shipped by kubespray-offline had linux-amd64 hardcoded instead of linux-{{ image_arch }}. On an x86_64 machine this works fine and you'd never notice.

But the moment you try to run this on an ARM64 host, the download fails with a 404 because the file simply doesn't exist at that path. Swapping in {{ image_arch }} makes the template portable across architectures.

Cluster Variables

Beyond the offline-specific settings, a few cluster-level variables need attention. These go in inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml or can be passed as extra vars:

# Network plugin — Flannel is lightweight and works well for lab environments
kube_network_plugin: flannel
flannel_interface: enp0s9

# Proxy mode
kube_proxy_mode: iptables

# Disable nodelocaldns — one less thing to debug in a lab
enable_nodelocaldns: false

# Enable Helm and metrics-server
helm_enabled: true
metrics_server_enabled: true

The flannel_interface setting matters more than it might look. VirtualBox VMs typically have multiple network interfaces — enp0s3 for NAT, enp0s8 for the host-only network, enp0s9 for the internal network.

Flannel needs to bind to the right one, and if you don't specify it, it might pick the NAT interface and nothing will route correctly. I spent an annoying amount of time on this the first time around.

Running the Playbook

With the configuration in place, the actual deployment happens in two playbook runs.

The first playbook sets up the offline repository configuration on all target nodes — making sure yum and pip point to the admin server:

cd /root/kubespray-offline/outputs/kubespray-2.30.0

ansible-playbook -i inventory/mycluster/inventory.ini \
  offline-repo/playbook/offline-repo.yml

This is a short run. It pushes the repo files and pip configuration to each node, and verifies that packages can be resolved from the internal mirror.

The second playbook is the main event — the full Kubernetes cluster deployment:

ansible-playbook -i inventory/mycluster/inventory.ini \
  cluster.yml \
  -e kube_version="1.34.3"

Most of the time goes to the containerd installation, etcd bootstrap, and control plane initialization. Flannel comes up quickly once the kubelet is running.

If something fails mid-run — and in an air-gap environment, something probably will on your first attempt — Ansible’s idempotency means you can fix the issue and re-run the same command. It’ll skip the tasks that already completed successfully.

A few things that tend to go wrong on the first try:

The flannel timeout. If flannel can’t find its subnet environment file, it usually means the default route is missing on the internal interface. The fix is to add one before running the playbook:

# On each k8s node
nmcli connection modify enp0s9 +ipv4.routes "0.0.0.0/0 192.168.10.10 200"
nmcli connection up enp0s9

The metric 200 keeps this from conflicting with any existing default route on another interface.

Image pull failures. If you see failed to pull image: no route to host, the image is either missing from the private registry or containerd's mirror configuration hasn't been applied yet. Quick check:

# List everything in the registry
curl -s http://192.168.10.10:35000/v2/_catalog | python3 -m json.tool

# Verify containerd mirror config on the node
cat /etc/containerd/certs.d/docker.io/hosts.toml

Post-Install Verification

Once the playbook finishes without errors, SSH into the control plane node and run through the basics.

kubectl get nodes -o wide

You should see both nodes in Ready state:

NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP      OS-IMAGE
k8s-node1   Ready    control-plane   5m    v1.34.3   192.168.10.11    Rocky Linux 10
k8s-node2   Ready    <none>          4m    v1.34.3   192.168.10.12    Rocky Linux 10

Next, check that all system pods are running:

kubectl get pods -A

The output should show healthy pods for coredns, flannel, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, etcd, and metrics-server. Every single one of these images should have been pulled from 192.168.10.10:35000 — that's the whole point.

You can confirm the image source on any deployment:

kubectl get deploy -n kube-system -o wide

Every image field should show the 192.168.10.10:35000 prefix. If any image references an external registry like registry.k8s.io or docker.io directly, something in the offline.yml configuration was missed.

A quick smoke test to make sure workloads actually schedule and run:

kubectl run test-nginx --image=192.168.10.10:35000/library/nginx:alpine --port=80
kubectl get pod test-nginx -w

Wait for it to reach Running status, then clean up:

kubectl delete pod test-nginx

If Helm was enabled, verify that too:

helm version
helm list -A

At this point, you have a working Kubernetes cluster that was deployed entirely from local resources. No internet traffic left the 192.168.10.0/24 network during the installation. The registry holds all 55 container images, the nginx file server has every binary, and the nodes are configured to resolve everything internally.

The planning and infrastructure setup that came before this section took the most time. The actual Kubespray run is the easy part; getting the offline supply chain right is where the real work lives.

Day-2 Operations

The cluster is up. Pods are running, kubectl get nodes shows everything Ready, and you've confirmed that all system images were pulled from your internal registry at 192.168.10.10:35000. Now what?

A Kubernetes cluster that can only run its own system components isn’t very useful. You need to deploy actual workloads — your applications, third-party tools, monitoring stacks — and all of those images still need to come from somewhere inside your network. Same goes for Helm charts if you’re using them. This section covers the three approaches I worked through for getting application images into the cluster, plus how to set up Helm in an OCI-native way that plays nicely with your existing registry infrastructure.

Deploying Applications via the Private Registry

The most straightforward approach: pull the image you need on a machine that has internet access (or use a previously downloaded tarball), tag it with your internal registry prefix, push it, and reference the full path in your Kubernetes manifests.

On the admin server, where podman is already installed:

# Pull from Docker Hub (admin server has internet via NAT)
podman pull nginx:alpine

# Tag it for the internal registry
podman tag nginx:alpine 192.168.10.10:35000/library/nginx:alpine

# Push to the private registry
podman push 192.168.10.10:35000/library/nginx:alpine --tls-verify=false

You can verify the image landed correctly:

curl -s http://192.168.10.10:35000/v2/_catalog | python3 -m json.tool

{
    "repositories": [
        "library/nginx",
        "flannel/flannel",
        "flannel/flannel-cni-plugin",
        "coredns/coredns",
        ...
    ]
}

To check which tags exist for a given image:

curl -s http://192.168.10.10:35000/v2/library/nginx/tags/list

{
    "name": "library/nginx",
    "tags": ["alpine"]
}

Now, when you write a Deployment manifest, you reference the full internal path:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-nginx
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
      - name: nginx
        image: 192.168.10.10:35000/library/nginx:alpine
        ports:
        - containerPort: 80

This works, and it’s easy to reason about. Every manifest explicitly states where the image comes from. The downside is obvious, though — every image reference in every manifest, every Helm values file, every quick kubectl run command needs the full 192.168.10.10:35000/... prefix. If you ever migrate to a different registry or change the port, you're doing a find-and-replace across your entire deployment inventory.

Containerd Registry Mirror Configuration

A better approach for most teams: configure containerd on each node to automatically redirect image pulls to your internal registry. When a pod spec says image: nginx:alpine, containerd checks the internal registry first before (failing to) reach Docker Hub.

The configuration lives in containerd’s certs.d directory. For each upstream registry you want to mirror, you create a hosts.toml file:

/etc/containerd/certs.d/
├── docker.io/
│   └── hosts.toml
├── quay.io/
│   └── hosts.toml
├── registry.k8s.io/
│   └── hosts.toml
└── gcr.io/
    └── hosts.toml

Here’s the hosts.toml for Docker Hub:

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."http://192.168.10.10:35000"]
  capabilities = ["pull", "resolve"]
  skip_verify = true

For quay.io:

# /etc/containerd/certs.d/quay.io/hosts.toml
server = "https://quay.io"

[host."http://192.168.10.10:35000"]
  capabilities = ["pull", "resolve"]
  skip_verify = true

Same pattern for registry.k8s.io and gcr.io.

After placing these files, restart containerd on each node:

systemctl restart containerd

From this point on, a pod spec that says image: nginx:alpine will resolve through the mirror. Containerd sees the pull request for docker.io/library/nginx:alpine, checks hosts.toml, finds the mirror entry pointing to 192.168.10.10:35000, and pulls from there instead. Your manifests stay clean — no registry prefixes, no special paths.

One thing to be aware of: the image still needs to exist in the internal registry under the correct repository path. When you pushed library/nginx:alpine earlier, that matches the default Docker Hub library path. For non-library images (say, grafana/grafana:latest), you need to push with the matching path: 192.168.10.10:35000/grafana/grafana:latest.

You can set this up manually on each node, but that’s tedious. The better way is to let Kubespray handle it.

Automating Mirror Setup with Kubespray

In your inventory’s group_vars/all/offline.yml, add the containerd_registries_mirrors block:

# inventory/mycluster/group_vars/all/offline.yml (append)

containerd_registries_mirrors:
  - prefix: docker.io
    mirrors:
      - host: "http://192.168.10.10:35000"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: quay.io
    mirrors:
      - host: "http://192.168.10.10:35000"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: registry.k8s.io
    mirrors:
      - host: "http://192.168.10.10:35000"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: gcr.io
    mirrors:
      - host: "http://192.168.10.10:35000"
        capabilities: ["pull", "resolve"]
        skip_verify: true

Apply it across the cluster with a targeted Ansible run:

cd /root/kubespray-offline/outputs/kubespray-2.30.0

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml \
  --tags containerd

This regenerates the hosts.toml files on every node and restarts containerd automatically. Much cleaner than SSH-ing into each machine.

To verify the mirror is working, try deploying something with a plain Docker Hub image reference:

kubectl run test-nginx --image=nginx:alpine --restart=Never

kubectl describe pod test-nginx | grep "Successfully pulled"

You should see the pull succeed even though the nodes have no internet access. The image is transparently served from 192.168.10.10:35000.

Helm Chart Management with OCI Registry

Helm has supported OCI registries as chart storage since Helm 3.8, and it’s been GA since 3.12. This is a big win for air-gapped environments because you don’t need a separate chart repository server like ChartMuseum — your existing container registry at 192.168.10.10:35000 can store both container images and Helm charts.

The workflow for getting a chart into the registry:

# Say you have a chart directory at /root/nginx-chart/
# Package it into a .tgz archive
helm package /root/nginx-chart

# Output: Successfully packaged chart and saved it to:
#         /root/nginx-chart-1.0.0.tgz

Push the packaged chart to the OCI registry:

helm push nginx-chart-1.0.0.tgz oci://192.168.10.10:35000/helm-charts

If your registry is running plain HTTP (no TLS), Helm will refuse the push by default. You need to tell it the registry is insecure. Create or edit ~/.config/helm/registries.json:

{
  "insecureRegistries": ["192.168.10.10:35000"]
}

Or you can set it per-command (Helm 3.13+):

helm push nginx-chart-1.0.0.tgz oci://192.168.10.10:35000/helm-charts \
  --insecure-skip-tls-verify

Verify the chart is stored:

curl -s http://192.168.10.10:35000/v2/helm-charts/nginx-chart/tags/list

{
    "name": "helm-charts/nginx-chart",
    "tags": ["1.0.0"]
}

Install the chart directly from the OCI registry:

helm install my-nginx oci://192.168.10.10:35000/helm-charts/nginx-chart \
  --version 1.0.0

You can also pull the chart locally first if you want to inspect it before installing:

helm pull oci://192.168.10.10:35000/helm-charts/nginx-chart --version 1.0.0

# This downloads nginx-chart-1.0.0.tgz to the current directory
tar -tzf nginx-chart-1.0.0.tgz | head -10

helm install my-nginx ./nginx-chart-1.0.0.tgz

For teams that use helm template to render manifests before applying them (a common pattern in GitOps workflows), you can template directly from OCI:

helm template my-nginx oci://192.168.10.10:35000/helm-charts/nginx-chart \
  --version 1.0.0 \
  --set replicaCount=3 \
  > rendered-nginx.yaml

kubectl apply -f rendered-nginx.yaml

Migrating Public Charts to the Internal Registry

In practice, you’ll want to pull popular charts from the internet (on a connected machine), then push them into your internal registry for air-gapped consumption. Here’s the pattern I used:

# On a machine with internet access:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Pull the chart as a .tgz file
helm pull bitnami/postgresql --version 16.4.1

# Transfer postgresql-16.4.1.tgz to the admin server (USB, scp, whatever)
# Then on the admin server:
helm push postgresql-16.4.1.tgz oci://192.168.10.10:35000/helm-charts

Now anyone in the cluster can install PostgreSQL:

helm install my-pg oci://192.168.10.10:35000/helm-charts/postgresql \
  --version 16.4.1 \
  --set auth.postgresPassword=secretpassword

One gotcha: the chart itself might reference container images from Docker Hub or other public registries. If you’ve configured the containerd mirrors from section 7.2, those pulls will redirect to your internal registry automatically — but only if you’ve already pushed those images. For a PostgreSQL Helm chart, that means you need bitnami/postgresql, bitnami/postgres-exporter, and whatever else the chart's values.yaml references. Check the default values before deploying:

helm show values oci://192.168.10.10:35000/helm-charts/postgresql \
  --version 16.4.1 | grep -i "image:" -A 3

This tells you exactly which images to stage in your registry.

Why Not ChartMuseum?

You might be wondering about ChartMuseum, the traditional standalone Helm repository server. It still works fine, and the setup is quick:

podman run -d --name chartmuseum -p 8080:8080 \
  -v /data/chartmuseum/charts:/charts \
  -e STORAGE=local -e STORAGE_LOCAL_ROOTDIR=/charts \
  ghcr.io/helm/chartmuseum:v0.16.4

helm repo add internal http://192.168.10.10:8080

But the OCI approach has a clear advantage here: one fewer service to maintain. Your container registry is already running, already backed up (you are backing up /var/lib/registry, right?), and already monitored. Adding Helm charts to it is a zero-infrastructure-cost operation. ChartMuseum is one more container to keep alive, one more port to open, one more thing to troubleshoot at 2 AM when a deployment pipeline fails.

For new setups in 2025, I’d go OCI-native unless you have a specific reason not to.

Troubleshooting & Conclusion

Flannel Subnet Timeout

This one cost me a solid hour. The cluster.yml playbook ran smoothly for about fifteen minutes, then hung at a task called "Wait for flannel subnet.env." After the timeout, the play failed and left the cluster in a half-configured state.

The error looked like this in the Ansible output:

TASK [network_plugin/flannel : Wait for flannel subnet.env file presence] ******
fatal: [k8s-node1]: FAILED! => {"changed": false, "msg": "Timeout when waiting for file /run/flannel/subnet.env"}
fatal: [k8s-node2]: FAILED! => {"changed": false, "msg": "Timeout when waiting for file /run/flannel/subnet.env"}

Flannel’s logs on the node told the real story:

journalctl -u containerd | grep flannel

E0214 03:22:18.123456  Failed to find any valid interface: no compatible interfaces found

The root cause: Flannel needs a default route to determine which network interface to bind to. In this lab, the k8s nodes have three interfaces — enp0s3 (VirtualBox NAT), enp0s8 (host-only), and enp0s9 (the internal 192.168.10.0/24 network where the cluster actually lives). The default route pointed at enp0s3, which is the VirtualBox NAT adapter and has nothing to do with cluster traffic. Flannel picked up that interface, couldn't reach other nodes through it, and gave up.

Two things fixed this. First, I told Flannel explicitly which interface to use by setting flannel_interface in the inventory:

# inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_interface: enp0s9

Second, I added a default route through the internal network on each node so that Flannel’s subnet acquisition would work:

nmcli connection modify enp0s9 +ipv4.routes "0.0.0.0/0 192.168.10.10 200"
nmcli connection up enp0s9

The metric value 200 matters here — it keeps this route as a lower-priority fallback so it doesn't interfere with the existing default route on enp0s3. After making both changes, I re-ran the playbook and Flannel came up within seconds.

One more thing: if you’re re-running cluster.yml after a failed attempt, it's worth doing a reset.yml first to clean up the partial state. Otherwise you can end up with stale CNI configurations that cause even more confusing errors.

ansible-playbook -i inventory/mycluster/inventory.ini reset.yml
ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml

etcd Download Fails with 404

This one is subtle and only shows up if you’re not running on x86_64. The symptom:

TASK [download : Download_file | Download item] *******************************
fatal: [k8s-node1]: FAILED! => {
    "msg": "HTTP Error 404: Not Found",
    "url": "http://192.168.10.10/kubernetes/etcd/etcd-v3.5.26-linux-amd64.tar.gz"
}

The file doesn’t exist on the nginx file server because the actual binary staged during the download phase was etcd-v3.5.26-linux-arm64.tar.gz (or whatever your architecture is). The problem is in offline.yml, where the etcd download URL is hardcoded with amd64:

# The problematic line in offline.yml:
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-v{{ etcd_version }}-linux-amd64.tar.gz"

The fix is to replace amd64 with Kubespray's image_arch variable, which resolves to the correct architecture at runtime:

# Corrected:
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-v{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"

This pattern — hardcoded amd64 where {{ image_arch }} should be — might appear in other URLs too. It's worth doing a quick grep across your offline.yml before running the playbook:

grep -n "amd64" inventory/mycluster/group_vars/all/offline.yml

If any lines show up that aren’t comments, replace them with {{ image_arch }}. On a pure x86_64 lab like the one in this post, you won't hit this issue because amd64 happens to be correct. But if you're planning to reuse the same offline.yml in a mixed-architecture or ARM-based environment, fix it now.

Image Pull Failures After Cluster Setup

After the cluster was running, I tried deploying a test workload and hit this:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Warning  Failed     12s   kubelet            Failed to pull image "nginx:alpine":
           failed to resolve reference "docker.io/library/nginx:alpine":
           failed to do request: dial tcp 104.18.xx.xx:443: connect: no route to host

The kubelet on the node tried to pull nginx:alpine from Docker Hub and failed, because of course there's no internet. This means one of two things: either the image isn't in the private registry, or the containerd mirror isn't configured on that node.

Debugging this is a two-step process. First, check whether the image exists in the registry:

curl -s http://192.168.10.10:35000/v2/_catalog | python3 -m json.tool

If library/nginx doesn't show up in the repository list, the image was never pushed. Go back to section 7.1, push it, and try again.

If the image is there, check the mirror configuration on the failing node:

cat /etc/containerd/certs.d/docker.io/hosts.toml

The file should exist and point to http://192.168.10.10:35000. If it's missing or has the wrong content, either re-apply the Kubespray containerd tags (section 7.2.1) or create the file manually.

After fixing the configuration, restart containerd and delete the failed pod so the kubelet retries the pull:

systemctl restart containerd
kubectl delete pod <pod-name>

The new pod should come up with the image pulled from the internal registry. You can confirm by checking the events:

kubectl describe pod <pod-name> | grep "Successfully pulled"

Normal  Pulled  3s  kubelet  Successfully pulled image "nginx:alpine" in 1.2s

SELinux and Firewall Considerations

Rocky Linux 10 ships with SELinux in enforcing mode and firewalld active by default. During this lab setup, both were disabled early on to reduce variables — setenforce 0 and systemctl stop firewalld. That's fine for a lab. For anything closer to production, here's what you need to keep in mind.

Containerd and kubelet both work with SELinux in enforcing mode, but you need the right policies installed. Kubespray handles most of this automatically when selinux_state is set in the inventory. The container-selinux package (pulled from the local RPM mirror if you've synced it) provides the base policies. Where things get tricky is with custom volume mounts — if your pods mount host paths that SELinux doesn't have a context for, you'll see Permission denied errors that look like filesystem issues but are actually policy violations. The audit2allow tool is your friend here, or you can label the directories with chcon -Rt svirt_sandbox_file_t /path/to/data before mounting.

For firewalld, the key ports that need to be open on the control plane node are 6443 (API server), 2379–2380 (etcd), and 10250 (kubelet). Worker nodes need 10250 and whatever NodePort range you’ve configured (default 30000–32767). Flannel’s VXLAN traffic uses UDP port 8472. If you’re running the private registry on the admin server, port 35000 needs to be reachable from all nodes.

Rather than listing every firewall-cmd invocation, here's the practical advice: if you're in a lab, turn firewalld off and focus on getting the cluster working. Once everything is stable, turn it back on and add rules one service at a time, testing after each change. The Kubespray documentation has a full port matrix you can reference.

Conclusion

Setting up Kubernetes in an air-gapped environment is, frankly, a lot of work. What takes twenty minutes with kubeadm init on an internet-connected machine turns into a multi-day project when every binary, every image, and every package has to be pre-staged.

That said, kubespray-offline takes the worst part out of the equation. The download-all.sh script and its ten sub-scripts handle the tedious work of figuring out which files, images, and packages are needed, downloading them, and organizing the output directory structure. Without it, you'd be manually reading Kubespray's source code to build those lists yourself.

The pieces that still require hands-on attention come down to a few specific areas. Network interface selection is one — Flannel needs to know which interface to bind to, and in multi-NIC VMs that’s never automatic. Architecture strings in URL templates is another — the amd64 vs {{ image_arch }} issue is easy to miss until the playbook fails halfway through. And the post-install work of setting up containerd mirrors and populating the registry with application images is something you'll keep doing for the lifetime of the cluster.

If I were doing this again from scratch, I’d change a few things about the order of operations. I’d set up the containerd registry mirrors as part of the initial offline.yml configuration rather than adding them after the fact. I'd also script the "push all application images" step more aggressively — have a text file listing every image your workloads need, and a loop that pulls, tags, and pushes each one in batch. That way, when a new team member needs to deploy something, the image is already waiting in the registry.

Kubespray Skills Reference

GitHub - sigridjineth/kubespray-skills: Kubespray skills for Kubernetes cluster management

While writing this post, I distilled the entire workflow into a set of reusable Claude Code skills for the kubespray-skills project. The idea is simple: instead of re-reading this 5,000-line article every time you need to recall a specific command or configuration block, you can point Claude Code at the relevant skill file and get accurate, context-aware guidance on the spot.

Three skills came directly out of this air-gap work:

kubespray-airgap covers the end-to-end offline deployment pipeline — the download-all.sh workflow, offline.yml configuration, containerd registry mirrors, the image load-and-push sequence, and the most common failure modes. This one existed before in a minimal form, but the rewrite now includes the full kubespray-offline tooling that we walked through in sections 5 and 6.

kubespray-offline-infra is dedicated to the supporting infrastructure that an air-gapped cluster depends on: BIND for DNS, chrony for NTP, iptables NAT for the network gateway, reposync + nginx for the YUM/DNF mirror, and devpi for the PyPI mirror. Everything from section 3 and 4 of this post lives here. If your DNS forwarding breaks or your nodes can't sync time, this is the skill to consult.

kubespray-helm-airgap handles Helm chart management in offline environments — packaging charts into .tgz archives, pushing them to an OCI registry, setting up ChartMuseum as an alternative, and the often-overlooked step of staging the container images that the charts reference. Section 7.3 of this post maps directly to this skill.

These three join seven other skills in the project — kubespray-lab-setup, kubespray-deployment, kubespray-operations, kubespray-ha-configuration, kubespray-certificates, kubespray-monitoring, and kubespray-troubleshooting — bringing the total to ten. Together they cover the complete Kubernetes cluster lifecycle with Kubespray, from spinning up a Vagrant lab to upgrading a production HA cluster.

The skills are available in the above linked github repository. Each one follows the same structure: YAML frontmatter with a trigger-based description, then the actual reference content with working code examples, configuration blocks, and troubleshooting tables.

Final Thoughts

The air-gap constraint forces you to think carefully about supply chain and dependency management in a way that internet-connected clusters let you ignore. Every image has a provenance. Every binary has a version. Nothing appears magically from the internet at deploy time. That discipline is worth carrying forward even if your next cluster does have internet access.

Why HA and Upgrade Strategy Matter

Everyone loves the “Day 1” excitement of spinning up a new Kubernetes cluster. The terminal logs fly by, the nodes report Ready, and you feel like you have successfully built a modern infrastructure. But the real work begins on “Day 2.” That is when the network flakes out, a disk fills up, or a critical security vulnerability forces an immediate patch.

In a production environment, stability is the only metric that truly counts. If you run a single Control Plane node, you are living dangerously. I recall an incident early in my career where a simple OS security update required a server reboot. Because we lacked High Availability (HA), that reboot meant the API server vanished. Existing pods kept running, but we lost the ability to deploy fixes, scale up during a traffic spike, or even query the cluster status. We were effectively flying blind until the node came back online.

To sleep soundly at night, you generally need a minimum of three Control Plane nodes. This number exists for a specific reason: maintaining the etcd quorum.

Etcd is the brain of your cluster, storing the state of everything. If you lose quorum, the cluster goes into read-only mode or stops functioning entirely. With three nodes, you can lose one and still maintain a majority (two), keeping the cluster operational.

Here is a simplified view of what a robust Kubespray inventory looks like compared to a fragile one. We define multiple control plane nodes to distribute the risk.

# A fragile, single-point-of-failure setup
[kube_control_plane]
node1

# A production-ready HA setup
[kube_control_plane]
node1
node2
node3

[etcd]
node1
node2
node3

Beyond just surviving hardware failures, HA is the prerequisite for a sane upgrade strategy. Kubernetes evolves rapidly. New versions arrive every few months, bringing performance improvements and security patches. If you treat your cluster as a static monument that should never be touched, it quickly becomes a liability.

I have seen teams paralyzed by fear, sticking to End-of-Life (EOL) versions because they worry an upgrade will break everything.

A proper HA setup changes that dynamic. It allows you to perform Rolling Upgrades. You can drain one node, upgrade it, reboot it, and bring it back online — all while the other nodes handle the traffic. The users never notice a thing. This turns upgrades from a terrifying event into a routine maintenance task.

In this guide, we use Kubespray because it handles this complexity for us. It automates the distribution of components and manages the rolling upgrade logic, ensuring that we adhere to these best practices without having to manually wire every connection.

What This Guide Covers

Running Kubernetes in production is rarely about the initial kubeadm init. It’s about what happens three months later — when certificates expire, a node crashes, or a security patch forces a version upgrade.

This guide moves beyond the “Hello World” of cluster creation. We are building a High Availability (HA) cluster using Kubespray, and more importantly, we are going to break it, fix it, and upgrade it. The goal is to simulate a real-world environment where uptime matters, even when you need to swap out a control plane node.

We will start by provisioning a local lab environment using Vagrant and VirtualBox. This isn’t a single-node Minikube setup; we are spinning up a multi-node architecture running Rocky Linux 10. The infrastructure looks like this:

• 1 Admin/LoadBalancer Node: Runs HAProxy and executes Ansible.
• 3 Control Plane Nodes: Ensures the API server and etcd have redundancy.
• 2 Worker Nodes: Runs the actual workloads.

Here is the target inventory structure we will build in inventory.ini.

[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15

Once the cluster is up, we shift focus to Day-2 Operations. This is where the real learning happens. We will configure an external HAProxy to manage API traffic and test if the cluster survives when a control plane node goes dark.

You will also see how to handle lifecycle events using Kubespray’s playbooks. We aren’t just running the installation script once; we will use specific playbooks to scale the cluster and remove faulty nodes safely.

# Example: Scaling the cluster by adding a new worker
ansible-playbook -i inventory/mycluster/inventory.ini -v scale.yml --limit=k8s-node5

Finally, we tackle the task that makes most operators nervous: Upgrades. We will walk through a rolling upgrade from Kubernetes v1.32.9 to v1.32.10, and eventually to v1.34.3, ensuring that workloads stay running while the underlying infrastructure shifts beneath them. By the end, you should feel comfortable managing the full lifecycle of a production-grade Kubernetes cluster.

Here is the drafted content for Section 2. Kubernetes HA Fundamentals, written in English. I’ve focused on a natural, practical tone that avoids robotic “AI-isms” while keeping technical precision.

Control Plane Components and Their Roles

Setting up a single-node cluster on a laptop is satisfying. You run a script, see “Ready,” and everything just works. But moving that setup to production is a different story. The moment you start managing real traffic, the fragility of a single control plane becomes a liability you can’t afford.

This section covers why High Availability (HA) isn’t just a “nice-to-have” feature but a baseline requirement for any serious Kubernetes environment.

Before we talk about redundancy, we need to agree on what we are actually replicating. The Control Plane isn’t a monolith; it’s a collection of specific processes that coordinate the entire cluster.

When you run a command like kubectl apply -f deployment.yaml, you aren’t talking to the cluster as a whole — you are talking to these specific components:

1. kube-apiserver: The front door. It handles all REST requests, validates them, and updates the state in etcd. It is the only component that talks directly to the database.
2. etcd: The brain. This is a consistent, distributed key-value store where all cluster data lives. If you lose etcd, you lose the cluster.
3. kube-scheduler: The decision maker. It watches for new Pods with no assigned node and selects the best one based on resources and constraints.
4. kube-controller-manager: The reconciler. It runs controller loops (like the Node Controller or ReplicaSet Controller) to ensure the current state matches the desired state.

You can see these components running as static pods on your control plane node. If you are logged into a control plane node, a quick check looks like this.

# Checking the static pod manifests
$ ls /etc/kubernetes/manifests/
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# Verifying they are running
$ crictl ps --name kube
CONTAINER ID   IMAGE                    NAME                      STATE
a1b2c3d4e5f6   .../kube-apiserver       kube-apiserver            Running
b2c3d4e5f6g7   .../etcd                 etcd                      Running
c3d4e5f6g7h8   .../kube-scheduler       kube-scheduler            Running
d4e5f6g7h8i9   .../kube-controller-mgr  kube-controller-manager   Running

Why Single Control Plane Is a Risk

In a single control plane setup, all the components listed above run on one machine. This creates a classic Single Point of Failure (SPOF).

If that one machine goes down, the impact is immediate and severe:

• The API goes silent: You cannot run kubectl commands. CI/CD pipelines fail. Monitoring tools that query the API stop gathering data.
• Scheduling stops: If a worker node crashes while the control plane is down, the Pods on that worker are gone. No scheduler means no new Pods are created to replace them.
• State is locked: You cannot scale up, change configurations, or roll back deployments.

The applications already running on healthy worker nodes might keep running for a while, but the cluster effectively becomes a “zombie” — functioning but brain-dead. You are one disk failure or one bad OS patch away from a total outage.

Recommended HA Architecture

To fix this, we don’t just add a backup; we create a cluster of control planes. The standard production architecture involves three control plane nodes.

Why three? It comes down to etcd.

Etcd uses the Raft consensus algorithm, which requires a majority (quorum) to write data.

• 1 node: Quorum is 1. If it fails, you stop.
• 2 nodes: Quorum is 2 (Majority of 2 is 2). If one fails, you only have 1 left. You lose quorum. This is actually worse than a single node because you have twice the hardware risk for zero gain in availability.
• 3 nodes: Quorum is 2. You can lose 1 node and still have 2 left to form a majority. The cluster keeps writing data.

Here is what the topology typically looks like in a inventory.ini file for Kubespray.

[kube_control_plane]
control-node-01 ansible_host=192.168.10.11
control-node-02 ansible_host=192.168.10.12
control-node-03 ansible_host=192.168.10.13

[etcd]
control-node-01
control-node-02
control-node-03

In this setup, we usually place a Load Balancer (like HAProxy or a cloud LB) in front of the three API servers. The worker nodes and your local kubectl talk to the Load Balancer, which distributes traffic to the healthy control plane nodes.

Worker Node HA: Built-in but Dependent on Control Plane

People often confuse Worker HA with Control Plane HA. Kubernetes handles Worker HA natively. If a worker node fails, the ReplicaSet controller notices that the Pods are gone and spins up replacements on other available nodes.

But there is a catch: This relies entirely on the Control Plane.

If your Control Plane is down, the cluster has no way of knowing a worker node has failed. The logic that says “Current Replicas < Desired Replicas” lives in the Controller Manager on the master node.

So, while Kubernetes is designed to heal worker failures automatically, that self-healing capability is only as reliable as your Control Plane. Without a robust CP setup, your Worker HA guarantees are effectively null and void during a master outage.

Common Failure Scenarios Without HA

I have seen clusters break in many creative ways. Without HA, minor operational tasks turn into major downtime events.

• OS Patching & Reboots: You need to apply a security patch to the kernel. On a single master, rebooting means taking the entire API offline for 5–10 minutes. With HA, you can roll through the nodes one by one without anyone noticing.
• Disk Filling Up: Logs or backups fill the root partition. Etcd is extremely sensitive to disk latency and space. If the disk fills up on a single master, etcd panics and goes read-only or crashes.
• Expired Certificates: If the API server certificates expire on a single node, you are locked out. In an HA setup, if one node has an issue, you can still access the others to troubleshoot and rotate certificates.

The goal of High Availability is to make day-to-day maintenance boring and predictable.

3. Day-2 Operations: The Case for Regular Upgrades

Building a Kubernetes cluster is often the easy part. Keeping it secure and stable over the next two years — that is the real challenge.

In the world of Kubernetes, “Day-2” refers to everything that happens after the initial install. Among these tasks, upgrades are often the most dreaded. However, treating a cluster as a “set it and forget it” artifact is a guaranteed recipe for future instability.

Kubernetes as a Continuously Evolving Platform

Kubernetes moves fast. The project releases a new minor version roughly every four months. Unlike a traditional Linux distribution where you might sit on a Long Term Support (LTS) release for five years, Kubernetes forces you to keep moving.

Support windows are short. Generally, only the latest three minor versions are supported (N-2 policy). For example, if you are running v1.28 and the current version is v1.32, you are already falling out of the support window. This means no security patches for critical components like kube-apiserver or kubelet.

New versions bring more than just patches; they introduce essential performance improvements and security fixes. Stagnation holds your infrastructure back from becoming more efficient and secure.

Technical Debt from Deferred Upgrades

The longer you wait to upgrade, the harder it gets. I often compare it to skipping dental appointments; what could have been a routine cleaning eventually turns into a root canal.

If you try to jump across multiple versions at once — say, from v1.30 straight to v1.34 — you run into a minefield of breaking changes. Kubernetes frequently retires old APIs. If you haven’t updated your manifests incrementally, your deployments might simply fail during a major jump.

Fixing one deprecated API in a manifest is easy. Fixing hundreds across fifty repositories while your production upgrade is stalled? That is the technical debt you want to avoid. Regular, incremental upgrades are the only way to keep this debt manageable.

Prerequisites for Zero-Downtime Upgrades

You cannot just replace a binary and hope for the best. A production upgrade must be seamless to the end user.

To achieve this, your cluster needs to meet specific conditions before you even think about running an upgrade playbook:

1. High Availability (HA): As discussed in Section 2, you need multiple Control Plane nodes. Upgrading the API server involves restarting it. If you have only one, your API goes down. If you have three, you can upgrade them one by one (Rolling Update) without losing availability.
2. Spare Capacity: When a worker node is being upgraded, it must be drained of all running workloads. You need enough resource headroom on other nodes to accept these displaced Pods.
3. Pod Disruption Budgets (PDB): You should define how many replicas of an application can be down at once. This prevents the upgrade process from accidentally taking down all instances of a critical service.

The upgrade process conceptually involves isolating a node (cordon), moving its workload elsewhere (drain), upgrading the software, and bringing it back (uncordon). Kubespray automates this entire dance, but the underlying infrastructure must be ready to support it. We will see this in action in Section 9.

Declarative Cluster Management with Kubespray and Git

The most stable way to manage upgrades is through Infrastructure as Code (IaC). We avoid making ad-hoc changes directly on the servers. Instead, we define the desired state of the cluster in configuration files.

Kubespray fits perfectly into this workflow. Your cluster configuration — inventory, variables, and versions — lives in text files. Ideally, you manage these files in a Git repository. When you want to upgrade, you modify the kube_version variable in a file like group_vars/k8s_cluster/k8s-cluster.yml and commit the change.

# inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

# 🔴 OLD STATE
# kube_version: v1.32.9

# 🟢 NEW STATE
kube_version: v1.32.10

This declarative approach provides a permanent record of when an upgrade happened and who triggered it.

Note: In our lab environment later, we will sometimes use the command line (-e kube_version=…) to override this version for speed and simplicity. However, in a real production environment, updating the inventory file via Git is the recommended best practice.

This draft focuses on resolving the critical issues around self-referential SSH, Python 3.12 package management, and script reliability, while restoring descriptive depth where needed.

Lab Environment Setup

Before we deploy Kubernetes, we need a solid foundation. In a real data center, this would involve racking servers and cabling switches. In our lab, we will simulate this infrastructure using VirtualBox and Vagrant.

This section walks through building a 6-node cluster environment. By the end, you will have a fully functional lab that mimics a production-grade HA setup.

• Prerequisites: Ensure you have VirtualBox 7.2.4+ and Vagrant 2.4.9+ installed.

Architecture Overview

We are building a standard high-availability topology. To make it realistic, we separate the “management” layer from the cluster itself.

                    ┌─────────────┐
                    │  admin-lb   │
                    │ HAProxy/NFS │
                    │ .10.10      │
                    └──────┬──────┘
                           │ :6443 (API Load Balancer)
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │k8s-node1 │ │k8s-node2 │ │k8s-node3 │
        │  CP+etcd │ │  CP+etcd │ │  CP+etcd │
        │ .10.11   │ │ .10.12   │ │ .10.13   │
        └──────────┘ └──────────┘ └──────────┘
              ┌────────────┴────────────┐
              ▼                         ▼
        ┌──────────┐             ┌──────────┐
        │k8s-node4 │             │k8s-node5 │
        │  Worker  │             │ (Standby)│
        │ .10.14   │             │ .10.15   │
        └──────────┘             └──────────┘

• Admin/LB Node (1EA): This node wears multiple hats. It acts as the Ansible controller (to run Kubespray), the external Load Balancer (HAProxy) for the API server, an NFS server for shared storage, and a jump host with management tools (kubectl, k9s, helm).
• Control Plane Nodes (3EA): These three nodes run the core Kubernetes components (kube-apiserver, etcd, controller-manager, scheduler). Three is the magic number for etcd quorum; it allows the cluster to survive the loss of one node.
• Worker Nodes (2EA): These nodes run your actual workloads (Pods). k8s-node4 will be active initially, while k8s-node5 will be provisioned as a VM but added to the cluster later in Section 7.

VM Specifications and Network Layout

We will use Rocky Linux 10 as the base OS. Each VM needs two network interfaces:

1. NAT Network (NIC1, typically enp0s3): For internet access (downloading packages) and Vagrant management.
2. Host-Only Network (NIC2, typically enp0s9): For internal cluster communication. We will use this interface (enp0s9) for the CNI (Flannel) and API traffic.

4.3 Vagrantfile Walkthrough

The Vagrantfile defines our entire infrastructure as code. It loops through a configuration to create all 6 nodes at once.

Key points to note:

• We use bento/rockylinux-10 as the base box.
• We set nicpromisc2 to allow-all. This enables promiscuous mode on the host-only adapter, which is critical for CNI overlay traffic (like Flannel VXLAN) to reach its destination properly.
• We use linked_clone = true to save disk space by sharing the base image across VMs, rather than duplicating it for each node.

# Vagrantfile
BOX_IMAGE = "bento/rockylinux-10"
N = 5 # Number of k8s nodes (3 CP + 2 Workers)

Vagrant.configure("2") do |config|
  
  # 1. Define K8s Nodes (k8s-node1 to k8s-node5)
  (1..N).each do |i|
    config.vm.define "k8s-node#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.provider "virtualbox" do |vb|
        vb.name = "k8s-node#{i}"
        vb.cpus = 4
        vb.memory = 2048
        # Enable promiscuous mode for CNI traffic
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.linked_clone = true
      end
      subconfig.vm.hostname = "k8s-node#{i}"
      subconfig.vm.network "private_network", ip: "192.168.10.1#{i}"
      subconfig.vm.provision "shell", path: "init_cfg.sh"
    end
  end

  # 2. Define Admin/LB Node
  config.vm.define "admin-lb" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.provider "virtualbox" do |vb|
      vb.name = "admin-lb"
      vb.cpus = 2
      vb.memory = 1024
      vb.linked_clone = true
    end
    subconfig.vm.hostname = "admin-lb"
    subconfig.vm.network "private_network", ip: "192.168.10.10"
    subconfig.vm.provision "shell", path: "admin-lb.sh"
  end
end

Admin-LB Node Bootstrap Script (admin-lb.sh)

This script configures the admin node. It installs HAProxy for load balancing, sets up NFS for storage, prepares SSH keys for Ansible, and installs cluster management tools.

#!/bin/bash
# admin-lb.sh

echo "[TASK 1] OS Prep (Timezone, Firewall, SELinux, SSH)"
timedatectl set-timezone Asia/Seoul
systemctl disable --now firewalld
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

# Enable Root Login for consistency (optional for admin-lb, critical for nodes)
echo "root:qwe123" | chpasswd
sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl restart sshd

echo "[TASK 2] Local DNS Setup"
# Clean up cloud-init entries that might conflict with our static IPs
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
cat <<EOF >> /etc/hosts
192.168.10.10 k8s-api-srv.admin-lb.com admin-lb
192.168.10.11 k8s-node1
192.168.10.12 k8s-node2
192.168.10.13 k8s-node3
192.168.10.14 k8s-node4
192.168.10.15 k8s-node5
EOF

echo "[TASK 3] Install HAProxy"
dnf install -y haproxy

# HAProxy Config: API LB + Stats + Prometheus Metrics
cat <<EOF > /etc/haproxy/haproxy.cfg
global
    log 127.0.0.1 local2
    stats socket /var/lib/haproxy/stats

defaults
    mode tcp
    timeout connect 10s
    timeout client 1m
    timeout server 1m

# Kubernetes API Server LB
frontend k8s-api
    bind *:6443
    default_backend k8s-api-backend

backend k8s-api-backend
    option tcp-check
    balance roundrobin
    server k8s-node1 192.168.10.11:6443 check
    server k8s-node2 192.168.10.12:6443 check
    server k8s-node3 192.168.10.13:6443 check

# HAProxy Stats Dashboard (http://<admin-ip>:9000/haproxy_stats)
listen stats
    bind *:9000
    mode http
    stats enable
    stats uri /haproxy_stats

# Prometheus Metrics Exporter (http://<admin-ip>:8405/metrics)
frontend prometheus
    bind *:8405
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
    no log
EOF
systemctl enable --now haproxy

echo "[TASK 4] Install NFS Server"
dnf install -y nfs-utils
mkdir -p /srv/nfs/share
chown nobody:nobody /srv/nfs/share
chmod 755 /srv/nfs/share
# Note: 'async' improves performance for lab environments but risks data loss in production
echo '/srv/nfs/share *(rw,async,no_root_squash,no_subtree_check)' > /etc/exports
systemctl enable --now nfs-server
exportfs -rav

echo "[TASK 5] Install Tools (kubectl, k9s, helm)"
# Install kubectl
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubectl
EOF
dnf install -y -q kubectl --disableexcludes=kubernetes

# Install k9s (using curl since wget might be missing)
curl -fsSL -o /tmp/k9s_linux_amd64.tar.gz https://github.com/derailed/k9s/releases/latest/download/k9s_linux_amd64.tar.gz
tar -xzf /tmp/k9s_linux_amd64.tar.gz -C /usr/local/bin/ k9s
chmod +x /usr/local/bin/k9s

# Install Helm
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.16.2 bash

echo "[TASK 6] SSH Key Distribution"
dnf install -y python3-pip git sshpass
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa
# Distribute key to all k8s nodes (admin-lb itself doesn't need to be in inventory)
for i in {1..5}; do
  sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.1$i
done

echo "[TASK 7] Clone Kubespray"
git clone -b v2.29.1 https://github.com/kubernetes-sigs/kubespray.git /root/kubespray
# Use --break-system-packages for Rocky 10 (Python 3.12+ PEP 668)
pip3 install --break-system-packages -r /root/kubespray/requirements.txt

4.5 Worker/Control Plane Node Init Script (init_cfg.sh)

This script prepares the cluster nodes. It disables swap, loads required kernel modules, sets up local DNS, and configures SSH to allow Ansible connections.

#!/bin/bash
# init_cfg.sh

echo "[TASK 1] Timezone & Swap"
timedatectl set-timezone Asia/Seoul
swapoff -a
sed -i '/swap/d' /etc/fstab

echo "[TASK 2] Disable Firewalld & SELinux"
systemctl disable --now firewalld
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

echo "[TASK 3] Kernel Modules & Sysctl"
cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

echo "[TASK 4] Local DNS"
# Remove cloud-init generated hostname entries that conflict with our static mappings
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts

# We also add the admin node alias 'k8s-api-srv.admin-lb.com' for use in Section 6 (External LB)
cat <<EOF >> /etc/hosts
192.168.10.10 k8s-api-srv.admin-lb.com admin-lb
192.168.10.11 k8s-node1
192.168.10.12 k8s-node2
192.168.10.13 k8s-node3
192.168.10.14 k8s-node4
192.168.10.15 k8s-node5
EOF

echo "[TASK 5] SSH Config (Allow Root Login for Ansible)"
echo "root:qwe123" | chpasswd
# Use sed to safely modify existing config without duplication
sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl restart sshd

Deploying the Environment with Vagrant

With the three files (Vagrantfile, admin-lb.sh, init_cfg.sh) in the same directory, we are ready to launch.

1. Start the VMs

• $ vagrant up

1. This process will take a few minutes as it downloads the OS image and provisions all 6 nodes.
2. Verify the Status:
   Run vagrant status to confirm all machines are up.

$ vagrant status
Current machine states:

k8s-node1                 running (virtualbox)
k8s-node2                 running (virtualbox)
k8s-node3                 running (virtualbox)
k8s-node4                 running (virtualbox)
k8s-node5                 running (virtualbox)
admin-lb                  running (virtualbox)

Log in to the Admin Node:
All subsequent commands for deploying Kubernetes will be run from inside this node.

$ vagrant ssh admin-lb
[vagrant@admin-lb ~]$ sudo -i
[root@admin-lb ~]#

Test Connectivity:
First, verify network connectivity and DNS resolution.

Next, verify that HAProxy is running by checking the stats page response (200 OK).

ping -c 2 k8s-node1
ping -c 2 k8s-node4

curl -s -I http://192.168.10.10:9000/haproxy_stats | head -1
# HTTP/1.1 200 OK

If everything responds, your infrastructure is ready. We can now proceed to Section 5 to deploy the cluster using Kubespray.

Deploying Kubernetes with Kubespray

With the lab environment fully provisioned — the admin-lb node running HAProxy and NFS, three control plane VMs, and two worker VMs all booted and initialized — we are now ready to deploy a production-grade Kubernetes cluster using Kubespray.

This section walks through every step of the deployment process in detail: understanding the Kubespray project structure, configuring the inventory, tuning cluster variables, executing the installation playbook, and performing thorough post-deployment validation.

Kubespray Directory Structure and Key Files

Kubespray is an Ansible-based project that wraps kubeadm with extensive automation for deploying and managing Kubernetes clusters. During the admin-lb bootstrap phase (covered in Section 4), we cloned the Kubespray v2.29.1 repository into /root/kubespray. Before modifying any configuration, it is essential to understand the project layout.

/root/kubespray/
├── ansible.cfg                  # Ansible configuration (forks, pipelining, etc.)
├── cluster.yml                  # Main deployment playbook
├── scale.yml                    # Add-node playbook
├── remove-node.yml              # Remove-node playbook
├── upgrade-cluster.yml          # Rolling upgrade playbook
├── reset.yml                    # Full cluster teardown playbook
├── roles/                       # Ansible roles for each component
│   ├── download/                # Container image and binary downloads
│   ├── kubernetes/              # kubeadm, kubelet, static pod manifests
│   ├── kubernetes-apps/         # CoreDNS, metrics-server, ingress, etc.
│   ├── etcd/                    # etcd cluster bootstrap and management
│   ├── network_plugin/          # CNI plugins (flannel, calico, cilium, etc.)
│   └── container-engine/        # containerd, CRI-O runtime setup
├── inventory/
│   └── sample/                  # Template inventory (copy this)
│       ├── inventory.ini        # Node definitions and group assignments
│       └── group_vars/
│           ├── all/
│           │   └── all.yml      # Global variables (LB config, DNS, etc.)
│           ├── k8s_cluster/
│           │   ├── k8s-cluster.yml    # Core K8s settings (version, CNI, proxy mode)
│           │   └── addons.yml         # Optional add-ons (metrics-server, ingress, etc.)
│           └── etcd.yml         # etcd-specific variables
└── contrib/
    └── inventory_builder/       # Dynamic inventory generator script

Preparing the Inventory Directory

Rather than editing the sample inventory directly, we copy it to create our own cluster-specific configuration:

cd /root/kubespray
cp -rfp inventory/sample inventory/mycluster

This gives us a clean, isolated workspace under inventory/mycluster/ where all our customizations live. The original inventory/sample/ remains untouched, which is important when we later upgrade Kubespray versions via git checkout — our custom inventory directory won't be overwritten.

Inventory Configuration (inventory.ini)

The inventory file is the heart of a Kubespray deployment. It defines which nodes exist, what roles they play, and how Ansible groups them for targeted playbook execution.

The Complete Inventory File

# /root/kubespray/inventory/mycluster/inventory.ini

[all]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.20.11
k8s-node2 ansible_host=192.168.10.12 ip=192.168.20.12
k8s-node3 ansible_host=192.168.10.13 ip=192.168.20.13
k8s-node4 ansible_host=192.168.10.14 ip=192.168.20.14

[kube_control_plane]
k8s-node1
k8s-node2
k8s-node3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4

[k8s_cluster:children]
kube_control_plane
kube_node

Breakdown of Each Section

[all] — Node Definitions

Every node in the cluster is listed here with two critical variables:

• ansible_host: The IP address Ansible uses to SSH into the node. In our lab, this is the enp0s8 interface on the 192.168.10.0/24 management network.
• ip: The IP address Kubernetes components bind to for intra-cluster communication. This is the enp0s9 interface on the 192.168.20.0/24 cluster network.

This dual-network design is intentional: management traffic (Ansible SSH, admin access) stays on one network, while Kubernetes API, etcd, and pod-to-pod traffic flows on a separate, dedicated network. In production environments, this separation improves both security and performance.

[kube_control_plane] — Control Plane Nodes

Three nodes are assigned to the control plane group: k8s-node1, k8s-node2, and k8s-node3. Each will run the full set of Kubernetes control plane components as static pods:

• kube-apiserver
• kube-controller-manager
• kube-scheduler

The order of nodes in this group matters. The first node (k8s-node1) acts as the initial control plane node during kubeadm init. Subsequent nodes join via kubeadm join. When adding a new control plane node later, it must always be appended to the end of this group — never inserted in the middle.

[etcd:children] — etcd Cluster Membership

[etcd:children]
kube_control_plane

By using children, we declare that etcd membership is inherited from the kube_control_plane group. This means all three control plane nodes also run etcd, forming a 3-member etcd cluster. This is the "stacked etcd" topology recommended for most deployments — etcd runs on the same nodes as the control plane, reducing infrastructure complexity while still maintaining a proper quorum.

With 3 etcd members, the cluster can tolerate 1 node failure. The quorum formula is (n/2) + 1, so a 3-node etcd cluster requires at least 2 members to be healthy. This is why odd numbers (3, 5, 7) are always used for etcd — even numbers provide no additional fault tolerance over the odd number below them.

[kube_node] — Worker Nodes

Currently only k8s-node4 is listed as a worker. We will add k8s-node5 later using the scale.yml playbook (covered in Section 7). Worker nodes run the kubelet and kube-proxy, but no control plane components.

[k8s_cluster:children] — Aggregate Group

This is a convenience group that encompasses all Kubernetes nodes (both control plane and workers). Kubespray uses it to apply cluster-wide configurations such as CNI plugin installation, kubelet settings, and container runtime setup.

Ansible Variable Precedence in Kubespray

Before customizing any settings, it is critical to understand how Ansible resolves variable conflicts. Kubespray defines variables at multiple levels, and the precedence hierarchy determines which value wins when the same variable is set in multiple places.

The Precedence Hierarchy (Lowest to Highest)

1. Role defaults         (roles/xxx/defaults/main.yml)          ← Lowest priority
2. Role vars             (roles/xxx/vars/main.yml)
3. Inventory group_vars  (inventory/mycluster/group_vars/...)
4. Inventory host_vars   (inventory/mycluster/host_vars/...)
5. Playbook vars         (vars: section in playbook YAML)
6. CLI extra-vars        (--extra-vars / -e on command line)    ← Highest priority

What This Means in Practice

Role defaults are the baseline values that Kubespray ships with. For example, the default Kubernetes version, default CNI plugin, default kube-proxy mode, and so on are all defined in role defaults. These are intentionally low-priority — they are meant to be overridden.

Role vars are set by the Kubespray developers for values that should generally not be overridden by users. These take precedence over role defaults but can still be overridden by inventory-level variables.

Inventory group_vars are where most of our customization happens. The files under inventory/mycluster/group_vars/ are the primary configuration surface for Kubespray:

group_vars/
├── all/
│   └── all.yml              # Global settings: LB config, DNS, timezone
├── k8s_cluster/
│   ├── k8s-cluster.yml      # K8s core: version, CNI, proxy mode, feature gates
│   └── addons.yml           # Add-ons: metrics-server, ingress, dashboard
└── etcd.yml                 # etcd settings: metrics, compaction, quotas

host_vars allow per-node overrides. This is rarely needed but can be useful for heterogeneous hardware or special-purpose nodes.

CLI --extra-vars (-e) have the highest precedence and override everything. This is the mechanism we use to specify the exact Kubernetes version at deployment time:

ansible-playbook cluster.yml -e kube_version="1.32.9"

Even if kube_version is set to a different value in group_vars/k8s_cluster/k8s-cluster.yml, the -e flag on the command line always wins. This is particularly useful during upgrades, where we increment the version via the CLI without modifying any files.

A common mistake is to set a variable in group_vars and wonder why it has no effect. The cause is almost always that the same variable is defined at a higher precedence level (often in role/vars, which is hard to override from group_vars). In such cases, the only reliable way to override it is via -e on the command line.

Customizing Cluster Settings (CNI, Proxy Mode, Add-ons)

With the inventory structure understood, let’s configure the key cluster parameters. We edit three files under inventory/mycluster/group_vars/.

5.4.1 Global Settings — all/all.yml

# /root/kubespray/inventory/mycluster/group_vars/all/all.yml

## Load Balancer Configuration
## (Default: client-side LB via nginx-proxy on each worker)
## We will modify these later in Section 6 when configuring external LB
# apiserver_loadbalancer_domain_name: "k8s-api-srv.admin-lb.com"
# loadbalancer_apiserver:
#   address: 192.168.10.10
#   port: 6443
# loadbalancer_apiserver_localhost: true   # default: enables nginx-proxy on workers

For the initial deployment, we leave the load balancer settings at their defaults. This means Kubespray will configure client-side load balancing using nginx static pods on each worker node (Case 1 architecture, detailed in Section 6).

Core Kubernetes Settings — k8s_cluster/k8s-cluster.yml

This is the most important configuration file. Here we define the CNI plugin, kube-proxy mode, service/pod CIDRs, and other core cluster parameters:

# /root/kubespray/inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

## CNI Plugin
kube_network_plugin: flannel

## Flannel Interface Selection
## CRITICAL: In multi-NIC environments, flannel must be told which interface to use
## for VXLAN overlay traffic. Without this, flannel may pick the wrong NIC
## (e.g., the NAT interface enp0s3 instead of the cluster network enp0s9).
flannel_interface: enp0s9

## Kube-proxy Mode
## Options: iptables (default in Kubespray), ipvs
## iptables is simpler and sufficient for small-to-medium clusters
kube_proxy_mode: iptables

## DNS Configuration
## CoreDNS is the default and only supported DNS provider
## Kubespray default clusterDNS is 10.233.0.3 (not the kubeadm default of 10.233.0.10)
## This is set via dns_domain and other dns_* variables

## NodeLocal DNS Cache
## Disabled in this lab for simplicity
enable_nodelocaldns: false

## Service and Pod CIDRs (defaults shown — usually no need to change)
# kube_service_addresses: 10.233.0.0/18
# kube_pods_subnet: 10.233.64.0/18

Why Flannel? In this lab environment, we use Flannel for its simplicity. Flannel provides basic L3 overlay networking via VXLAN and is easy to understand and debug. For production environments requiring network policies, Calico or Cilium would be the recommended choices.

The flannel_interface Setting Explained: VirtualBox VMs typically have multiple network interfaces. In our lab:

• enp0s3 — NAT interface (VirtualBox default, used for internet access)
• enp0s8 — Host-only adapter #1 (management network, 192.168.10.0/24)
• enp0s9 — Host-only adapter #2 (cluster network, 192.168.20.0/24)

Without explicitly setting flannel_interface: enp0s9, Flannel might select enp0s3 (the NAT interface) for its VXLAN tunnel endpoints. This would cause pod-to-pod communication to fail because the NAT interface does not provide direct connectivity between VMs. By specifying enp0s9, we ensure Flannel uses the dedicated cluster network.

Why iptables Mode? The iptables kube-proxy mode is the battle-tested default. While IPVS mode offers better performance at scale (O(1) vs. O(n) for service routing rules), iptables mode is simpler to debug and perfectly adequate for clusters with fewer than a thousand services. For this lab, iptables is the appropriate choice.

DNS Address Note: Kubespray sets the cluster DNS service IP to 10.233.0.3 by default, which differs from kubeadm's default of 10.233.0.10. This is a Kubespray-specific convention. The kubelet on every node is configured with --cluster-dns=10.233.0.3, and the CoreDNS service is created with this ClusterIP. If you're migrating from a kubeadm-managed cluster, be aware of this difference.

Add-ons Configuration — k8s_cluster/addons.yml

# /root/kubespray/inventory/mycluster/group_vars/k8s_cluster/addons.yml

## Metrics Server
## Enables the Kubernetes Metrics Server for resource usage collection
## Required for: kubectl top, HPA (Horizontal Pod Autoscaler), VPA
metrics_server_enabled: true

## Other add-ons (disabled in this lab, shown for reference)
# ingress_nginx_enabled: false
# dashboard_enabled: false
# helm_enabled: false

The Metrics Server is the only add-on we enable at deployment time. It provides the /apis/metrics.k8s.io endpoint that powers kubectl top nodes, kubectl top pods, and the Horizontal Pod Autoscaler. We will install additional monitoring components (Prometheus, Grafana) separately using Helm in Section 8.

etcd Settings — etcd.yml

For the initial deployment, we keep etcd settings at their defaults. Later (in Section 8.3), we will enable etcd metrics:

# /root/kubespray/inventory/mycluster/group_vars/etcd.yml

## etcd metrics (will be enabled later for Prometheus monitoring)
# etcd_metrics: true
# etcd_listen_metrics_urls: "http://0.0.0.0:2381"

Running cluster.yml and Verifying Deployment

With the inventory and variables configured, we are ready to deploy the cluster.

Pre-Flight Checks

Before running the playbook, verify Ansible connectivity to all nodes:

cd /root/kubespray

# Test SSH connectivity to all nodes
ansible -i inventory/mycluster/inventory.ini all -m ping

Expected output:

k8s-node1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
k8s-node2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
k8s-node3 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
k8s-node4 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

If any node returns UNREACHABLE, check SSH key distribution (the admin-lb.sh bootstrap script should have handled this) and verify network connectivity on the management network (192.168.10.0/24).

Executing the Deployment

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml \
  -e kube_version="1.32.9"

Command Breakdown:

• -i inventory/mycluster/inventory.ini — specifies the inventory file
• cluster.yml — the main deployment playbook
• -e kube_version="1.32.9" — overrides the Kubernetes version via CLI (highest precedence)

What Happens During Deployment

The cluster.yml playbook orchestrates a complex, multi-stage deployment process. At a high level, it proceeds through the following phases:

Phase 1: Prerequisite Checks and OS Configuration

• Validates the target OS (Rocky Linux 10, Ubuntu, etc.)
• Ensures required kernel modules are loaded (overlay, br_netfilter)
• Verifies sysctl settings (net.bridge.bridge-nf-call-iptables, net.ipv4.ip_forward)
• Disables swap (required by kubelet)
• Configures container runtime prerequisites

Phase 2: Container Runtime Installation

• Installs containerd as the CRI (Container Runtime Interface)
• Configures containerd with appropriate settings for Kubernetes
• Sets up the containerd systemd service

Phase 3: Download Binaries and Container Images

• Downloads kubeadm, kubelet, kubectl binaries to /tmp/releases/
• Pulls container images for all Kubernetes components:
• registry.k8s.io/kube-apiserver:v1.32.9
• registry.k8s.io/kube-controller-manager:v1.32.9
• registry.k8s.io/kube-scheduler:v1.32.9
• registry.k8s.io/kube-proxy:v1.32.9
• registry.k8s.io/pause:3.10
• registry.k8s.io/coredns/coredns:v1.12.0
• registry.k8s.io/etcd:3.5.25
• docker.io/flannel/flannel:v0.26.7
• docker.io/flannel/flannel-cni-plugin:v1.6.2

Phase 4: etcd Cluster Bootstrap

• Initializes etcd on the first control plane node (k8s-node1)
• Joins k8s-node2 and k8s-node3 to the etcd cluster
• Configures etcd with peer and client TLS certificates
• Verifies the 3-member etcd quorum

Phase 5: Control Plane Initialization

• Runs kubeadm init on k8s-node1 (the first control plane node)
• Generates static pod manifests for kube-apiserver, kube-controller-manager, kube-scheduler
• Creates the cluster CA certificates and kubeconfig files
• Joins k8s-node2 and k8s-node3 to the control plane via kubeadm join --control-plane

Phase 6: Worker Node Join

• Installs kubelet on worker nodes
• Uploads control plane CA certificates to workers
• Runs kubeadm join to register workers with the cluster
• Configures nginx-proxy static pods on workers (for client-side load balancing)

Phase 7: CNI Plugin and Add-ons

• Deploys Flannel DaemonSet across all nodes
• Deploys CoreDNS as a Deployment in kube-system namespace
• Deploys kube-proxy DaemonSet
• Deploys Metrics Server (if metrics_server_enabled: true)
• Applies node labels and taints as configured

Deployment Duration

The complete deployment takes approximately 8 minutes in this lab environment. The bulk of the time is spent in Phase 3 (downloading container images) and Phase 5 (waiting for control plane components to become healthy after each join).

Verifying Successful Deployment

Once the playbook completes without errors, copy the kubeconfig to the admin-lb node and verify:

# Copy kubeconfig from the first control plane node
mkdir -p /root/.kube
scp k8s-node1:/root/.kube/config /root/.kube/config

# Verify kubectl connectivity
kubectl get nodes -o wide

Expected output:

NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE              KERNEL-VERSION                CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   8m    v1.32.9   192.168.20.11   <none>        Rocky Linux 10.0 ..   6.12.x-xxx.el10.x86_64        containerd://2.1.5
k8s-node2   Ready    control-plane   7m    v1.32.9   192.168.20.12   <none>        Rocky Linux 10.0 ..   6.12.x-xxx.el10.x86_64        containerd://2.1.5
k8s-node3   Ready    control-plane   6m    v1.32.9   192.168.20.13   <none>        Rocky Linux 10.0 ..   6.12.x-xxx.el10.x86_64        containerd://2.1.5
k8s-node4   Ready    <none>          5m    v1.32.9   192.168.20.14   <none>        Rocky Linux 10.0 ..   6.12.x-xxx.el10.x86_64        containerd://2.1.5

Key things to verify:

• All 4 nodes show Ready status
• Control plane nodes have the control-plane role label
• All nodes report the correct Kubernetes version (v1.32.9)
• INTERNAL-IP shows the cluster network addresses (192.168.20.x), not the management or NAT addresses
• Container runtime is containerd://2.1.5

Post-Deployment Validation (Nodes, Pods, etcd, Certificates)

A successful cluster.yml run does not guarantee a fully healthy cluster. Thorough post-deployment validation is essential.

System Pod Health Check

kubectl get pods -n kube-system -o wide

Expected output (abbreviated):

NAME                                READY   STATUS    RESTARTS   AGE   IP              NODE
coredns-xxxxxxxxx-xxxxx             1/1     Running   0          8m    10.233.64.x     k8s-node1
coredns-xxxxxxxxx-xxxxx             1/1     Running   0          8m    10.233.64.x     k8s-node2
flannel-xxxxx                       1/1     Running   0          7m    192.168.20.11   k8s-node1
flannel-xxxxx                       1/1     Running   0          7m    192.168.20.12   k8s-node2
flannel-xxxxx                       1/1     Running   0          7m    192.168.20.13   k8s-node3
flannel-xxxxx                       1/1     Running   0          7m    192.168.20.14   k8s-node4
kube-apiserver-k8s-node1            1/1     Running   0          8m    192.168.20.11   k8s-node1
kube-apiserver-k8s-node2            1/1     Running   0          7m    192.168.20.12   k8s-node2
kube-apiserver-k8s-node3            1/1     Running   0          7m    192.168.20.13   k8s-node3
kube-controller-manager-k8s-node1   1/1     Running   0          8m    192.168.20.11   k8s-node1
kube-controller-manager-k8s-node2   1/1     Running   0          7m    192.168.20.12   k8s-node2
kube-controller-manager-k8s-node3   1/1     Running   0          7m    192.168.20.13   k8s-node3
kube-proxy-xxxxx                    1/1     Running   0          7m    192.168.20.11   k8s-node1
kube-proxy-xxxxx                    1/1     Running   0          7m    192.168.20.12   k8s-node2
kube-proxy-xxxxx                    1/1     Running   0          7m    192.168.20.13   k8s-node3
kube-proxy-xxxxx                    1/1     Running   0          7m    192.168.20.14   k8s-node4
kube-scheduler-k8s-node1            1/1     Running   0          8m    192.168.20.11   k8s-node1
kube-scheduler-k8s-node2            1/1     Running   0          7m    192.168.20.12   k8s-node2
kube-scheduler-k8s-node3            1/1     Running   0          7m    192.168.20.13   k8s-node3
metrics-server-xxxxxxxxx-xxxxx      1/1     Running   0          7m    10.233.64.x     k8s-node2
nginx-proxy-k8s-node4               1/1     Running   0          6m    192.168.20.14   k8s-node4

What to look for:

• Static pods (apiserver, controller-manager, scheduler): One instance per control plane node. These are not managed by a Deployment or DaemonSet — they are directly managed by the kubelet on each node based on manifest files in /etc/kubernetes/manifests/.
• Flannel DaemonSet: One pod per node (all 4 nodes). This provides the CNI overlay network.
• kube-proxy DaemonSet: One pod per node (all 4 nodes). This handles service-to-pod routing.
• CoreDNS Deployment: Two replicas (default) for DNS high availability.
• Metrics Server: One pod. Provides the metrics.k8s.io API.
• nginx-proxy: One static pod on each worker node only (k8s-node4). This is the client-side load balancer that distributes API requests from the worker to all three control plane nodes (covered in detail in Section 6).

etcd Cluster Health

SSH into any control plane node and use the etcdctl.sh wrapper script (installed by Kubespray) to inspect the etcd cluster:

# List all etcd members
ssh k8s-node1 etcdctl.sh member list -w table

Expected output:

+------------------+---------+-----------+----------------------------+----------------------------+
|        ID        | STATUS  |   NAME    |         PEER ADDRS         |        CLIENT ADDRS        |
+------------------+---------+-----------+----------------------------+----------------------------+
| 1a2b3c4d5e6f7890 | started | etcd1     | https://192.168.20.11:2380 | https://192.168.20.11:2379 |
| 2b3c4d5e6f789012 | started | etcd2     | https://192.168.20.12:2380 | https://192.168.20.12:2379 |
| 3c4d5e6f78901234 | started | etcd3     | https://192.168.20.13:2380 | https://192.168.20.13:2379 |
+------------------+---------+-----------+----------------------------+----------------------------+

# Check endpoint health and latency
ssh k8s-node1 etcdctl.sh endpoint status -w table

Expected output:

+----------------------------+------------------+---------+---------+-----------+-----------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM |
+----------------------------+------------------+---------+---------+-----------+-----------+
| https://192.168.20.11:2379 | 1a2b3c4d5e6f7890 | 3.5.25  |  5.4 MB |     true  |         2 |
| https://192.168.20.12:2379 | 2b3c4d5e6f789012 | 3.5.25  |  5.4 MB |    false  |         2 |
| https://192.168.20.13:2379 | 3c4d5e6f78901234 | 3.5.25  |  5.4 MB |    false  |         2 |
+----------------------------+------------------+---------+---------+-----------+-----------+

Key things to verify:

• All 3 members show started status
• etcd version is 3.5.25 (the version bundled with Kubespray v2.29.1)
• Exactly one member is the leader (IS LEADER: true)
• All members share the same RAFT TERM (consistent cluster state)
• Peer and client addresses use the cluster network (192.168.20.x)
• Peer communication uses port 2380 (TLS-encrypted peer-to-peer)
• Client communication uses port 2379 (TLS-encrypted client access)

API Server Endpoint Verification

Test direct API server connectivity from the admin-lb node:

# Test each control plane node's API server individually
curl -sk https://192.168.10.11:6443/version
curl -sk https://192.168.10.12:6443/version
curl -sk https://192.168.10.13:6443/version

Expected output (from each):

{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  "buildDate": "...",
  "goVersion": "go1.23.x",
  "compiler": "gc",
  "platform": "linux/amd64"
}

If all three return the correct version, the API servers are healthy and reachable from the management network.

Container Images on Nodes

Verify that all required container images were downloaded:

ssh k8s-node1 crictl images

Expected output:

IMAGE                                            TAG        IMAGE ID       SIZE
docker.io/flannel/flannel                        v0.26.7    xxxxxxxxxxxx   30.0MB
docker.io/flannel/flannel-cni-plugin             v1.6.2     xxxxxxxxxxxx   4.28MB
registry.k8s.io/coredns/coredns                  v1.12.0    xxxxxxxxxxxx   18.2MB
registry.k8s.io/etcd                             3.5.25     xxxxxxxxxxxx   64.8MB
registry.k8s.io/kube-apiserver                   v1.32.9    xxxxxxxxxxxx   30.5MB
registry.k8s.io/kube-controller-manager          v1.32.9    xxxxxxxxxxxx   28.7MB
registry.k8s.io/kube-proxy                       v1.32.9    xxxxxxxxxxxx   28.5MB
registry.k8s.io/kube-scheduler                   v1.32.9    xxxxxxxxxxxx   21.3MB
registry.k8s.io/metrics-server/metrics-server    v0.7.x     xxxxxxxxxxxx   17.6MB
registry.k8s.io/pause                            3.10       xxxxxxxxxxxx   320kB

Downloaded Binaries

Kubespray downloads Kubernetes binaries to /tmp/releases/ on each node:

ssh k8s-node1 tree /tmp/releases

Expected output:

/tmp/releases
├── kubeadm-v1.32.9-amd64
├── kubectl-v1.32.9-amd64
└── kubelet-v1.32.9-amd64

These binaries are cached on the node. During upgrades, new version binaries will be downloaded alongside the existing ones, and the active symlinks will be updated.

Kubernetes Certificate Inspection

Kubespray uses kubeadm to generate all cluster certificates. You can inspect them:

ssh k8s-node1 kubeadm certs check-expiration

This shows the expiration dates for all certificates. By default, kubeadm generates certificates with a 1-year validity period. Kubespray automatically renews certificates during cluster upgrades, so regular upgrades also serve as a certificate rotation mechanism — another reason not to defer upgrades.

CoreDNS Verification

Verify that cluster DNS resolution is working:

kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never -- \
  nslookup kubernetes.default.svc.cluster.local

Expected output:

Server:    10.233.0.3
Address 1: 10.233.0.3

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.233.0.1

This confirms:

• The kubelet is correctly passing --cluster-dns=10.233.0.3 to pods
• CoreDNS is running and reachable at the service IP 10.233.0.3
• The kubernetes service (the API server) correctly resolves to 10.233.0.1 (the first IP in the service CIDR)

Metrics Server Verification

# Check if the metrics API is available
kubectl top nodes

Expected output:

NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-node1   152m         7%     1124Mi          29%
k8s-node2   98m          4%     987Mi           25%
k8s-node3   87m          4%     945Mi           24%
k8s-node4   45m          2%     512Mi           13%

If kubectl top returns results, the Metrics Server is healthy and collecting resource utilization data from the kubelet's /metrics/resource endpoint on each node.

Understanding the K8S API Endpoint

Why API Endpoint Redundancy Matters

In a Kubernetes cluster, every component that interacts with the control plane ultimately talks to the kube-apiserver. This includes kubelet on every node, kube-proxy, kubectl commands from administrators, and any in-cluster workload that queries the Kubernetes API. If there is only a single kube-apiserver and it goes down, the entire cluster becomes unmanageable: new pods cannot be scheduled, services cannot be updated, and node health checks stop functioning.

When you deploy a multi-control-plane HA cluster (as we did in the previous section with three control plane nodes), you now have three kube-apiserver instances running on k8s-node1 (192.168.10.11), k8s-node2 (192.168.10.12), and k8s-node3 (192.168.10.13). But having multiple API servers alone is not enough — every client component on every node needs a strategy for how to reach these API servers and what to do when one of them fails.

This is the API endpoint problem, and Kubespray provides three distinct approaches to solving it.

Case 1: Client-Side Load Balancing with NGINX Static Pods

This is Kubespray’s default behavior when no external load balancer is configured. In this mode, Kubespray deploys an NGINX reverse proxy as a static pod on every worker node. This proxy listens on localhost:6443 and round-robins traffic to all control plane nodes.

How Worker-Side NGINX Proxy Works

When you deploy a cluster with Kubespray’s default settings, the following variable is implicitly set to true:

# inventory/mycluster/group_vars/all/all.yml
loadbalancer_apiserver_localhost: true  # default

With this setting enabled, Kubespray performs the following on every worker node during cluster deployment:

1. Generates an NGINX configuration file at /etc/nginx/nginx.conf that defines an upstream block pointing to all control plane nodes.
2. Creates a static pod manifest at /etc/kubernetes/manifests/nginx-proxy.yaml that runs an NGINX container using the host network.
3. Configures kubelet on the worker to use https://localhost:6443 as its API server endpoint.
4. Configures kube-proxy to also connect through https://localhost:6443.

The resulting architecture looks like this:

┌─────────────────────────────────────────────────────────────┐
│                        Worker Node (k8s-node4)              │
│                                                             │
│  ┌──────────┐      ┌────────────────────┐                   │
│  │ kubelet   │─────▶│  nginx-proxy       │                   │
│  │ kube-proxy│─────▶│  (static pod)      │                   │
│  └──────────┘      │  localhost:6443     │                   │
│                     └────────┬───────────┘                   │
└──────────────────────────────┼───────────────────────────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                 ▼
     ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
     │  k8s-node1   │ │  k8s-node2   │ │  k8s-node3   │
     │  apiserver   │ │  apiserver   │ │  apiserver   │
     │ :6443        │ │ :6443        │ │ :6443        │
     └──────────────┘ └──────────────┘ └──────────────┘

Because the NGINX proxy runs as a static pod directly on the worker node, no external infrastructure is required. Each worker independently manages its own API server connection pool.

You can verify the nginx-proxy pod is running on each worker node:

$ kubectl get pods -A -o wide | grep nginx-proxy
kube-system   nginx-proxy-k8s-node4   1/1   Running   0   12m   192.168.10.14   k8s-node4   <none>
kube-system   nginx-proxy-k8s-node5   1/1   Running   0   12m   192.168.10.15   k8s-node5   <none>

Note that control plane nodes do not run nginx-proxy — they connect directly to their own local kube-apiserver instance since kube-apiserver is already running on localhost:6443.

Configuration Deep Dive (nginx.conf.j2)

SSH into a worker node and inspect the generated NGINX configuration:

$ ssh k8s-node4 cat /etc/nginx/nginx.conf

error_log stderr notice;

worker_processes 1;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;

events {
  multi_accept on;
  use epoll;
  worker_connections 16384;
}

stream {
  upstream kube_apiserver {
    least_conn;
    server 192.168.10.11:6443;
    server 192.168.10.12:6443;
    server 192.168.10.13:6443;
  }

  server {
    listen        127.0.0.1:6443;
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

Key points to understand about this configuration:

• stream block (not http): This is a Layer 4 (TCP) proxy, not Layer 7 (HTTP). NGINX forwards raw TCP connections without inspecting HTTP headers or TLS content. This is critical because the TLS handshake happens directly between the client (kubelet) and kube-apiserver — NGINX never terminates TLS.
• least_conn load balancing: Distributes connections to the backend with the fewest active connections. This provides better distribution than round-robin when some API requests take longer than others.
• listen 127.0.0.1:6443: The proxy only listens on localhost. It is not accessible from outside the node.
• proxy_connect_timeout 1s: If a control plane node is unreachable, NGINX will fail over to the next backend within 1 second.
• proxy_timeout 10m: Long-lived connections (such as kubectl exec or watch streams) are kept alive for up to 10 minutes.

Now inspect the static pod manifest:

$ ssh k8s-node4 cat /etc/kubernetes/manifests/nginx-proxy.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-proxy
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-nginx
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  nodeSelector:
    kubernetes.io/os: linux
  priorityClassName: system-node-critical
  containers:
    - name: nginx-proxy
      image: registry.k8s.io/pause:3.10
      resources:
        requests:
          cpu: 25m
          memory: 32M
      # ... volume mounts for /etc/nginx/nginx.conf

The key setting here is hostNetwork: true, which means the pod uses the node's network namespace directly. This allows the NGINX process to bind to 127.0.0.1:6443 on the host, making it transparently available to kubelet and other local processes.

Verify that kubelet on the worker is indeed pointing to localhost:

$ ssh k8s-node4 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://localhost:6443

Similarly, check the kube-proxy configuration:

$ kubectl get configmap kube-proxy -n kube-system -o yaml | grep server
    server: https://localhost:6443

Failure Simulation: Control Plane Node Down

To verify that client-side load balancing works correctly during a control plane failure, we can stop one of the control plane nodes and observe that the cluster continues operating normally.

Step 1: Check current cluster state

$ kubectl get nodes
NAME        STATUS   ROLES           AGE   VERSION
k8s-node1   Ready    control-plane   35m   v1.32.9
k8s-node2   Ready    control-plane   33m   v1.32.9
k8s-node3   Ready    control-plane   31m   v1.32.9
k8s-node4   Ready    <none>          28m   v1.32.9
k8s-node5   Ready    <none>          28m   v1.32.9

Step 2: Stop kube-apiserver on k8s-node1

# On k8s-node1, move the static pod manifest to stop kube-apiserver
$ ssh k8s-node1 mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/

Step 3: Verify the cluster is still functional

# From admin-lb, run kubectl commands — they should still work
$ kubectl get nodes
NAME        STATUS     ROLES           AGE   VERSION
k8s-node1   Ready      control-plane   36m   v1.32.9
k8s-node2   Ready      control-plane   34m   v1.32.9
k8s-node3   Ready      control-plane   32m   v1.32.9
k8s-node4   Ready      <none>          29m   v1.32.9
k8s-node5   Ready      <none>          29m   v1.32.9

The cluster remains fully operational because the NGINX proxy on each worker automatically routes traffic to the remaining two API servers (k8s-node2 and k8s-node3). The proxy_connect_timeout 1s setting ensures the failover happens almost instantly.

You can observe this in the NGINX error log on a worker:

$ ssh k8s-node4 crictl logs $(ssh k8s-node4 crictl ps --name nginx-proxy -q) 2>&1 | tail
... connect() failed (111: Connection refused) while connecting to upstream,
    upstream: "192.168.10.11:6443" ...

NGINX detected that 192.168.10.11:6443 is unreachable and stopped routing traffic to it.

Step 4: Restore the API server

$ ssh k8s-node1 mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

Within a few seconds, kubelet on k8s-node1 will detect the manifest and restart kube-apiserver. NGINX on the workers will begin routing traffic to all three backends again.

Pros of Client-Side Load Balancing:

• No external infrastructure required (no separate load balancer to manage or maintain)
• Fast failover (1-second connect timeout per backend)
• Each worker is independently resilient — no shared single point of failure
• Works in air-gapped or restricted network environments

Cons of Client-Side Load Balancing:

• NGINX configuration must be updated on every worker when control plane nodes are added/removed (Kubespray handles this automatically during playbook runs)
• External clients (such as administrators running kubectl from their laptops) must target a specific control plane node IP — there is no single virtual IP for external access
• Certificate SAN management can become complex if you need external access through multiple entry points

Case 2: External LB (HAProxy) + Client-Side LB

In production environments, you typically want a single, stable endpoint that external clients (developers, CI/CD pipelines, monitoring systems) can use to reach the Kubernetes API. This is where an external load balancer comes in.

In Case 2, we add an external HAProxy load balancer on the admin-lb node (192.168.10.10) while keeping the client-side NGINX proxy on worker nodes. This gives you the best of both worlds: workers use the local NGINX proxy for resilient internal connectivity, and external users connect through HAProxy.

External Clients (kubectl, CI/CD, Monitoring)
                    │
                    ▼
        ┌───────────────────────┐
        │   admin-lb (HAProxy)  │
        │   192.168.10.10:6443  │
        └───────────┬───────────┘
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
  ┌──────────┐ ┌──────────┐ ┌──────────┐
  │ node1    │ │ node2    │ │ node3    │
  │ apiserver│ │ apiserver│ │ apiserver│
  └──────────┘ └──────────┘ └──────────┘
       ▲            ▲            ▲
       └────────────┼────────────┘
                    │
        ┌───────────────────────┐
        │  Worker nodes still   │
        │  use nginx-proxy      │
        │  (localhost:6443)     │
        └───────────────────────┘

Adding the External LB Endpoint

In our lab, HAProxy was already configured on the admin-lb node during the initial Vagrant provisioning (see Section 4). The HAProxy configuration in /etc/haproxy/haproxy.cfg includes:

frontend kubernetes-api
    bind *:6443
    mode tcp
    option tcplog
    default_backend kubernetes-api-backend

backend kubernetes-api-backend
    mode tcp
    option tcp-check
    balance roundrobin
    server k8s-node1 192.168.10.11:6443 check fall 3 rise 2
    server k8s-node2 192.168.10.12:6443 check fall 3 rise 2
    server k8s-node3 192.168.10.13:6443 check fall 3 rise 2

HAProxy performs Layer 4 TCP proxying, health-checking each backend every few seconds. If a control plane node fails health checks three times (fall 3), it is removed from the pool. When it passes two consecutive checks (rise 2), it is re-added.

You can verify HAProxy is forwarding correctly:

$ curl -sk https://192.168.10.10:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  ...
}

However, at this point, the kube-apiserver TLS certificate does not include 192.168.10.10 or the HAProxy domain name in its Subject Alternative Names (SANs). This means that any client performing proper TLS verification will reject the connection:

$ curl -v https://192.168.10.10:6443/version
* SSL: no alternative certificate subject name matches target host name '192.168.10.10'
curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.10.10'

We must update the certificates to include the external LB address.

Certificate SAN Update for External LB IP/Domain

To make the kube-apiserver certificate valid when accessed through the HAProxy IP (192.168.10.10) or a domain name (k8s-api-srv.admin-lb.com), we need to add these as supplementary SANs.

Step 1: Update the Kubespray group variables

Edit inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml and add the following:

## Supplementary addresses that can be added in kubernetes ssl keys.
## That can be useful for example to setup a determinate endpoint
## for FQDN registrations and target that endpoint with HAProxy.
supplementary_addresses_in_ssl_keys:
  - 192.168.10.10
  - k8s-api-srv.admin-lb.com

This tells Kubespray to include these additional addresses in the SAN field of the kube-apiserver’s TLS certificate.

Step 2: Apply the certificate update

You do not need to re-run the entire cluster.yml playbook. Instead, target only the control plane tag with a limit to control plane nodes:

$ cd /root/kubespray

$ ansible-playbook cluster.yml \
    --tags "control-plane" \
    --limit kube_control_plane \
    -e kube_version="1.32.9"

This playbook run will:

1. Regenerate the kube-apiserver TLS certificates with the new SANs
2. Restart kube-apiserver on each control plane node to pick up the new certificates
3. Leave everything else (etcd, worker nodes, CNI) untouched

The run typically completes in about 3–4 minutes.

Step 3: Verify the updated certificate

After the playbook completes, inspect the certificate SANs:

$ ssh k8s-node1 openssl x509 -in /etc/kubernetes/ssl/apiserver.crt -noout -text \
    | grep -A 20 "Subject Alternative Name"
            X509v3 Subject Alternative Name:
                DNS:k8s-node1, DNS:k8s-node2, DNS:k8s-node3,
                DNS:lb-apiserver.kubernetes.local,
                DNS:localhost, DNS:kubernetes, DNS:kubernetes.default,
                DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local,
                DNS:k8s-api-srv.admin-lb.com,
                IP Address:10.233.0.1, IP Address:192.168.10.11,
                IP Address:192.168.10.12, IP Address:192.168.10.13,
                IP Address:127.0.0.1, IP Address:192.168.10.10

The new entries k8s-api-srv.admin-lb.com and 192.168.10.10 are now present in the certificate SANs.

Step 4: Test external access through HAProxy

# Using IP address
$ curl -sk https://192.168.10.10:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  ...
}

# Using domain name (ensure DNS or /etc/hosts resolves this)
$ curl -sk https://k8s-api-srv.admin-lb.com:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  ...
}

Both endpoints now work without TLS errors.

Step 5: Update the admin kubeconfig to use the external LB

Now that HAProxy is a valid entry point, update the admin kubeconfig on the admin-lb node to use it:

# Copy the kubeconfig from a control plane node
$ scp k8s-node1:/etc/kubernetes/admin.conf /root/.kube/config

# Update the server URL to point to HAProxy
$ kubectl config set-cluster cluster.local \
    --server=https://k8s-api-srv.admin-lb.com:6443

Or manually edit ~/.kube/config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-ca-cert>
    server: https://k8s-api-srv.admin-lb.com:6443  # Changed from https://192.168.10.11:6443
  name: cluster.local

Now all kubectl commands from the admin-lb node will go through HAProxy, which distributes them across all three control plane nodes.

Failure Simulation with External LB

With the external LB in place, let’s simulate a more realistic failure scenario.

Step 1: Observe HAProxy backend status

Open the HAProxy stats page in a browser:

http://192.168.10.10:9000/haproxy_stats

You should see all three backends (k8s-node1, k8s-node2, k8s-node3) in a green/UP state.

Step 2: Shut down an entire control plane node

# From the host machine (outside the VMs)
$ vagrant halt k8s-node1

Step 3: Verify HAProxy detects the failure

Refresh the HAProxy stats page. Within a few seconds (determined by fall 3 and the health check interval), k8s-node1 will transition to a red/DOWN state. The remaining two backends stay UP.

Step 4: Verify cluster operations continue

$ kubectl get nodes
NAME        STATUS     ROLES           AGE   VERSION
k8s-node1   NotReady   control-plane   45m   v1.32.9
k8s-node2   Ready      control-plane   43m   v1.32.9
k8s-node3   Ready      control-plane   41m   v1.32.9
k8s-node4   Ready      <none>          38m   v1.32.9
k8s-node5   Ready      <none>          38m   v1.32.9

$ kubectl create deployment test-nginx --image=nginx --replicas=2
deployment.apps/test-nginx created

$ kubectl get pods -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP            NODE
test-nginx-7c79c4bf97-abc12   1/1     Running   0          10s   10.233.90.5   k8s-node4
test-nginx-7c79c4bf97-def34   1/1     Running   0          10s   10.233.91.3   k8s-node5

The cluster is fully operational. Both HAProxy (for external access) and the NGINX proxy on workers (for internal communication) have seamlessly failed over to the remaining control plane nodes.

Step 5: Restore the node

$ vagrant up k8s-node1

After the node boots and kubelet starts, it will rejoin the cluster. HAProxy will detect the restored backend via health checks (rise 2) and begin routing traffic to it again.

# Clean up the test deployment
$ kubectl delete deployment test-nginx

Summary of Case 2:

In this configuration:

• Worker nodes use the local nginx-proxy (localhost:6443) — unchanged from Case 1
• External clients (kubectl, CI/CD, monitoring) use HAProxy (192.168.10.10:6443)
• Certificates include both the control plane IPs and the HAProxy IP/domain as SANs
• Two independent failover mechanisms protect the cluster: HAProxy for external traffic, nginx-proxy for internal traffic

This is the most common production configuration because it provides maximum resilience without requiring changes to the internal cluster networking model.

Case 3: External LB as Single Endpoint for All Components

In some environments, teams prefer a fully centralized approach where every component — including kubelet and kube-proxy on worker nodes — connects to the API server exclusively through the external load balancer. This eliminates the per-node NGINX proxy and creates a simpler, more uniform architecture.

All Clients (kubelet, kube-proxy, kubectl, CI/CD)
                    │
                    ▼
        ┌───────────────────────┐
        │   admin-lb (HAProxy)  │
        │   192.168.10.10:6443  │
        └───────────┬───────────┘
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
  ┌──────────┐ ┌──────────┐ ┌──────────┐
  │ node1    │ │ node2    │ │ node3    │
  │ apiserver│ │ apiserver│ │ apiserver│
  └──────────┘ └──────────┘ └──────────┘

In this mode, no nginx-proxy static pods exist on worker nodes. Every API request from every node passes through HAProxy.

Disabling Client-Side LB (loadbalancer_apiserver_localhost: false)

To switch to external-LB-only mode, update the Kubespray configuration with the following variables:

Edit inventory/mycluster/group_vars/all/all.yml:

## External LB configuration
## This domain name will be used in kubeconfigs, kubelet configs, and
## kube-proxy configs on all nodes.
apiserver_loadbalancer_domain_name: "k8s-api-srv.admin-lb.com"

## External load balancer address and port
loadbalancer_apiserver:
  address: 192.168.10.10
  port: 6443

## CRITICAL: Disable client-side (localhost) load balancing
## When set to false, nginx-proxy static pods will be removed from worker nodes
## and all components will use the external LB endpoint instead.
loadbalancer_apiserver_localhost: false

Make sure the supplementary_addresses_in_ssl_keys from Case 2 is still present in k8s-cluster.yml:

# inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
supplementary_addresses_in_ssl_keys:
  - 192.168.10.10
  - k8s-api-srv.admin-lb.com

Apply the changes:

Since we are changing a fundamental cluster networking parameter, this requires a broader playbook run:

$ cd /root/kubespray

$ ansible-playbook cluster.yml \
    -e kube_version="1.32.9"

This playbook run will:

1. Remove the nginx-proxy static pod manifest from all worker nodes
2. Remove the /etc/nginx/nginx.conf file from worker nodes
3. Regenerate kubelet configuration on all nodes to point to https://k8s-api-srv.admin-lb.com:6443
4. Update the kube-proxy ConfigMap server endpoint
5. Regenerate all kubeconfig files (admin.conf, kubelet.conf, scheduler.conf, controller-manager.conf) with the new server URL
6. Restart kubelet on all nodes to pick up the new configuration

⚠️ Important: This is a disruptive change. During the playbook run, worker nodes will be reconfigured to point to the external LB. If HAProxy is not running or the DNS name does not resolve, worker nodes will lose connectivity to the API server. Always verify HAProxy is healthy before applying this change.

Verifying kubelet and kube-proxy Endpoint Configuration

After the playbook completes, systematically verify that every component is pointing to the external LB.

1. Verify nginx-proxy pods are gone from worker nodes

$ kubectl get pods -A -o wide | grep nginx-proxy
# No output — nginx-proxy pods have been removed

Confirm the static pod manifest no longer exists on worker nodes:

$ ssh k8s-node4 ls /etc/kubernetes/manifests/
# nginx-proxy.yaml should NOT be listed

And the NGINX configuration file has been cleaned up:

$ ssh k8s-node4 cat /etc/nginx/nginx.conf
cat: /etc/nginx/nginx.conf: No such file or directory

2. Verify kubelet configuration on worker nodes

$ ssh k8s-node4 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://k8s-api-srv.admin-lb.com:6443

Previously (in Case 1 and Case 2), this was https://localhost:6443. Now it points directly to the external LB domain name.

Check another worker to confirm consistency:

$ ssh k8s-node5 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://k8s-api-srv.admin-lb.com:6443

3. Verify kubelet configuration on control plane nodes

Control plane nodes have a slightly different behavior. Depending on the Kubespray version and settings, control plane kubelet may point to localhost:6443 (directly to the local kube-apiserver) or to the external LB. Verify:

$ ssh k8s-node1 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://k8s-api-srv.admin-lb.com:6443

In Case 3, even control plane nodes route through the external LB for consistency (though the request may loop back to the local apiserver via HAProxy).

4. Verify kube-proxy configuration

$ kubectl get configmap kube-proxy -n kube-system -o yaml | grep server
    server: https://k8s-api-srv.admin-lb.com:6443

5. Verify admin kubeconfig

$ ssh k8s-node1 cat /etc/kubernetes/admin.conf | grep server
    server: https://k8s-api-srv.admin-lb.com:6443

6. Test API access from multiple paths

# Through HAProxy IP
$ curl -sk https://192.168.10.10:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  ...
}

# Through HAProxy domain name
$ curl -sk https://k8s-api-srv.admin-lb.com:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  ...
}

# Directly to a control plane node (still works — useful for debugging)
$ curl -sk https://192.168.10.11:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.9",
  ...
}

7. Verify cluster health

$ kubectl get nodes -o wide
NAME        STATUS   ROLES           AGE   VERSION    INTERNAL-IP      OS-IMAGE
k8s-node1   Ready    control-plane   50m   v1.32.9    192.168.10.11    Rocky Linux 10.0
k8s-node2   Ready    control-plane   48m   v1.32.9    192.168.10.12    Rocky Linux 10.0
k8s-node3   Ready    control-plane   46m   v1.32.9    192.168.10.13    Rocky Linux 10.0
k8s-node4   Ready    <none>          43m   v1.32.9    192.168.10.14    Rocky Linux 10.0
k8s-node5   Ready    <none>          43m   v1.32.9    192.168.10.15    Rocky Linux 10.0

$ kubectl get pods -A
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system   coredns-xxxxxxxxx-xxxxx             1/1     Running   0          50m
kube-system   coredns-xxxxxxxxx-xxxxx             1/1     Running   0          50m
kube-system   kube-apiserver-k8s-node1            1/1     Running   0          50m
kube-system   kube-apiserver-k8s-node2            1/1     Running   0          48m
kube-system   kube-apiserver-k8s-node3            1/1     Running   0          46m
kube-system   kube-controller-manager-k8s-node1   1/1     Running   0          50m
kube-system   kube-controller-manager-k8s-node2   1/1     Running   0          48m
kube-system   kube-controller-manager-k8s-node3   1/1     Running   0          46m
kube-system   kube-proxy-xxxxx                    1/1     Running   0          43m
kube-system   kube-proxy-xxxxx                    1/1     Running   0          43m
kube-system   kube-proxy-xxxxx                    1/1     Running   0          50m
kube-system   kube-proxy-xxxxx                    1/1     Running   0          48m
kube-system   kube-proxy-xxxxx                    1/1     Running   0          46m
kube-system   kube-scheduler-k8s-node1            1/1     Running   0          50m
kube-system   kube-scheduler-k8s-node2            1/1     Running   0          48m
kube-system   kube-scheduler-k8s-node3            1/1     Running   0          46m

Notice that there are no nginx-proxy-* pods in the listing. All traffic now flows through the external HAProxy.

Failure Domain Consideration in Case 3:

In this architecture, HAProxy becomes a critical single point of failure for the entire cluster. If the admin-lb node goes down:

• Worker nodes lose API server connectivity (kubelet cannot report node status)
• kube-proxy cannot receive Service/Endpoint updates
• kubectl commands from external clients fail
• New pods cannot be scheduled

In production, you would mitigate this by:

• Running HAProxy in a highly available pair with keepalived (VRRP) for a floating virtual IP
• Using a cloud provider’s managed load balancer (AWS NLB, GCP Internal LB, Azure LB)
• Deploying multiple HAProxy instances behind DNS round-robin

Choosing the Right Configuration for Your Environment

For most production deployments, Case 2 (External LB + Client-Side LB) is the recommended approach. It provides two independent failover paths, no single point of failure, and a stable external endpoint for administrative and CI/CD access. The slight additional complexity of running nginx-proxy on workers is fully managed by Kubespray and requires no manual intervention.

Case 3 (External LB Only) is appropriate when:

• You have a highly available load balancer infrastructure (cloud-managed LB, keepalived pair)
• You want a simpler mental model where all traffic follows a single path
• Your organization’s network team manages the LB and prefers centralized control

Case 1 (Client-Side Only) is best suited for:

• Development and testing environments
• Air-gapped or isolated networks with no LB infrastructure
• Temporary clusters where simplicity is prioritized over external accessibility

In the remaining sections of this guide, we will continue with the Case 2 configuration (external HAProxy + client-side NGINX proxy) as it is the most representative of real-world production setups.

Node Management

In a production Kubernetes cluster, nodes are not static. Worker nodes need to be scaled out to handle increased workloads, decommissioned when no longer needed, or replaced when hardware fails. Control plane nodes occasionally require rotation — for OS patching, hardware refresh, or disaster recovery. Kubespray provides dedicated playbooks for each of these lifecycle operations, making it possible to manage nodes declaratively and repeatably.

This section walks through every major node management scenario: adding workers, gracefully removing them, force-removing unhealthy nodes that have gone offline, replacing a control plane node, and performing a full cluster reset.

Lab Context Recap

Our cluster currently has 3 control plane nodes (k8s-node1, k8s-node2, k8s-node3) and 1 worker node (k8s-node4). The admin/LB node (admin-lb) runs HAProxy and Kubespray. Kubernetes version is v1.32.9.

Adding a Worker Node (scale.yml)

When your cluster needs more compute capacity, Kubespray’s scale.yml playbook lets you add new worker nodes without disrupting existing workloads. Unlike cluster.yml, which operates on the entire cluster, scale.yml is purpose-built for incremental node addition — it only touches the new node and the minimal set of existing resources needed to integrate it.

Scale Playbook Walkthrough

Before running the playbook, let’s understand what scale.yml does under the hood. The playbook executes the following sequence of operations:

1. Download — Pulls the required container images and binaries to the new node (containerd images, CNI plugins, kubelet binary, kubeadm binary).
2. etcd (conditional) — If the new node is an etcd member (not the case for worker-only additions), it would configure and join the etcd cluster. For worker nodes, this step is skipped entirely.
3. Install kubelet — Installs and configures the kubelet service on the new node, including the kubelet configuration file, systemd unit, and certificate bootstrap token.
4. Upload control plane certificates — Copies the necessary CA certificates from an existing control plane node so the new worker can establish trust with the API server.
5. kubeadm join — Runs kubeadm join on the new node, which registers it with the cluster and starts the kubelet.
6. Apply labels and taints — Applies any node labels or taints defined in the inventory.
7. Configure CNI — Deploys the CNI plugin (Flannel in our case) so the node can participate in the pod network.

Important: scale.yml can only be used for adding worker nodes. You cannot use scale.yml to add new control plane nodes. For control plane additions, you must use cluster.yml — this is covered in Section 7.4.

Step-by-Step Execution and Verification

Step 1: Update the Inventory

First, add the new node to inventory.ini. We'll add k8s-node5 as a worker:

# inventory/mycluster/inventory.ini

[all]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15  # <-- NEW

[kube_control_plane]
k8s-node1
k8s-node2
k8s-node3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4
k8s-node5  # <-- NEW

[k8s_cluster:children]
kube_control_plane
kube_node

Step 2: Run the Scale Playbook

Execute scale.yml with the --limit flag to target only the new node. Always specify the Kubernetes version explicitly to ensure consistency:

cd /root/kubespray

ansible-playbook scale.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  --limit=k8s-node5 \
  -e kube_version="1.32.9"

The playbook typically completes in about 3 minutes for a single worker node. You’ll see output similar to:

PLAY RECAP ***********************************************************************
k8s-node5                  : ok=198  changed=62   unreachable=0    failed=0    skipped=412  rescued=0    ignored=0

Step 3: Verify the New Node

Once the playbook completes, verify the node has joined the cluster:

kubectl get nodes -o wide

NAME        STATUS   ROLES           AGE    VERSION   INTERNAL-IP      OS-IMAGE            KERNEL-VERSION
k8s-node1   Ready    control-plane   1d     v1.32.9   192.168.10.11    Rocky Linux 10.0    ...
k8s-node2   Ready    control-plane   1d     v1.32.9   192.168.10.12    Rocky Linux 10.0    ...
k8s-node3   Ready    control-plane   1d     v1.32.9   192.168.10.13    Rocky Linux 10.0    ...
k8s-node4   Ready    <none>          1d     v1.32.9   192.168.10.14    Rocky Linux 10.0    ...
k8s-node5   Ready    <none>          30s    v1.32.9   192.168.10.15    Rocky Linux 10.0    ...

The new k8s-node5 should appear with STATUS: Ready and the correct Kubernetes version. You can also verify that the CNI plugin is running:

kubectl get pods -n kube-system -o wide | grep flannel

kube-flannel-xxxxx   1/1   Running   0   45s   192.168.10.15   k8s-node5   ...

If you’re using client-side load balancing (the default Kubespray behavior), you’ll also see an nginx-proxy static pod on the new worker:

kubectl get pods -n kube-system -o wide | grep nginx-proxy

nginx-proxy-k8s-node4   1/1   Running   0   1d    192.168.10.14   k8s-node4   ...
nginx-proxy-k8s-node5   1/1   Running   0   45s   192.168.10.15   k8s-node5   ...

This nginx-proxy is configured to load-balance API requests across all three control plane nodes, ensuring the new worker has full HA connectivity to the API server from the moment it joins.

Removing a Worker Node (remove-node.yml)

When decommissioning a worker node — whether for cost optimization, hardware retirement, or cluster right-sizing — Kubespray’s remove-node.yml playbook handles the full lifecycle: draining workloads, cleaning up cluster metadata, and resetting the node.

Graceful Removal with PDB Considerations

Before diving into the removal process, it’s critical to understand how PodDisruptionBudget (PDB) objects can affect node removal.

When Kubespray drains a node, it runs the equivalent of kubectl drain --ignore-daemonsets --delete-emptydir-data. This eviction process respects PDB constraints. If a PDB specifies maxUnavailable: 0 for a set of pods, the drain operation will block indefinitely because Kubernetes refuses to evict pods that would violate the disruption budget.

Example scenario: Suppose you have a deployment with 2 replicas and a PDB that says maxUnavailable: 0:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      app: my-app

If both replicas happen to be running on the node you’re trying to drain (or if draining would bring the available count below the PDB requirement), the drain command will hang. In the Kubespray playbook output, you’ll see the drain task stall with no progress.

Mitigation strategies:

• Before removing a node, check for PDBs: kubectl get pdb --all-namespaces
• Temporarily adjust the PDB to allow disruption: maxUnavailable: 1
• Or manually reschedule pods off the target node before starting the removal
• Use kubectl drain <node> --timeout=300s to set a timeout on the drain (Kubespray's default behavior includes a drain timeout, but verify your configuration)

Remove Playbook Walkthrough

The remove-node.yml playbook executes the following steps:

1. Confirmation prompt — Unless skip_confirmation=true is passed, the playbook asks for manual confirmation before proceeding. This is a safety mechanism to prevent accidental node removal.
2. Drain the node — Cordons the node (marks it as unschedulable) and evicts all non-DaemonSet pods. Workloads are rescheduled to other available nodes.
3. Remove etcd member — If the node is an etcd member, it is removed from the etcd cluster. For worker-only nodes, this step is skipped.
4. kubeadm reset — Runs kubeadm reset on the target node, which tears down the kubelet, removes certificates, and cleans up local Kubernetes state.
5. Delete node metadata — Removes the Node object from the Kubernetes API so it no longer appears in kubectl get nodes.

Execution:

Let’s remove the k8s-node5 worker node we just added:

cd /root/kubespray

ansible-playbook remove-node.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  -e node=k8s-node5 \
  -e skip_confirmation=true \
  -e kube_version="1.32.9"

Key parameters:

Parameter Description -e node=k8s-node5 Specifies which node to remove. Can be a comma-separated list for multiple nodes. -e skip_confirmation=true Bypasses the interactive confirmation prompt. Useful for automation.

The playbook typically completes in about 2 minutes. Output:

PLAY RECAP ***********************************************************************
k8s-node1                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
k8s-node5                  : ok=15   changed=9    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0

Notice that k8s-node1 (a control plane node) also shows activity — this is because the playbook connects to a control plane node to execute kubectl drain and kubectl delete node commands.

Verification:

kubectl get nodes -o wide

NAME        STATUS   ROLES           AGE    VERSION   INTERNAL-IP      OS-IMAGE            KERNEL-VERSION
k8s-node1   Ready    control-plane   1d     v1.32.9   192.168.10.11    Rocky Linux 10.0    ...
k8s-node2   Ready    control-plane   1d     v1.32.9   192.168.10.12    Rocky Linux 10.0    ...
k8s-node3   Ready    control-plane   1d     v1.32.9   192.168.10.13    Rocky Linux 10.0    ...
k8s-node4   Ready    <none>          1d     v1.32.9   192.168.10.14    Rocky Linux 10.0    ...

k8s-node5 is gone. Any pods that were running on it have been rescheduled to k8s-node4 or one of the control plane nodes (if they tolerate control plane taints).

Don’t forget to update the inventory. After removing a node, clean up inventory.ini by removing the k8s-node5 entry from both the [all] and [kube_node] groups. While Kubespray won't break if a removed node remains in the inventory, keeping the inventory in sync with reality is a best practice for Day-2 operations.

Force-Removing an Unhealthy Node

Not all node removals are graceful. In production, you will encounter scenarios where a node has suffered a catastrophic failure — hardware crash, kernel panic, network partition, or a cloud instance that simply vanished. In these cases, you can’t SSH into the node to run kubeadm reset, and a normal remove-node.yml run will fail because Ansible can't reach the target.

Kubespray handles this with two special flags: reset_nodes=false and allow_ungraceful_removal=true.

Simulating a Node Failure

To demonstrate force removal, let’s simulate a hard failure of k8s-node5. First, add it back to the cluster (if you removed it in the previous section), then force-stop the VM:

# On the Vagrant host (not admin-lb)
vagrant halt k8s-node5 --force

Or if you’re simulating in a different environment, simply power off the machine:

# On the target node itself (before it goes down)
sudo systemctl poweroff --force

After the node goes offline, Kubernetes will eventually mark it as NotReady:

kubectl get nodes

NAME        STATUS     ROLES           AGE    VERSION
k8s-node1   Ready      control-plane   1d     v1.32.9
k8s-node2   Ready      control-plane   1d     v1.32.9
k8s-node3   Ready      control-plane   1d     v1.32.9
k8s-node4   Ready      <none>          1d     v1.32.9
k8s-node5   NotReady   <none>          10m    v1.32.9

The node transitions to NotReady after the kubelet stops sending heartbeats (default: 40 seconds for the node controller to mark it, then pod-eviction-timeout for pod rescheduling).

Using reset_nodes=false and allow_ungraceful_removal=true

Now attempt a normal removal — it will fail because Ansible cannot SSH into the offline node:

# This will FAIL:
ansible-playbook remove-node.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  -e node=k8s-node5 \
  -e skip_confirmation=true

fatal: [k8s-node5]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh..."}

To force-remove the node without SSH access, use the ungraceful removal flags:

ansible-playbook remove-node.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  -e node=k8s-node5 \
  -e skip_confirmation=true \
  -e reset_nodes=false \
  -e allow_ungraceful_removal=true

Key parameters:

Parameter Description -e reset_nodes=false Skips the kubeadm reset step on the target node. Since the node is unreachable, there's nothing to reset remotely. -e allow_ungraceful_removal=true Permits removing the node from the cluster even though it couldn't be drained or cleaned up. Kubespray will only remove the cluster-side metadata (Node object, etcd member if applicable).

What happens during ungraceful removal:

1. Drain is skipped — Since the node is unreachable, the drain step is bypassed. Pods that were running on the dead node will remain in Terminating state until the garbage collection timeout expires (or until the Node object is deleted).
2. kubeadm reset is skipped — No SSH to the node, so no local cleanup.
3. Node metadata is deleted — The Node object is removed from the Kubernetes API. Once deleted, all pods that were “stuck” on that node are freed for rescheduling by their respective controllers (Deployments, StatefulSets, etc.).

Verification:

kubectl get nodes

NAME        STATUS   ROLES           AGE    VERSION
k8s-node1   Ready    control-plane   1d     v1.32.9
k8s-node2   Ready    control-plane   1d     v1.32.9
k8s-node3   Ready    control-plane   1d     v1.32.9
k8s-node4   Ready    <none>          1d     v1.32.9

The NotReady node is gone. Workloads are rescheduled.

Caution: If the “dead” node later comes back online (e.g., after a network partition heals), its kubelet will try to re-register with the API server using its old certificates. This can cause unexpected behavior. Best practice is to fully wipe and reprovision the node before allowing it to rejoin the cluster, or ensure its kubelet service is disabled/stopped.

Replacing a Control Plane Node

Replacing a control plane node is the most complex node management operation. Unlike workers, control plane nodes host critical static pods (kube-apiserver, kube-controller-manager, kube-scheduler) and may be etcd members. The replacement process involves:

1. Removing the old control plane node (including its etcd membership)
2. Updating the inventory
3. Adding the new node as a control plane member using cluster.yml

Kubespray handles most of the complexity, but there are important ordering constraints to be aware of.

Removing a Control Plane and Worker Node

Let’s say k8s-node3 (our third control plane node) needs to be replaced. First, we remove it from the cluster:

ansible-playbook remove-node.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  -e node=k8s-node3 \
  -e skip_confirmation=true \
  -e kube_version="1.32.9"

This playbook will:

1. Drain k8s-node3 — evict all pods (including those scheduled by tolerations)
2. Remove k8s-node3 from the etcd cluster — runs etcdctl member remove to safely remove it from the quorum
3. Run kubeadm reset on k8s-node3 — cleans up certificates, static pod manifests, and kubelet state
4. Delete the Node object from the API

During etcd member removal, the cluster transitions from a 3-member to a 2-member etcd cluster. This is still operational (2 out of 3 is quorum), but you’ve lost fault tolerance — a 2-member etcd cluster cannot tolerate any additional failures. This is why you should minimize the time window between removing and re-adding a control plane node.

Verify the etcd cluster state after removal:

ssh k8s-node1 etcdctl.sh member list -w table

+------------------+---------+-----------+---------------------------+---------------------------+
|        ID        | STATUS  |   NAME    |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+-----------+---------------------------+---------------------------+
| 1a2b3c4d5e6f7890 | started | k8s-node1 | https://192.168.10.11:2380| https://192.168.10.11:2379|
| 2b3c4d5e6f789012 | started | k8s-node2 | https://192.168.10.12:2380| https://192.168.10.12:2379|
+------------------+---------+-----------+---------------------------+---------------------------+

Only 2 members remain. Let’s also check the Kubernetes node list:

kubectl get nodes -o wide

NAME        STATUS   ROLES           AGE    VERSION   INTERNAL-IP
k8s-node1   Ready    control-plane   1d     v1.32.9   192.168.10.11
k8s-node2   Ready    control-plane   1d     v1.32.9   192.168.10.12
k8s-node4   Ready    <none>          1d     v1.32.9   192.168.10.14

k8s-node3 is completely gone from both the Kubernetes cluster and the etcd cluster.

Critical Limitation: The first node listed in the [kube_control_plane] group (typically k8s-node1) cannot be removed using remove-node.yml. This node is treated as the "initial control plane" by Kubespray and kubeadm, and its removal requires a full cluster rebuild. If you need to replace node1, you must first add a new control plane node, then use cluster.yml with a reconfigured inventory that lists the new node first.

Re-adding a Node as Control Plane (cluster.yml)

Now we’ll add k8s-node3 back, but this time as a fresh control plane member. After reprovisioning the VM (or ensuring the old state has been cleaned up via kubeadm reset), update the inventory.

Critical: New control plane nodes must be added at the END of the [kube_control_plane] group. Kubespray and kubeadm treat the first node in the group as the "initial" control plane. If you insert a new node before existing members, you risk breaking the cluster.

# inventory/mycluster/inventory.ini

[all]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14

[kube_control_plane]
k8s-node1
k8s-node2
k8s-node3  # <-- Re-added at the END

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4

[k8s_cluster:children]
kube_control_plane
kube_node

Now run cluster.yml (not scale.yml!) to add the control plane node:

ansible-playbook cluster.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  -e kube_version="1.32.9"

Why cluster.yml and not scale.yml? The scale.yml playbook only handles worker node additions. Control plane nodes require the full cluster.yml playbook because they need:

• etcd member addition and data synchronization
• Static pod manifest generation for apiserver, controller-manager, scheduler
• Certificate generation and distribution
• kubeadm control plane join (different from worker join)

The playbook is idempotent — it will skip tasks for k8s-node1, k8s-node2, and k8s-node4 that are already configured, and only perform meaningful work on k8s-node3. However, running cluster.yml does touch all nodes (unlike scale.yml --limit), so expect it to take longer than a simple worker addition.

etcd Member Changes and NGINX Config Updates

After cluster.yml completes, verify that the control plane is fully restored:

etcd cluster — back to 3 members:

ssh k8s-node1 etcdctl.sh member list -w table

+------------------+---------+-----------+---------------------------+---------------------------+
|        ID        | STATUS  |   NAME    |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+-----------+---------------------------+---------------------------+
| 1a2b3c4d5e6f7890 | started | k8s-node1 | https://192.168.10.11:2380| https://192.168.10.11:2379|
| 2b3c4d5e6f789012 | started | k8s-node2 | https://192.168.10.12:2380| https://192.168.10.12:2379|
| 3c4d5e6f78901234 | started | k8s-node3 | https://192.168.10.13:2380| https://192.168.10.13:2379|
+------------------+---------+-----------+---------------------------+---------------------------+

The new k8s-node3 etcd member has automatically joined and synchronized data from the existing members.

Verify etcd endpoint health:

ssh k8s-node1 etcdctl.sh endpoint status -w table

+---------------------------+------------------+---------+---------+-----------+-----------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM |
+---------------------------+------------------+---------+---------+-----------+-----------+
| https://192.168.10.11:2379| 1a2b3c4d5e6f7890 | 3.5.25  |   25 MB |   false   |        4  |
| https://192.168.10.12:2379| 2b3c4d5e6f789012 | 3.5.25  |   25 MB |    true   |        4  |
| https://192.168.10.13:2379| 3c4d5e6f78901234 | 3.5.25  |   25 MB |   false   |        4  |
+---------------------------+------------------+---------+---------+-----------+-----------+

All three endpoints are healthy with consistent DB sizes.

Kubernetes nodes — 3 control planes restored:

kubectl get nodes -o wide

NAME        STATUS   ROLES           AGE    VERSION   INTERNAL-IP
k8s-node1   Ready    control-plane   1d     v1.32.9   192.168.10.11
k8s-node2   Ready    control-plane   1d     v1.32.9   192.168.10.12
k8s-node3   Ready    control-plane   60s    v1.32.9   192.168.10.13
k8s-node4   Ready    <none>          1d     v1.32.9   192.168.10.14

NGINX proxy configuration on workers (automatic update):

If you’re using client-side load balancing (the default), Kubespray automatically updates the nginx.conf on all worker nodes to include the new control plane node. You can verify this by checking the nginx configuration on k8s-node4:

ssh k8s-node4 cat /etc/nginx/nginx.conf

stream {
    upstream kube_apiserver {
        least_conn;
        server 192.168.10.11:6443;
        server 192.168.10.12:6443;
        server 192.168.10.13:6443;  # <-- Restored
    }

    server {
        listen 127.0.0.1:6443;
        proxy_pass kube_apiserver;
        proxy_timeout 10m;
        proxy_connect_timeout 1s;
    }
}

All three control plane IPs are present in the upstream block. The nginx-proxy static pod on k8s-node4 will automatically reload with this updated configuration, ensuring the worker's API requests are distributed across all three control plane nodes.

Static pods on the new control plane node:

kubectl get pods -n kube-system -o wide | grep k8s-node3

etcd-k8s-node3                      1/1   Running   0   60s   192.168.10.13   k8s-node3
kube-apiserver-k8s-node3             1/1   Running   0   60s   192.168.10.13   k8s-node3
kube-controller-manager-k8s-node3    1/1   Running   0   60s   192.168.10.13   k8s-node3
kube-scheduler-k8s-node3             1/1   Running   0   60s   192.168.10.13   k8s-node3

All four critical static pods are running on the restored control plane node.

If using HAProxy (external LB):

Remember that HAProxy on admin-lb is configured independently of Kubespray. If your HAProxy backend already lists all three control plane IPs in /etc/haproxy/haproxy.cfg, no changes are needed — HAProxy will automatically start routing traffic to the new node once its health check passes:

backend k8s-api
    option  httpchk GET /healthz
    http-check expect status 200
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server k8s-node1 192.168.10.11:6443 check check-ssl verify none
    server k8s-node2 192.168.10.12:6443 check check-ssl verify none
    server k8s-node3 192.168.10.13:6443 check check-ssl verify none

You can verify HAProxy backend status at http://192.168.10.10:9000/haproxy_stats — all three backends should show as UP (green).

Full Cluster Reset (reset.yml)

When you need to tear down the entire Kubernetes cluster — for rebuilding from scratch, testing a fresh deployment, or decommissioning the infrastructure — Kubespray provides reset.yml. This playbook completely reverses everything that cluster.yml did: it stops all Kubernetes services, removes all binaries, cleans up configuration files, and leaves the nodes in a pre-Kubernetes state.

What reset.yml does on every node:

1. Drains and deletes all nodes from the Kubernetes API
2. Stops kubelet and removes the systemd unit
3. Stops and removes etcd data and binaries (on etcd members)
4. Runs kubeadm reset — removes certificates, static pod manifests, and kubeconfig files
5. Removes CNI configuration — cleans up /etc/cni/net.d/ and CNI binaries
6. Cleans up iptables rules — removes all Kubernetes-related iptables/ipvs rules
7. Removes container runtime artifacts — stops all running containers, removes containerd state
8. Deletes configuration directories — /etc/kubernetes/, /var/lib/kubelet/, /var/lib/etcd/, etc.

Execution:

cd /root/kubespray

ansible-playbook reset.yml \
  --become \
  -i inventory/mycluster/inventory.ini \
  -e skip_confirmation=true

Warning: This is a destructive, irreversible operation. All cluster data, including etcd state (and therefore all Kubernetes objects — deployments, services, secrets, configmaps, PVs, etc.) will be permanently deleted. Ensure you have backups of any critical data before running this playbook.

After reset.yml completes, the nodes are clean and ready for a fresh cluster.yml deployment. This makes reset.yml particularly useful in development and testing workflows where you frequently need to iterate on cluster configurations:

# Tear down
ansible-playbook reset.yml --become -i inventory/mycluster/inventory.ini -e skip_confirmation=true

# Rebuild with new settings
ansible-playbook cluster.yml --become -i inventory/mycluster/inventory.ini -e kube_version="1.32.9"

Verification after reset:

SSH into any node and confirm that Kubernetes components are gone:

ssh k8s-node1

# kubelet should not be running
systemctl status kubelet
# Unit kubelet.service could not be found.

# No Kubernetes directories
ls /etc/kubernetes/
# ls: cannot access '/etc/kubernetes/': No such file or directory

# No etcd data
ls /var/lib/etcd/
# ls: cannot access '/var/lib/etcd/': No such file or directory

# kubectl should fail (no kubeconfig)
kubectl get nodes
# The connection to the server localhost:8080 was refused

The node is now a blank slate, ready for reprovisioning.

Key takeaways

• Use scale.yml for workers, cluster.yml for control planes.
• Always add new control plane nodes at the end of the [kube_control_plane] group.
• The first node in [kube_control_plane] cannot be removed via remove-node.yml.
• For unreachable nodes, use reset_nodes=false + allow_ungraceful_removal=true.
• PodDisruptionBudgets can block node drains — check PDBs before removing nodes.
• Keep your inventory.ini in sync with the actual cluster state after every operation.
• Minimize the time window where etcd has fewer than 3 members to maintain fault tolerance.

8. Monitoring Setup

With a fully operational HA Kubernetes cluster — three control plane nodes, two workers, and an external HAProxy load balancer — the next critical step is observability. Without monitoring, you are flying blind: you cannot detect etcd quorum degradation, API server latency spikes, or node resource exhaustion until they become outages.

In this section, we will build a production-grade monitoring stack on top of our lab cluster. The setup consists of three layers:

1. Persistent storage — An NFS-based dynamic provisioner so that Prometheus and Grafana retain their data across pod restarts.
2. kube-prometheus-stack — A Helm-based deployment that bundles Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics into a single, cohesive package.
3. etcd metrics — Dedicated scrape configuration so that Prometheus can collect metrics directly from the etcd cluster running on the control plane nodes.

By the end of this section, you will have Prometheus scraping the Kubernetes API server, kubelet, node-exporter, HAProxy, and etcd, with Grafana dashboards providing real-time visibility into every layer of the cluster.

NFS Subdir External Provisioner for Persistent Storage

Why We Need a StorageClass

Both Prometheus and Grafana require persistent volumes. Prometheus stores its time-series database (TSDB) on disk, and Grafana persists dashboards, data sources, and user sessions. Without a StorageClass that supports dynamic provisioning, every PersistentVolumeClaim (PVC) would require a cluster administrator to manually create a matching PersistentVolume (PV)—impractical for a monitoring stack that may create multiple PVCs during installation.

In our lab environment, the admin-lb node (192.168.10.10) already runs an NFS server exporting /srv/nfs/share. This was configured during the initial bootstrap via the admin-lb.sh script. We will leverage this NFS export as the backing storage for all dynamic PVCs.

Installing the NFS Subdir External Provisioner

The nfs-subdir-external-provisioner is a Kubernetes controller that watches for new PVCs and automatically creates subdirectories on the NFS share as PVs. Each PVC gets its own directory named ${namespace}-${pvcName}-${pvName}, making it easy to identify which data belongs to which workload.

First, add the Helm repository and update:

helm repo add nfs-subdir-external-provisioner \
  https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update

Then install the provisioner, pointing it at our NFS server:

helm install nfs-provisioner \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set nfs.server=192.168.10.10 \
  --set nfs.path=/srv/nfs/share \
  --set storageClass.defaultClass=true

Verifying the Provisioner

After installation, confirm the StorageClass is created and set as the default:

kubectl get storageclass

Expected output:

NAME                   PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-provisioner-nfs-subdir-...    Delete          Immediate           true                   30s

The (default) annotation confirms that any PVC without an explicit storage class will be fulfilled by this provisioner. You can also verify the provisioner pod is running:

kubectl get pods -l app=nfs-subdir-external-provisioner

NAME                                              READY   STATUS    RESTARTS   AGE
nfs-provisioner-nfs-subdir-external-provisioner-xxx   1/1     Running   0          45s

At this point, any component that creates a PVC — including Prometheus and Grafana — will automatically get a dynamically provisioned NFS-backed persistent volume.

Installing kube-prometheus-stack with Helm

The kube-prometheus-stack Helm chart is the de facto standard for deploying a complete monitoring solution on Kubernetes. A single helm install gives you:

• Prometheus — Time-series database and scraping engine
• Grafana — Visualization and dashboarding platform
• Alertmanager — Alert routing and deduplication
• node-exporter — Host-level metrics (CPU, memory, disk, network) from every node
• kube-state-metrics — Kubernetes object-level metrics (deployments, pods, nodes, PVCs)
• Pre-built Grafana dashboards — Dozens of dashboards for Kubernetes, node, and Prometheus self-monitoring
• Pre-built PrometheusRules — Alert rules for common failure conditions

Adding the Helm Repository

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update

Custom Values (Prometheus, Grafana, Node Exporter)

Rather than installing with default settings, we create a custom values file to tailor the stack to our lab environment. The key customizations are:

1. NodePort services for Prometheus and Grafana — so we can access them from the host machine without an Ingress controller.
2. Additional scrape configurations — to collect metrics from HAProxy and etcd, which are not Kubernetes-native workloads.
3. Grafana admin password — set explicitly for reproducibility.
4. Node exporter tolerations — to ensure node-exporter runs on control plane nodes as well.

Create the values file:

cat <<'EOF' > ~/kubespray/custom-values.yaml
# ============================================================
# Prometheus Configuration
# ============================================================
prometheus:
  prometheusSpec:
    # Expose Prometheus via NodePort so we can access it from outside the cluster
    # Access URL: http://<any-node-ip>:30001
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

    # Persistent storage for TSDB
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

    # Additional scrape configs for non-k8s targets
    additionalScrapeConfigs:
      # --- HAProxy Metrics ---
      # HAProxy exposes a Prometheus-compatible metrics endpoint
      # on port 8405 (configured in haproxy.cfg with stats socket)
      - job_name: "haproxy"
        static_configs:
          - targets:
              - "192.168.10.10:8405"
        metrics_path: "/metrics"

      # --- etcd Metrics ---
      # etcd exposes metrics on port 2381 (configured via etcd_listen_metrics_urls)
      # We scrape all 3 etcd members individually
      - job_name: "etcd"
        static_configs:
          - targets:
              - "192.168.10.11:2381"
              - "192.168.10.12:2381"
              - "192.168.10.13:2381"
        metrics_path: "/metrics"

  service:
    type: NodePort
    nodePort: 30001

# ============================================================
# Grafana Configuration
# ============================================================
grafana:
  adminPassword: "prom-operator"

  service:
    type: NodePort
    nodePort: 30002

  # Persistent storage for Grafana dashboards and settings
  persistence:
    enabled: true
    size: 5Gi

  # Sidecar configuration: automatically imports ConfigMaps
  # with label grafana_dashboard="1" as dashboards
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      labelValue: "1"
      searchNamespace: ALL

# ============================================================
# Node Exporter Configuration
# ============================================================
nodeExporter:
  # Ensure node-exporter runs on ALL nodes, including control planes
  tolerations:
    - effect: NoSchedule
      operator: Exists

# ============================================================
# Alertmanager Configuration
# ============================================================
alertmanager:
  service:
    type: NodePort
    nodePort: 30003
EOF

Let us walk through each section in detail.

Prometheus Additional Scrape Configs

The additionalScrapeConfigs field is the mechanism for adding scrape targets that are not discovered via Kubernetes service discovery. In our setup, two external targets need explicit configuration:

HAProxy metrics (192.168.10.10:8405)

Our HAProxy configuration on the admin-lb node includes a Prometheus metrics endpoint. The relevant section in /etc/haproxy/haproxy.cfg is:

frontend stats
    bind *:8405
    http-request use-service prometheus-exporter if { path /metrics }

This exposes standard HAProxy metrics such as haproxy_frontend_current_sessions, haproxy_backend_up, haproxy_server_bytes_in_total, etc. These metrics are critical for monitoring the external load balancer that fronts our Kubernetes API servers.

etcd metrics (192.168.10.11-13:2381)

etcd does not expose metrics by default. We will enable this in Section 8.3. Once enabled, each etcd member exposes metrics on port 2381, including etcd_server_has_leader, etcd_disk_wal_fsync_duration_seconds, etcd_network_peer_round_trip_time_seconds, and etcd_mvcc_db_total_size_in_bytes.

Service Types and NodePorts

In a lab environment without an Ingress controller or cloud load balancer, NodePort is the simplest way to expose services externally. The fixed ports make access predictable:

Service NodePort Access URL Prometheus 30001 http://192.168.10.14:30001 Grafana 30002 http://192.168.10.14:30002 Alertmanager 30003 http://192.168.10.14:30003

You can use any node IP in the cluster (worker or control plane). The NodePort is accessible on every node regardless of where the pod is actually scheduled.

Grafana Sidecar Dashboard Loader

The sidecar.dashboards configuration tells Grafana to watch for ConfigMaps across all namespaces (searchNamespace: ALL) that carry the label grafana_dashboard: "1". When such a ConfigMap is created, the Grafana sidecar container automatically loads the JSON dashboard it contains—no manual import required. We will use this mechanism in Section 8.2.2 to add custom dashboards.

Installing the Stack

With the values file ready, install the chart:

helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --version 80.13.3 \
  -f ~/kubespray/custom-values.yaml

We pin the chart version to 80.13.3 for reproducibility. The --create-namespace flag creates the monitoring namespace if it does not already exist.

The installation takes approximately 2–3 minutes. Monitor the progress:

kubectl -n monitoring get pods -w

Wait until all pods reach Running or Completed status:

NAME                                                        READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-stack-alertmanager-0            2/2     Running   0          2m
kube-prometheus-stack-grafana-xxxxxxxxx-xxxxx                3/3     Running   0          2m
kube-prometheus-stack-kube-state-metrics-xxxxxxxxx-xxxxx     1/1     Running   0          2m
kube-prometheus-stack-operator-xxxxxxxxx-xxxxx               1/1     Running   0          2m
kube-prometheus-stack-prometheus-node-exporter-xxxxx         1/1     Running   0          2m
kube-prometheus-stack-prometheus-node-exporter-xxxxx         1/1     Running   0          2m
kube-prometheus-stack-prometheus-node-exporter-xxxxx         1/1     Running   0          2m
kube-prometheus-stack-prometheus-node-exporter-xxxxx         1/1     Running   0          2m
prometheus-kube-prometheus-stack-prometheus-0                2/2     Running   0          2m

Note that prometheus-node-exporter pods appear on every node (including control planes, thanks to our toleration). Verify services:

kubectl -n monitoring get svc

Confirm the NodePort assignments match our values file (30001 for Prometheus, 30002 for Grafana, 30003 for Alertmanager).

Accessing the Monitoring UIs

Prometheus — Open http://192.168.10.14:30001 in your browser.

Navigate to Status → Targets to verify all scrape targets are healthy. You should see targets for:

• kubernetes-apiservers — the Kubernetes API server metrics endpoint
• kubernetes-nodes — kubelet metrics from each node
• kubernetes-nodes-cadvisor — container-level resource metrics
• node-exporter — host-level metrics from each node
• kube-state-metrics — Kubernetes object metrics
• haproxy — HAProxy load balancer metrics (from additionalScrapeConfigs)
• etcd — will appear as DOWN initially (we enable this in Section 8.3)

Grafana — Open http://192.168.10.14:30002 in your browser.

Login with:

• Username: admin
• Password: prom-operator

The kube-prometheus-stack chart automatically provisions dozens of dashboards. Browse Dashboards → Browse to explore them. Key pre-installed dashboards include:

• Kubernetes / Compute Resources / Cluster — cluster-wide CPU and memory utilization
• Kubernetes / Compute Resources / Node (Pods) — per-node resource breakdown
• Node Exporter / Nodes — detailed host metrics
• Kubernetes / Networking / Cluster — network throughput and errors
• Prometheus / Overview — Prometheus self-monitoring (scrape duration, target health, TSDB size)

Adding Grafana Dashboards via ConfigMap

While the kube-prometheus-stack ships with many useful dashboards, there are several community dashboards that provide deeper or more specialized visibility. We will add three additional dashboards:

Dashboard ID Name Purpose 12693 Kubernetes Monitoring Comprehensive cluster overview with resource usage trends 15661 Node Exporter Full Extremely detailed host metrics (disk I/O, network sockets, CPU frequency, thermal zones) Custom k8s-system-api-server Kubernetes API Server performance (request latency, error rates, inflight requests)

The Grafana sidecar we configured earlier watches for ConfigMaps with the label grafana_dashboard: "1". To add a dashboard, we simply create a ConfigMap containing the dashboard JSON and apply the label.

Downloading Community Dashboards

Grafana community dashboards can be downloaded from grafana.com/grafana/dashboards as JSON files. The download URL pattern is:

https://grafana.com/api/dashboards/{DASHBOARD_ID}/revisions/latest/download

Download the two community dashboards:

# Dashboard 12693 - Kubernetes Monitoring
curl -fsSL \
  https://grafana.com/api/dashboards/12693/revisions/latest/download \
  -o /tmp/dashboard-12693.json

# Dashboard 15661 - Node Exporter Full
curl -fsSL \
  https://grafana.com/api/dashboards/15661/revisions/latest/download \
  -o /tmp/dashboard-15661.json

Creating ConfigMaps from Dashboard JSON

For each dashboard, create a ConfigMap in the monitoring namespace with the appropriate label:

# Dashboard 12693
kubectl create configmap grafana-dashboard-12693 \
  --from-file=k8s-monitoring.json=/tmp/dashboard-12693.json \
  -n monitoring

kubectl label configmap grafana-dashboard-12693 \
  grafana_dashboard="1" \
  -n monitoring

# Dashboard 15661
kubectl create configmap grafana-dashboard-15661 \
  --from-file=node-exporter-full.json=/tmp/dashboard-15661.json \
  -n monitoring

kubectl label configmap grafana-dashboard-15661 \
  grafana_dashboard="1" \
  -n monitoring

Creating the API Server Dashboard via YAML

For the custom API Server dashboard, we define it directly as a YAML manifest. This dashboard focuses on Kubernetes API Server performance — a critical metric for HA clusters where API server latency directly impacts cluster operations.

cat <<'EOF' > ~/kubespray/grafana-dashboard-apiserver.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-apiserver
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  k8s-system-api-server.json: |-
    {
      "annotations": {
        "list": []
      },
      "description": "Kubernetes API Server Monitoring",
      "editable": true,
      "gnetId": null,
      "graphTooltip": 1,
      "links": [],
      "panels": [
        {
          "title": "API Server Request Rate",
          "type": "timeseries",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "sum(rate(apiserver_request_total[5m])) by (verb)",
              "legendFormat": "{{ verb }}"
            }
          ],
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
        },
        {
          "title": "API Server Request Latency (99th percentile)",
          "type": "timeseries",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (verb, le))",
              "legendFormat": "{{ verb }}"
            }
          ],
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
        },
        {
          "title": "API Server Error Rate",
          "type": "timeseries",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "sum(rate(apiserver_request_total{code=~\"5..\"}[5m])) by (resource)",
              "legendFormat": "{{ resource }}"
            }
          ],
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }
        },
        {
          "title": "API Server Inflight Requests",
          "type": "timeseries",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "sum(apiserver_current_inflight_requests) by (request_kind)",
              "legendFormat": "{{ request_kind }}"
            }
          ],
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }
        }
      ],
      "schemaVersion": 36,
      "style": "dark",
      "tags": ["kubernetes", "apiserver"],
      "templating": { "list": [] },
      "time": { "from": "now-1h", "to": "now" },
      "title": "Kubernetes / System / API Server",
      "uid": "k8s-system-api-server"
    }
EOF

Apply the manifest:

kubectl apply -f ~/kubespray/grafana-dashboard-apiserver.yaml

Verifying Dashboard Import

The Grafana sidecar detects new ConfigMaps within a few seconds. You do not need to restart Grafana or any pods. Verify by listing the ConfigMaps with the dashboard label:

kubectl -n monitoring get configmap -l grafana_dashboard="1"

Expected output:

NAME                             DATA   AGE
grafana-dashboard-12693          1      2m
grafana-dashboard-15661          1      2m
grafana-dashboard-apiserver      1      30s

Now open Grafana (http://192.168.10.14:30002) and navigate to Dashboards → Browse. You should see the three new dashboards alongside the pre-installed ones. Open each to confirm they are rendering correctly with live data from Prometheus.

The API Server dashboard is particularly valuable for HA monitoring because it lets you observe whether API server request latency is consistent across the three control plane nodes. Asymmetric latency may indicate a problem with one specific node’s etcd connection or resource exhaustion.

Enabling etcd Metrics Collection

etcd is the most critical stateful component in a Kubernetes cluster. It stores the entire cluster state — every resource definition, every secret, every lease. If etcd fails, the cluster becomes read-only at best and completely non-functional at worst. Monitoring etcd health is therefore non-negotiable for any production cluster.

However, etcd does not expose a metrics endpoint by default in Kubespray deployments. We must explicitly enable it.

Configuring etcd_listen_metrics_urls

Kubespray provides dedicated variables to control etcd metrics exposure. We need to set two variables in the etcd group variables file.

Edit the etcd configuration:

vi ~/kubespray/inventory/mycluster/group_vars/etcd.yml

Add (or modify) the following variables:

# Enable etcd metrics endpoint
etcd_metrics: true

# Bind the metrics endpoint to all interfaces on port 2381
# This allows Prometheus (running on worker nodes) to scrape etcd metrics
# from the control plane nodes over the network
etcd_listen_metrics_urls: "http://0.0.0.0:2381"

Let us examine what each variable does:

etcd_metrics: true

This variable is Kubespray-specific. When set to true, Kubespray passes the --listen-metrics-urls flag to each etcd member during configuration. Without this flag, etcd simply does not serve any HTTP endpoint for metrics.

etcd_listen_metrics_urls: "http://0.0.0.0:2381"

This sets the address and port on which each etcd member will listen for metrics scrape requests. The values are deliberate:

• http:// (not https://) — Metrics are served over plain HTTP. This avoids the complexity of configuring Prometheus with etcd client TLS certificates. Since metrics do not contain sensitive data (only operational telemetry), HTTP is acceptable in a lab environment. In production, you might prefer https:// with mutual TLS.
• 0.0.0.0 — Bind to all network interfaces. This is necessary because Prometheus runs on worker nodes, which communicate with control plane nodes over the pod network or host network. Binding to 127.0.0.1 would restrict access to local-only scraping.
• 2381 — The conventional port for etcd metrics. The standard etcd client port is 2379, and peer port is 2380, so 2381 follows the natural sequence and avoids conflicts.

Applying the Configuration with Kubespray

Now apply the configuration change using Kubespray’s cluster.yml playbook with the etcd tag, limited to the etcd node group:

cd ~/kubespray

ansible-playbook cluster.yml \
  --tags "etcd" \
  --limit etcd \
  -e kube_version="1.32.9"

This command takes approximately 2 minutes to complete. Here is what happens during the playbook run:

1. Kubespray detects the configuration change — The etcd_listen_metrics_urls variable modifies the etcd systemd unit file (or static pod manifest, depending on deployment type).
2. etcd members are restarted one at a time — Kubespray performs a rolling restart of the etcd cluster. Each member is stopped, its configuration is updated, and it is restarted before moving to the next member. This preserves quorum throughout the process.
3. Automatic backups are created — Before restarting each etcd member, Kubespray takes a snapshot backup of the etcd data. These backups are stored in /var/backups/ on each control plane node:

ssh k8s-node1 ls -la /var/backups/etcd-*
-rw------- 1 root root 3145728 Jul 15 10:30 /var/backups/etcd-snapshot-20250715103000.db

1. This automatic backup behavior is a valuable safety net — if the etcd restart somehow corrupts data, you have an immediate point-in-time recovery option.
2. etcd rejoins the cluster — After restart, each member reconnects to the cluster and synchronizes its data with the other members. You can verify cluster health:

ssh k8s-node1 etcdctl.sh member list -w table

+------------------+---------+-----------+----------------------------+----------------------------+------------+
   |        ID        | STATUS  |   NAME    |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
   +------------------+---------+-----------+----------------------------+----------------------------+------------+
   | 8e9e05c52164694d | started | k8s-node1 | https://192.168.10.11:2380 | https://192.168.10.11:2379 |      false |
   | 91bc3c398fb3c146 | started | k8s-node2 | https://192.168.10.12:2380 | https://192.168.10.12:2379 |      false |
   | fd422379fda50e48 | started | k8s-node3 | https://192.168.10.13:2380 | https://192.168.10.13:2379 |      false |
   +------------------+---------+-----------+----------------------------+----------------------------+------------+

All three members should show started status.

Verifying the Metrics Endpoint

After the playbook completes, confirm that each etcd member is now serving metrics:

# Test from the admin-lb node (or any node that can reach the control plane network)
curl -s http://192.168.10.11:2381/metrics | head -20

Expected output (truncated):

# HELP etcd_server_has_leader Whether or not a leader exists. 1 is existence, 0 is not.
# TYPE etcd_server_has_leader gauge
etcd_server_has_leader 1
# HELP etcd_server_leader_changes_seen_total The number of leader changes seen.
# TYPE etcd_server_leader_changes_seen_total counter
etcd_server_leader_changes_seen_total 2
# HELP etcd_disk_wal_fsync_duration_seconds The latency distributions of fsync called by WAL.
# TYPE etcd_disk_wal_fsync_duration_seconds histogram
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 542
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 1203
...

Repeat for all three control plane nodes to confirm consistency:

for ip in 192.168.10.11 192.168.10.12 192.168.10.13; do
  echo "=== $ip ==="
  curl -s http://$ip:2381/metrics | grep etcd_server_has_leader
done

All three should report etcd_server_has_leader 1.

Adding etcd Scrape Config to Prometheus

We already configured the etcd scrape job in Section 8.2.1 as part of additionalScrapeConfigs in our custom values file. Here is the relevant snippet again for reference:

additionalScrapeConfigs:
  - job_name: "etcd"
    static_configs:
      - targets:
          - "192.168.10.11:2381"
          - "192.168.10.12:2381"
          - "192.168.10.13:2381"
    metrics_path: "/metrics"

Since this was included in the initial Helm install, Prometheus has been attempting to scrape these endpoints from the beginning. Before enabling etcd_listen_metrics_urls, the etcd targets appeared as DOWN in Prometheus. Now that the metrics endpoint is active, Prometheus should automatically pick them up.

Verifying etcd Targets in Prometheus

Open the Prometheus UI at http://192.168.10.14:30001 and navigate to Status → Targets. Look for the etcd job. You should see three targets, all with state UP:

Endpoint                     State    Labels                  Last Scrape    Scrape Duration
http://192.168.10.11:2381    UP       instance="192.168..."   12s ago        23.4ms
http://192.168.10.12:2381    UP       instance="192.168..."   14s ago        21.1ms
http://192.168.10.13:2381    UP       instance="192.168..."   11s ago        22.8ms

If any target shows DOWN, common troubleshooting steps include:

Firewall — Verify that port 2381 is open on the control plane nodes:

• ssh k8s-node1 ss -tlnp | grep 2381

1. Expected: LISTEN 0 4096 *:2381 *:* users:(("etcd",..))

Network connectivity — Test from a worker node (where Prometheus is running):

• ssh k8s-node4 curl -s http://192.168.10.11:2381/health

1. Expected: {"health":"true","reason":""}

etcd configuration — Verify the etcd process has the metrics flag:

• ssh k8s-node1 ps aux | grep etcd | grep listen-metrics

1. You should see --listen-metrics-urls=http://0.0.0.0:2381 in the process arguments.

Sample PromQL Queries

You can run these queries in the Prometheus UI (http://192.168.10.14:30001/graph) to immediately visualize etcd health:

etcd cluster leader status:

etcd_server_has_leader

WAL fsync latency (99th percentile, per member):

histogram_quantile(0.99,
  rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
)

Database size across all members:

etcd_mvcc_db_total_size_in_bytes

Peer network round-trip time (99th percentile):

histogram_quantile(0.99,
  rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
)

Rate of leader changes (should be 0 in a healthy cluster):

rate(etcd_server_leader_changes_seen_total[15m])

Creating an etcd Grafana Dashboard

You can also import a community etcd dashboard into Grafana. Dashboard ID 3070 (etcd by etcd.io) is a popular choice. Download and create a ConfigMap following the same pattern as Section 8.2.2:

# Download the etcd dashboard
curl -fsSL \
  https://grafana.com/api/dashboards/3070/revisions/latest/download \
  -o /tmp/dashboard-etcd-3070.json

# Create ConfigMap with the dashboard label
kubectl create configmap grafana-dashboard-etcd \
  --from-file=etcd.json=/tmp/dashboard-etcd-3070.json \
  -n monitoring

kubectl label configmap grafana-dashboard-etcd \
  grafana_dashboard="1" \
  -n monitoring

After a few seconds, the dashboard will appear in Grafana under Dashboards → Browse. This dashboard provides real-time visibility into:

• etcd cluster membership and leader status
• Raft proposal commit rates
• Disk I/O performance (WAL fsync, backend commit)
• gRPC request rates and latencies
• Database size and compaction

Complete Target Verification

At this point, your Prometheus instance should have the following target groups all showing UP:

Job                        Targets   Status
─────────────────────────────────────────────
kubernetes-apiservers      3/3       UP
kubernetes-nodes           5/5       UP
kubernetes-nodes-cadvisor  5/5       UP
node-exporter              5/5       UP
kube-state-metrics         1/1       UP
haproxy                    1/1       UP
etcd                       3/3       UP

The HAProxy stats page is also accessible at http://192.168.10.10:9000/haproxy_stats for a quick visual check of API server backend health, independent of Prometheus.

Kubernetes Upgrade with Kubespray

Upgrading Kubernetes is one of the most critical Day-2 operations in any production environment. Unlike initial cluster deployment, upgrades must be performed carefully — a single misconfiguration can lead to API downtime, workload disruption, or even data loss in etcd. Kubespray provides well-structured playbooks that automate the upgrade process while maintaining high availability, but understanding what happens under the hood is essential for operating confidently.

Each upgrade type introduces progressively more complexity. A patch upgrade touches only the Kubernetes binaries. A minor upgrade may involve API deprecations and behavioral changes. A major upgrade combined with a Kubespray version bump means new Ansible roles, new default variables, updated container runtime versions, and potentially new etcd releases — all changing simultaneously.

Before proceeding with any upgrade, let’s first address a prerequisite that many operators overlook: the CNI plugin.

Pre-Upgrade: Flannel CNI Plugin Update

Before upgrading Kubernetes itself, it is important to verify that the CNI (Container Network Interface) plugin is compatible with the target Kubernetes version. In our lab, we use Flannel as the CNI. Flannel is deployed as a DaemonSet, which means it runs on every node in the cluster. Unlike node-level components (kubelet, kube-proxy), you cannot upgrade Flannel on a per-node basis — a DaemonSet update rolls out to all nodes at once.

This is a critical consideration: if you upgrade Flannel after upgrading only half of your nodes, you risk running incompatible versions of the CNI across different nodes, which can cause intermittent pod networking failures.

Best practice: Update the Flannel CNI plugin before starting the Kubernetes upgrade, while all nodes are still running the same Kubernetes version.

Checking the Current Flannel Version

# Check the current Flannel DaemonSet image
kubectl -n kube-system get daemonset kube-flannel -o jsonpath='{.spec.template.spec.containers[0].image}'

Output:

docker.io/flannel/flannel:v0.26.7

Updating Flannel via Kubespray

Kubespray manages the Flannel version through its role defaults. To update Flannel, you can either change the variable in your inventory or pass it as an extra variable. The relevant variable is flannel_image_tag in roles/network_plugin/flannel/defaults/main.yml.

However, the simplest approach is to let Kubespray handle it by running the cluster playbook with the network plugin tag:

cd /root/kubespray

# Check what Flannel version Kubespray v2.29.1 ships with
grep -r "flannel_image_tag" roles/network_plugin/flannel/defaults/main.yml

Output:

flannel_image_tag: "v0.26.7"

If the current version is already the latest supported by your Kubespray release, no action is needed. If you need to update it manually:

ansible-playbook cluster.yml \
  --tags "network" \
  --limit "kube_node,kube_control_plane" \
  -e kube_version="1.32.9"

Important: Because Flannel is a DaemonSet, the --limit flag does not restrict which nodes receive the update. The DaemonSet controller will roll out the new image to all nodes regardless. The --limit flag only controls which nodes Ansible connects to for running tasks, but the Kubernetes DaemonSet update propagates cluster-wide.

After the update, verify:

# Verify Flannel pods are running the new version on all nodes
kubectl -n kube-system get pods -l app=flannel -o wide

# Check that all Flannel pods are in Running state
kubectl -n kube-system get pods -l app=flannel -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.spec.nodeName}{"\n"}{end}'

Kubespray Upgrade Strategies (Unsafe vs. Graceful)

Kubespray provides two fundamentally different approaches to cluster upgrades. Understanding the difference is critical for choosing the right strategy based on your environment’s tolerance for risk and downtime.

Strategy 1: Unsafe Upgrade (cluster.yml)

The cluster.yml playbook is the same playbook used for initial cluster deployment. When used for upgrades, it applies changes to all nodes simultaneously without draining workloads first. This is called an "unsafe" upgrade because:

• Nodes are not cordoned before the upgrade
• Pods are not drained — they continue running during the upgrade
• If a kubelet restart kills running pods, they are simply restarted by their controllers
• All nodes of the same type (control plane or worker) may be upgraded in parallel

To use cluster.yml for upgrades, you must set upgrade_cluster_setup: true:

ansible-playbook cluster.yml \
  -e kube_version="1.32.10" \
  -e upgrade_cluster_setup=true

When to use: Development environments, non-production clusters, or situations where speed matters more than zero-downtime guarantees.

Strategy 2: Graceful Upgrade (upgrade-cluster.yml)

The upgrade-cluster.yml playbook is purpose-built for production upgrades. It performs a rolling upgrade, processing one node at a time (or a configurable batch size), with proper workload migration:

1. Cordon the node (mark it as unschedulable)
2. Drain the node (gracefully evict all pods, respecting PodDisruptionBudgets)
3. Upgrade the node (update containerd, kubelet, kube-proxy, static pod manifests)
4. Uncordon the node (mark it as schedulable again)
5. Repeat for the next node

ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.32.10"

Controlling the Rolling Upgrade Behavior

Several variables control how the rolling upgrade proceeds:

Serial Execution

The serial parameter in the playbook controls how many nodes are upgraded simultaneously:

# Default: 20% of nodes at a time
serial: ". 20%"

# Conservative: one node at a time
serial: 1

# Aggressive: all nodes at once (effectively unsafe)
serial: "100%"

For production environments, serial: 1 is the safest option. It ensures that at any point during the upgrade, at most one node is unavailable.

Upgrade Confirmation Prompt

If you want manual confirmation before each node upgrade:

ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.32.10" \
  -e upgrade_node_confirm=true

This pauses the playbook before each node and waits for you to press Enter. This is useful for verifying cluster health between node upgrades.

Timed Pause Between Nodes

Alternatively, you can set an automatic pause between node upgrades:

ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.32.10" \
  -e upgrade_node_pause_seconds=60

This gives the cluster 60 seconds to stabilize after each node upgrade before proceeding to the next one.

PodDisruptionBudget (PDB) Considerations

During the drain phase, Kubernetes respects PodDisruptionBudgets. If a workload has a PDB with maxUnavailable: 0, the drain will hang indefinitely waiting for permission to evict pods. This is a common source of upgrade failures.

Before starting an upgrade, audit your PDBs:

# List all PDBs and their disruption settings
kubectl get pdb --all-namespaces -o wide

# Check for PDBs that might block drain
kubectl get pdb --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: maxUnavailable={.spec.maxUnavailable}, disruptionsAllowed={.status.disruptionsAllowed}{"\n"}{end}'

If you find PDBs with disruptionsAllowed: 0, either temporarily adjust them or ensure the corresponding workloads have enough replicas to tolerate one pod being evicted.

Patch Upgrade: v1.32.9 → v1.32.10

A patch upgrade is the simplest type of Kubernetes upgrade. It only includes bug fixes and security patches — no new features, no API changes, no deprecations. The Kubespray version remains the same (v2.29.1), and no supporting tools (etcd, containerd) need to change.

Despite its simplicity, a patch upgrade is the perfect opportunity to validate your upgrade procedures and tooling before attempting more complex minor or major upgrades.

Pre-Upgrade Cluster State

Before starting, document the current state:

# Current node versions
kubectl get nodes -o wide

NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP      OS-IMAGE                       KERNEL-VERSION
k8s-node1   Ready    control-plane   1d    v1.32.9   192.168.10.11    Rocky Linux 10.0 (Obsidian)    6.12.x
k8s-node2   Ready    control-plane   1d    v1.32.9   192.168.10.12    Rocky Linux 10.0 (Obsidian)    6.12.x
k8s-node3   Ready    control-plane   1d    v1.32.9   192.168.10.13    Rocky Linux 10.0 (Obsidian)    6.12.x
k8s-node4   Ready    <none>          1d    v1.32.9   192.168.10.14    Rocky Linux 10.0 (Obsidian)    6.12.x
k8s-node5   Ready    <none>          1d    v1.32.9   192.168.10.15    Rocky Linux 10.0 (Obsidian)    6.12.x

# Current etcd cluster health
ssh k8s-node1 etcdctl.sh endpoint status -w table

# Current component versions (static pods on control plane)
kubectl -n kube-system get pods -l tier=control-plane -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

kube-apiserver-k8s-node1            registry.k8s.io/kube-apiserver:v1.32.9
kube-apiserver-k8s-node2            registry.k8s.io/kube-apiserver:v1.32.9
kube-apiserver-k8s-node3            registry.k8s.io/kube-apiserver:v1.32.9
kube-controller-manager-k8s-node1   registry.k8s.io/kube-controller-manager:v1.32.9
kube-controller-manager-k8s-node2   registry.k8s.io/kube-controller-manager:v1.32.9
kube-controller-manager-k8s-node3   registry.k8s.io/kube-controller-manager:v1.32.9
kube-scheduler-k8s-node1            registry.k8s.io/kube-scheduler:v1.32.9
kube-scheduler-k8s-node2            registry.k8s.io/kube-scheduler:v1.32.9
kube-scheduler-k8s-node3            registry.k8s.io/kube-scheduler:v1.32.9

# Current kube-proxy version
kubectl -n kube-system get daemonset kube-proxy -o jsonpath='{.spec.template.spec.containers[0].image}'

registry.k8s.io/kube-proxy:v1.32.9

Control Plane Rolling Upgrade

We start the upgrade with the control plane nodes and etcd. This is always done first because worker nodes depend on the API server, and Kubernetes guarantees backward compatibility: a newer control plane can manage older worker nodes, but not vice versa.

cd /root/kubespray

ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.32.10" \
  --limit "kube_control_plane:etcd"

This command takes approximately 14 minutes for three control plane nodes.

What Happens During the Control Plane Upgrade

The upgrade-cluster.yml playbook executes the following sequence for each control plane node, one at a time:

Phase 1: Pre-upgrade Downloads

Before touching any node, Kubespray downloads all required container images and binaries to every node in the --limit scope. This ensures that the actual upgrade step (which involves restarting components) completes as quickly as possible.

TASK [download : Download containers if pull is required or told to always pull] ****

The images downloaded for a patch upgrade:

registry.k8s.io/kube-apiserver:v1.32.10
registry.k8s.io/kube-controller-manager:v1.32.10
registry.k8s.io/kube-scheduler:v1.32.10
registry.k8s.io/kube-proxy:v1.32.10

Images that do not change in a patch upgrade (and are therefore skipped):

registry.k8s.io/coredns/coredns:v1.12.0         # CoreDNS version unchanged
registry.k8s.io/pause:3.10                        # Pause container unchanged
quay.io/coreos/etcd:v3.5.25                       # etcd version unchanged

Phase 2: Rolling Upgrade per Node (k8s-node1 first)

TASK [kubernetes/control-plane : Kubeadm | Cordon node] *************************
changed: [k8s-node1]

TASK [kubernetes/control-plane : Kubeadm | Drain node] **************************
changed: [k8s-node1]

At this point, k8s-node1 is cordoned (no new pods will be scheduled) and drained (existing pods are gracefully evicted). Since this is a control plane node, only the static pods (apiserver, controller-manager, scheduler) and DaemonSet pods remain — they cannot be evicted.

TASK [container-engine/containerd : Containerd | Ensure containerd is installed] ***
ok: [k8s-node1]

For a patch upgrade, the containerd version typically does not change. Kubespray verifies the installed version matches the expected version and skips reinstallation if they match.

TASK [kubernetes/control-plane : Kubeadm | Upgrade first control plane] *********
changed: [k8s-node1]

This is the core step. Kubespray runs kubeadm upgrade apply v1.32.10 on the first control plane node. This command:

1. Validates the upgrade path (v1.32.9 → v1.32.10 is allowed)
2. Downloads any missing component images
3. Updates the static pod manifests in /etc/kubernetes/manifests/
4. The kubelet detects the manifest changes and restarts the static pods
5. Waits for the new API server to become healthy

For subsequent control plane nodes (k8s-node2, k8s-node3), kubeadm upgrade node is used instead of kubeadm upgrade apply:

TASK [kubernetes/control-plane : Kubeadm | Upgrade subsequent control planes] ***
changed: [k8s-node2]
...
changed: [k8s-node3]

The difference:

• kubeadm upgrade apply — updates the cluster-level configuration and upgrades the first node
• kubeadm upgrade node — upgrades a single node using the cluster-level configuration already applied

Phase 3: kube-proxy DaemonSet Update

After the first control plane node is upgraded, Kubespray updates the kube-proxy DaemonSet image. Since kube-proxy is a DaemonSet, this update propagates to all nodes in the cluster, including worker nodes that have not been upgraded yet:

TASK [kubernetes/control-plane : Kubeadm | Update kube-proxy DaemonSet] *********
changed: [k8s-node1]

This is safe because kube-proxy v1.32.10 is backward-compatible with kubelet v1.32.9 running on the worker nodes.

Phase 4: Uncordon

TASK [kubernetes/control-plane : Kubeadm | Uncordon node] ***********************
changed: [k8s-node1]

The node is marked schedulable again, and the playbook moves to the next control plane node.

Monitoring the Upgrade in Real-Time

While the upgrade is running, you can monitor progress from another terminal:

# Watch node versions change in real-time
watch -n 2 'kubectl get nodes -o wide'

# Watch static pod restarts
watch -n 2 'kubectl -n kube-system get pods -l tier=control-plane -o wide'

# Monitor etcd cluster health (should remain healthy throughout)
ssh k8s-node1 etcdctl.sh endpoint status -w table

# Check API server availability through the load balancer
while true; do curl -sk -o /dev/null -w "%{http_code}\n" https://192.168.10.10:6443/healthz; sleep 1; done

During the rolling upgrade, you will briefly see mixed versions:

NAME        STATUS                     ROLES           VERSION
k8s-node1   Ready                      control-plane   v1.32.10   ← upgraded
k8s-node2   Ready,SchedulingDisabled   control-plane   v1.32.9    ← being upgraded
k8s-node3   Ready                      control-plane   v1.32.9    ← waiting
k8s-node4   Ready                      <none>          v1.32.9
k8s-node5   Ready                      <none>          v1.32.9

The HAProxy load balancer on admin-lb (192.168.10.10) continues to route traffic to the two healthy API servers while one is being upgraded. The API remains available throughout.

Post-Control-Plane-Upgrade Verification

After the control plane upgrade completes:

# All control plane nodes should show v1.32.10
kubectl get nodes -o wide

NAME        STATUS   ROLES           VERSION
k8s-node1   Ready    control-plane   v1.32.10
k8s-node2   Ready    control-plane   v1.32.10
k8s-node3   Ready    control-plane   v1.32.10
k8s-node4   Ready    <none>          v1.32.9    ← workers still on old version
k8s-node5   Ready    <none>          v1.32.9

# Verify static pod images
kubectl -n kube-system get pods -l tier=control-plane -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

All control plane static pods should now show v1.32.10.

# Verify kube-proxy was updated on ALL nodes (including workers)
kubectl -n kube-system get daemonset kube-proxy -o jsonpath='{.spec.template.spec.containers[0].image}'

registry.k8s.io/kube-proxy:v1.32.10

# Verify etcd cluster health (etcd version unchanged for patch upgrade)
ssh k8s-node1 etcdctl.sh member list -w table
ssh k8s-node1 etcdctl.sh endpoint status -w table

# Verify API server responds correctly through all endpoints
curl -sk https://192.168.10.11:6443/version | jq .gitVersion
curl -sk https://192.168.10.12:6443/version | jq .gitVersion
curl -sk https://192.168.10.13:6443/version | jq .gitVersion
curl -sk https://192.168.10.10:6443/version | jq .gitVersion  # via HAProxy

All should return "v1.32.10".

Worker Node Individual Upgrade

With the control plane upgraded, we now upgrade the worker nodes. Unlike control plane nodes, worker nodes should be upgraded individually to maintain workload availability. If you have workloads with replicas: 2 spread across two workers, upgrading both simultaneously would cause a full outage.

# Upgrade worker node k8s-node4
ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.32.10" \
  --limit "k8s-node4"

This takes approximately 2 minutes per worker node.

What Happens During a Worker Node Upgrade

The process is simpler than control plane upgrades:

1. Cordon k8s-node4
2. Drain k8s-node4 (all non-DaemonSet pods are evicted to other nodes)
3. Upgrade containerd (if version changed — not in patch upgrades)
4. Run kubeadm upgrade node (updates kubelet configuration)
5. Restart kubelet service
6. Uncordon k8s-node4

During the drain, you can observe pods migrating:

# Watch pods being rescheduled from node4 to node5
watch -n 1 'kubectl get pods -A -o wide --field-selector spec.nodeName=k8s-node4'

After k8s-node4 is done, upgrade k8s-node5:

# Upgrade worker node k8s-node5
ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.32.10" \
  --limit "k8s-node5"

Post-Worker-Upgrade Verification

# All nodes should now be v1.32.10
kubectl get nodes -o wide

NAME        STATUS   ROLES           VERSION
k8s-node1   Ready    control-plane   v1.32.10
k8s-node2   Ready    control-plane   v1.32.10
k8s-node3   Ready    control-plane   v1.32.10
k8s-node4   Ready    <none>          v1.32.10
k8s-node5   Ready    <none>          v1.32.10

# Verify all system pods are healthy
kubectl -n kube-system get pods -o wide

# Verify workloads are running normally
kubectl get pods --all-namespaces -o wide | grep -v Running | grep -v Completed

Updating Admin kubectl and kubeconfig

After the cluster upgrade, the admin node (admin-lb) should also have its kubectl binary updated to match the cluster version. While older kubectl versions can communicate with newer API servers (within one minor version skew), it is best practice to keep them aligned.

# On the admin-lb node (192.168.10.10)

# Check current kubectl version
kubectl version --client

Client Version: v1.32.9

# Update kubectl to match the cluster version
# First, ensure the Kubernetes yum repo points to v1.32
cat /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key

# Install the latest kubectl in the v1.32 series
dnf install kubectl --disableexcludes=kubernetes -y

# Verify the new version
kubectl version --client

Client Version: v1.32.10

Also refresh the kubeconfig from the first control plane node, as the certificate or cluster configuration may have been updated:

# Copy the updated kubeconfig from the first control plane
scp k8s-node1:/root/.kube/config /root/.kube/config

# Verify connectivity
kubectl cluster-info
kubectl get nodes -o wide

The patch upgrade from v1.32.9 to v1.32.10 is now complete.

Minor Upgrade: v1.32.10 → v1.33.7

A minor version upgrade is more significant than a patch upgrade. Minor releases introduce new features, new APIs, deprecate old APIs, and may change default behaviors. However, the upgrade procedure with Kubespray is nearly identical — the difference lies primarily in what changes inside the cluster.

Pre-Upgrade: Review Release Notes

Before any minor upgrade, review the Kubernetes changelog for breaking changes:

# Check what version Kubespray v2.29.1 supports for v1.33
grep -r "kube_version" roles/kubespray-defaults/defaults/main/download.yml | head -5

Kubernetes follows a strict version skew policy: you can only upgrade one minor version at a time. Jumping from v1.32 to v1.34 directly is not supported. You must go v1.32 → v1.33 → v1.34.

Running the Minor Upgrade

The commands are identical to the patch upgrade — only the kube_version value changes.

Step 1: Upgrade Control Plane

cd /root/kubespray

ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.33.7" \
  --limit "kube_control_plane:etcd"

This takes approximately 18 minutes for three control plane nodes. The longer duration compared to the patch upgrade is because:

• More container images need to be downloaded (some support components have new versions)
• kubeadm upgrade apply performs more extensive preflight checks when crossing minor versions
• Static pod restarts may take slightly longer as new features are initialized

The rolling upgrade process is identical:

Node 1: cordon → drain → kubeadm upgrade apply v1.33.7 → uncordon
Node 2: cordon → drain → kubeadm upgrade node → uncordon
Node 3: cordon → drain → kubeadm upgrade node → uncordon

During the upgrade, monitor the API server availability:

# Continuous health check through HAProxy
while true; do
  HTTP_CODE=$(curl -sk -o /dev/null -w "%{http_code}" https://192.168.10.10:6443/healthz)
  echo "$(date '+%H:%M:%S') - API Server: $HTTP_CODE"
  sleep 1
done

You should see uninterrupted 200 responses throughout the control plane upgrade, thanks to the HAProxy load balancer distributing requests across the three API servers.

Step 2: Upgrade Worker Nodes

# Upgrade workers one at a time
ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.33.7" \
  --limit "k8s-node4"

ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.33.7" \
  --limit "k8s-node5"

Each worker takes approximately 3 minutes.

Alternatively, you can upgrade all workers at once if you have sufficient capacity:

# Upgrade all workers (serial behavior controlled by playbook)
ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.33.7" \
  --limit "kube_node"

With the default serial: 20%, both workers would be upgraded sequentially.

Step 3: Verify the Minor Upgrade

# All nodes should show v1.33.7
kubectl get nodes -o wide

NAME        STATUS   ROLES           VERSION
k8s-node1   Ready    control-plane   v1.33.7
k8s-node2   Ready    control-plane   v1.33.7
k8s-node3   Ready    control-plane   v1.33.7
k8s-node4   Ready    <none>          v1.33.7
k8s-node5   Ready    <none>          v1.33.7

# Verify API version
curl -sk https://192.168.10.10:6443/version | jq .

{
  "major": "1",
  "minor": "33",
  "gitVersion": "v1.33.7",
  "buildDate": "...",
  "goVersion": "go1.23.x",
  "compiler": "gc",
  "platform": "linux/amd64"
}

# Verify system pods
kubectl -n kube-system get pods -o wide

# Check for any deprecated API usage in your workloads
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Step 4: Update Admin kubectl

For a minor version upgrade, the kubectl repo must be updated to the new minor version:

# On admin-lb node

# Update the Kubernetes yum repo to point to v1.33
cat > /etc/yum.repos.d/kubernetes.repo << 'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
EOF

# Install kubectl v1.33
dnf install kubectl --disableexcludes=kubernetes -y

# Verify
kubectl version --client

Client Version: v1.33.7

# Refresh kubeconfig
scp k8s-node1:/root/.kube/config /root/.kube/config

# Confirm connectivity
kubectl cluster-info
kubectl get nodes

The minor upgrade from v1.32.10 to v1.33.7 is now complete.

Major Upgrade with Kubespray Version Bump: v1.33.7 → v1.34.3

This is the most complex upgrade scenario. Not only are we upgrading Kubernetes by another minor version (v1.33 → v1.34), but we are also upgrading Kubespray itself from v2.29.1 to v2.30.0. A Kubespray version bump means:

• New Ansible roles and tasks
• Updated default variable values
• New versions of supporting components (etcd, containerd, CoreDNS)
• Potentially new Python dependencies for Ansible
• Changed Ansible collection requirements

This is the type of upgrade that most closely mirrors what happens in real production environments, where the infrastructure tooling evolves alongside the target platform.

Switching Kubespray Tags (v2.29.1 → v2.30.0)

First, we need to switch the Kubespray repository to the new release tag.

cd /root/kubespray

# Check current Kubespray version
git describe --tags

v2.29.1

# Fetch the latest tags
git fetch --all --tags

# Check available v2.30.x tags
git tag -l "v2.30*"

v2.30.0

# Switch to the new version
git checkout v2.30.0

Note: switching to 'v2.30.0'.
You are in 'detached HEAD' state...
HEAD is now at <commit-hash> Release v2.30.0

After switching, verify what Kubernetes versions this Kubespray release supports:

# Check the supported K8s version range
grep -r "kube_version_min_required\|kube_version" roles/kubespray-defaults/defaults/main/*.yml | grep -i version | head -10

Also check what component versions ship with this release:

# etcd version
grep "etcd_version" roles/download/defaults/main/main.yml

etcd_version: v3.5.26

# containerd version
grep "containerd_version" roles/download/defaults/main/main.yml

containerd_version: 2.2.1

# CoreDNS version
grep "coredns_version" roles/download/defaults/main/main.yml

This reveals that upgrading to Kubespray v2.30.0 will also upgrade:

• etcd: 3.5.25 → 3.5.26
• containerd: 2.1.5 → 2.2.1

These component upgrades happen automatically as part of the Kubernetes upgrade because Kubespray controls their versions through role defaults.

Python Virtual Environment for Dependency Isolation

Kubespray v2.30.0 may require different Python package versions than v2.29.1. Installing new dependencies directly on the system can break existing Ansible setups. A Python virtual environment provides isolation.

# Check the new requirements
cat /root/kubespray/requirements.txt

ansible==10.7.0
cryptography==46.0.3
jinja2==3.1.5
jmespath==1.1.0
MarkupSafe==3.0.2
netaddr==1.3.0
pbr==6.1.1
ruamel.yaml==0.18.14

Compare with what is currently installed:

pip3 list 2>/dev/null | grep -i -E "ansible|cryptography|jmespath|netaddr"

If versions differ, install the new requirements:

# Option 1: Direct installation (simpler, our lab approach)
pip3 install -r requirements.txt --break-system-packages

# Option 2: Virtual environment (recommended for production admin nodes)
python3 -m venv /root/kubespray-venv
source /root/kubespray-venv/bin/activate
pip3 install -r requirements.txt

If using a virtual environment, remember to activate it before running any Ansible commands:

source /root/kubespray-venv/bin/activate

Verify the key packages:

pip3 list | grep -i -E "ansible|cryptography|jmespath|netaddr"

ansible          10.7.0
cryptography     46.0.3
jmespath         1.1.0
netaddr          1.3.0

Also verify that Ansible can reach all nodes:

cd /root/kubespray
ansible -i inventory/mycluster/inventory.ini all -m ping

All nodes should return pong.

etcd Version Upgrade (3.5.25 → 3.5.26)

One of the most critical parts of a Kubespray version bump is the etcd upgrade. etcd is the single source of truth for all cluster state — a failed etcd upgrade can be catastrophic. Kubespray handles this carefully:

1. Before modifying any etcd member, Kubespray creates a backup snapshot
2. Each etcd member is upgraded one at a time
3. After each member upgrade, cluster health is verified before proceeding

Pre-Upgrade etcd Status

# etcd member list
ssh k8s-node1 etcdctl.sh member list -w table

+------------------+---------+-----------+---------------------------+---------------------------+
|        ID        | STATUS  |   NAME    |       PEER ADDRS          |      CLIENT ADDRS         |
+------------------+---------+-----------+---------------------------+---------------------------+
| 8e9e05c52164694d | started | etcd1     | https://192.168.10.11:2380| https://192.168.10.11:2379|
| 91bc3c398fb3c146 | started | etcd2     | https://192.168.10.12:2380| https://192.168.10.12:2379|
| fd422379fda50e48 | started | etcd3     | https://192.168.10.13:2380| https://192.168.10.13:2379|
+------------------+---------+-----------+---------------------------+---------------------------+

# etcd version before upgrade
ssh k8s-node1 etcdctl.sh endpoint status -w table

The version column should show 3.5.25.

etcd Upgrade Process (Automatic during K8s upgrade)

The etcd upgrade happens automatically when you run the upgrade playbook. Kubespray detects that the new release expects etcd v3.5.26 and upgrades each member:

TASK [etcd : Backup etcd data] **************************************************
changed: [k8s-node1]
changed: [k8s-node2]
changed: [k8s-node3]

TASK [etcd : Upgrade etcd member] ***********************************************
changed: [k8s-node1]

TASK [etcd : Wait for etcd cluster health] **************************************
ok: [k8s-node1]

TASK [etcd : Upgrade etcd member] ***********************************************
changed: [k8s-node2]

TASK [etcd : Wait for etcd cluster health] **************************************
ok: [k8s-node2]

TASK [etcd : Upgrade etcd member] ***********************************************
changed: [k8s-node3]

Each etcd member is restarted with the new binary. During the restart of a single member, the etcd cluster maintains quorum with the remaining two members (2/3 = majority), so there is no data availability interruption.

etcd Backup Verification

After the upgrade, verify that backups were created:

ssh k8s-node1 tree /var/backups/ | head -20

/var/backups/
├── etcd-20260207-before-upgrade/
│   └── member/
│       ├── snap/
│       │   └── db
│       └── wal/
│           └── ...

These backups are critical — if anything goes wrong during the upgrade, you can restore etcd from these snapshots. Kubespray creates timestamped backup directories under /var/backups/ on each etcd member.

Post-Upgrade etcd Verification

# Verify etcd version is now 3.5.26
ssh k8s-node1 etcdctl.sh endpoint status -w table

# Verify cluster health
ssh k8s-node1 etcdctl.sh endpoint health -w table

+---------------------------+--------+-------+-------+
|         ENDPOINT          | HEALTH |  TOOK | ERROR |
+---------------------------+--------+-------+-------+
| https://192.168.10.11:2379| true   |  12ms |       |
| https://192.168.10.12:2379| true   |  11ms |       |
| https://192.168.10.13:2379| true   |  13ms |       |
+---------------------------+--------+-------+-------+

containerd Upgrade (2.1.5 → 2.2.1)

The container runtime upgrade is another significant change that comes with the Kubespray version bump. containerd is the CRI (Container Runtime Interface) implementation that actually runs containers on each node. Upgrading it requires restarting the containerd service, which briefly affects container operations on that node.

Kubespray handles this as part of the rolling upgrade:

1. The node is already cordoned and drained (no user workloads running)
2. containerd binary is replaced
3. containerd service is restarted
4. kubelet is restarted (reconnects to the new containerd socket)
5. Static pods are re-created by the kubelet

containerd Upgrade is Transparent

You do not need to run separate commands for containerd — it is upgraded automatically during the upgrade-cluster.yml execution:

TASK [container-engine/containerd : Containerd | Download containerd] ***********
changed: [k8s-node1]

TASK [container-engine/containerd : Containerd | Install containerd] ************
changed: [k8s-node1]

TASK [container-engine/containerd : Containerd | Restart containerd] ************
changed: [k8s-node1]

Running the Full Upgrade

Now, execute the complete upgrade:

cd /root/kubespray

# Step 1: Upgrade control plane and etcd (includes etcd + containerd upgrades)
ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.34.3" \
  --limit "kube_control_plane:etcd"

This takes approximately 15 minutes for three control plane nodes. The additional time compared to the patch upgrade comes from the etcd restart/upgrade and the containerd binary replacement on each node.

# Step 2: Upgrade worker nodes
ansible-playbook upgrade-cluster.yml \
  -e kube_version="1.34.3" \
  --limit "kube_node"

Worker nodes take approximately 4 minutes each because containerd also needs to be upgraded on workers.

Verifying containerd Version

After the upgrade completes:

# Check containerd version on all nodes
ansible -i inventory/mycluster/inventory.ini all -m shell -a "containerd --version"

k8s-node1 | SUCCESS | containerd containerd.io 2.2.1 ...
k8s-node2 | SUCCESS | containerd containerd.io 2.2.1 ...
k8s-node3 | SUCCESS | containerd containerd.io 2.2.1 ...
k8s-node4 | SUCCESS | containerd containerd.io 2.2.1 ...
k8s-node5 | SUCCESS | containerd containerd.io 2.2.1 ...

# Verify container images on a node
ssh k8s-node1 crictl images

You should see images for Kubernetes v1.34.3 alongside the upgraded etcd v3.5.26 image.

Post-Upgrade Cluster Verification

# All nodes should show v1.34.3
kubectl get nodes -o wide

NAME        STATUS   ROLES           VERSION
k8s-node1   Ready    control-plane   v1.34.3
k8s-node2   Ready    control-plane   v1.34.3
k8s-node3   Ready    control-plane   v1.34.3
k8s-node4   Ready    <none>          v1.34.3
k8s-node5   Ready    <none>          v1.34.3

# API version check through all endpoints
for ip in 192.168.10.{11,12,13,10}; do
  echo -n "$ip: "
  curl -sk https://$ip:6443/version | jq -r .gitVersion
done

192.168.10.11: v1.34.3
192.168.10.12: v1.34.3
192.168.10.13: v1.34.3
192.168.10.10: v1.34.3

# Verify all system components
kubectl -n kube-system get pods -o wide

# Check downloaded release binaries
ssh k8s-node1 tree /tmp/releases

Helm and kubectl Version Updates

After a major upgrade, the admin tooling must be updated to match. This includes both kubectl and helm.

Updating kubectl to v1.34

# On admin-lb node

# Update the Kubernetes yum repo to v1.34
cat > /etc/yum.repos.d/kubernetes.repo << 'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
EOF

# Install kubectl
dnf clean metadata
dnf install kubectl --disableexcludes=kubernetes -y

# Verify
kubectl version --client

Client Version: v1.34.3

Updating Helm

Kubespray v2.30.0 may also ship with or recommend a newer Helm version. In our case, Helm is updated to v3.20.0:

# Check current Helm version
helm version --short

v3.17.3+...

# Install Helm v3.20.0
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.20.0 bash

Downloading https://get.helm.sh/helm-v3.20.0-linux-amd64.tar.gz
Verifying checksum...
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm

# Verify the new version
helm version --short

v3.20.0+...

Refreshing kubeconfig

As with previous upgrades, refresh the kubeconfig from the first control plane node:

scp k8s-node1:/root/.kube/config /root/.kube/config

# Final verification
kubectl cluster-info
kubectl get nodes -o wide
kubectl -n kube-system get pods

HAProxy Stats Verification

Finally, verify that the HAProxy load balancer still shows all three control plane backends as healthy:

# Open HAProxy stats page
# URL: http://192.168.10.10:9000/haproxy_stats

# Or check via CLI
curl -s http://192.168.10.10:9000/haproxy_stats\;csv | grep k8s_api | cut -d',' -f1-3,18

All three backends should show status UP.

Prometheus/Grafana Monitoring Check

If you have the kube-prometheus-stack installed (see Section 8), verify that all monitoring targets are still healthy after the upgrade:

# Prometheus targets (check via NodePort)
# URL: http://192.168.10.14:30001/targets

# Grafana dashboards (check via NodePort)
# URL: http://192.168.10.14:30002
# Login: admin / prom-operator

Key dashboards to check:

• Kubernetes / API server (dashboard 15661): Verify API server latency and error rates returned to normal
• etcd (dashboard 12693): Verify etcd leader elections did not spike, DB size is stable
• Node Exporter (dashboard from kube-prometheus-stack): Verify resource utilization across all nodes

Upgrade Summary

Over the course of this section, we performed three distinct upgrade paths, each building on the complexity of the previous one:

Patch Upgrade (v1.32.9 → v1.32.10): The simplest case — only Kubernetes binaries changed. No etcd, no containerd, no Kubespray changes. Demonstrated the core rolling upgrade mechanics: cordon, drain, kubeadm upgrade, uncordon.

Minor Upgrade (v1.32.10 → v1.33.7): Same procedure, but with potential API deprecations and longer upgrade times. Reinforced the importance of upgrading one minor version at a time and updating the admin kubectl repository pointer.

Major Upgrade with Kubespray Bump (v1.33.7 → v1.34.3): The full production scenario — new Kubespray tag, new Python dependencies, new etcd version, new containerd version, new Helm version, and the Kubernetes upgrade itself. This is the upgrade type that production teams encounter most frequently and the one that requires the most preparation.

Throughout all three upgrades, the cluster remained available. The HAProxy load balancer ensured API server continuity, the rolling upgrade strategy preserved workload availability, and etcd quorum was maintained during member restarts. This is the power of combining Kubespray’s automation with a properly designed HA architecture.

Conclusion

Key Takeaways

This guide walked through the complete lifecycle of a production-oriented Kubernetes cluster managed by Kubespray. Here are the most important lessons:

HA is not optional — it is the foundation. A 3-node control plane with an etcd quorum of 3 is the minimum viable production architecture. Without it, any single node failure renders the entire cluster unmanageable. The cost of 2 additional control plane nodes is negligible compared to the cost of downtime.

The API endpoint configuration has real consequences. The three cases we explored (client-side NGINX proxy, external LB + client-side proxy, external LB only) each have distinct trade-offs. Client-side proxies give workers independence from external infrastructure but add per-node complexity. External-only LB simplifies worker configuration but makes the load balancer a critical dependency. Choose based on your infrastructure constraints and failure tolerance.

Node management is a routine operation, not an emergency. Adding workers, removing failed nodes, and replacing control plane nodes should all be practiced regularly. The distinction between scale.yml (workers only) and cluster.yml (control plane additions) is critical — using the wrong playbook leads to incomplete configurations. Always add new control plane nodes at the end of the inventory group.

etcd is the heart of the cluster. Every piece of cluster state lives in etcd. Always use an odd number of members (3, 5, 7) for proper quorum. Always verify etcd health after any cluster modification. Always ensure backups are being created before upgrades.

Upgrades should be frequent and incremental. Patch upgrades (v1.32.9 → v1.32.10) are low-risk and take minutes. Minor upgrades (v1.32 → v1.33) require more attention but follow the same pattern. Skipping versions creates compounding technical debt and makes each subsequent upgrade more painful. Kubernetes only supports upgrading one minor version at a time.

Kubespray version matters. Each Kubespray release supports a specific range of Kubernetes versions. When crossing into a new Kubernetes minor version, check whether your Kubespray version supports it. Switching Kubespray versions (v2.29.1 → v2.30.0) also requires updating Python dependencies — always run pip install -r requirements.txt after switching tags.

Monitoring is not a nice-to-have. You cannot upgrade with confidence if you cannot see what is happening. kube-prometheus-stack with Prometheus, Grafana, and etcd metrics gives you the observability needed to verify cluster health before, during, and after upgrades.

Kubespray as a Production Lifecycle Management Framework

Kubespray is more than a deployment tool — it is a lifecycle management framework. The five core playbooks (cluster.yml, scale.yml, remove-node.yml, upgrade-cluster.yml, reset.yml) cover the full operational spectrum:

PhasePlaybookUse CaseDeploycluster.ymlInitial deployment, adding control plane nodesScalescale.ymlAdding worker nodesShrinkremove-node.ymlRemoving any node (graceful or forced)Upgradeupgrade-cluster.ymlRolling upgrades with drain/cordon/uncordonTeardownreset.ymlFull cluster reset

Because Kubespray is built on Ansible, every operation is:

• Idempotent: Running a playbook twice produces the same result. Safe to retry after failures.
• Declarative: The inventory and variables define the desired state; Kubespray converges the cluster to match.
• Auditable: Every playbook run can be logged, and the inventory can be version-controlled in Git.
• Extensible: Custom roles and tasks can be added for organization-specific requirements (certificate management, custom CNI configurations, security hardening).

The combination of Git-managed inventory, Ansible’s idempotency, and Kubespray’s rolling upgrade capabilities makes it a solid foundation for teams that need to manage Kubernetes clusters without relying on a managed cloud service.

Next Steps: Disaster Recovery, Automation, and Beyond

This guide covered the core operational workflows, but several advanced topics extend naturally from here:

Disaster Recovery (DR):

• Implement automated etcd snapshot backups (e.g., CronJob running etcdctl snapshot save to an off-cluster storage backend)
• Practice etcd restoration from snapshots: etcdctl snapshot restore
• Document and test the full cluster recovery procedure: restore etcd → restart control plane → verify state
• Consider etcd backup solutions like Velero for full cluster backup including PVs

CI/CD-Driven Upgrades:

• Integrate Kubespray playbooks into a CI/CD pipeline (Jenkins, GitLab CI, GitHub Actions)
• Use a Git branch per environment (dev, staging, production)
• Automate the upgrade sequence: run in staging → run health checks → promote to production
• Add pre-upgrade and post-upgrade verification steps (API health, etcd quorum, workload status) as pipeline gates

Security Hardening:

• Enable PodSecurityAdmission (or OPA/Gatekeeper) for workload policy enforcement
• Implement NetworkPolicies to restrict pod-to-pod communication
• Configure certificate rotation and monitor certificate expiration dates
• Harden etcd with TLS client authentication and restrict access to control plane nodes only

Advanced HA:

• Deploy HAProxy in HA with keepalived for virtual IP failover (eliminates LB as SPOF)
• Consider running 5 etcd members for higher fault tolerance in large clusters
• Implement pod topology spread constraints and anti-affinity rules for critical workloads

Multi-Cluster Management:

• Use Kubespray to manage multiple clusters from a single admin node
• Maintain separate inventories per cluster in the same Git repository
• Standardize configurations across clusters using shared group_vars with cluster-specific overrides

The techniques in this guide — HA deployment, rolling upgrades, node lifecycle management, and observability — form the operational foundation upon which all of these advanced topics are built. Master these fundamentals first, and the rest follows naturally.

Lab Cleanup: When you’re done, tear down the entire environment with vagrant destroy -f. All VMs and their data will be removed.

Chapter 1: Why Kubespray? Understanding the Automation Spectrum

If you have been following the Kubernetes learning path properly, you have probably gone through something like Kubernetes The Hard Way first. You manually provisioned VMs, generated certificates with OpenSSL, created kubeconfig files by hand, configured etcd as a systemd service, and set up each control plane component one by one. It was painful. It took hours. But you understood exactly what was happening.

Then you discovered kubeadm. Suddenly, all that certificate generation, etcd bootstrapping, and control plane deployment collapsed into a single command:

kubeadm init — config=kubeadm-config.yaml

The hours of manual work became minutes. But you still had to prepare each node yourself — disable swap, load kernel modules, install containerd, configure kubelet. You still had to SSH into each machine and run commands.

Now we arrive at Kubespray. One command, and everything happens:

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml

Fifteen minutes later, you have a production-ready Kubernetes cluster. The OS is configured. Containerd is installed. Certificates are generated. etcd is running. The control plane is up. Workers have joined. CNI is deployed. CoreDNS is answering queries.

This is the automation spectrum. Each step up the ladder makes deployment easier and faster. But there is a cost that nobody talks about until it bites them.

What Kubespray Actually Is

The Kubespray GitHub repository introduces itself with a simple tagline:

Deploy a Production Ready Kubernetes Cluster

This is not marketing speak. Kubespray genuinely aims to give you a cluster that you can run workloads on immediately, configured according to best practices that the Kubernetes community has learned through years of production experience.

The official documentation describes Kubespray as:

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks.

That phrase “domain knowledge” is the key. Kubespray is not just a script that runs commands in order. It encodes decisions about how to configure NTP synchronization, what kernel parameters to set, how to structure etcd clusters, when to use IPVS versus iptables for kube-proxy, and hundreds of other operational details.

Kubespray is maintained by the Kubernetes SIG Cluster Lifecycle group. Since version 2.3, it uses kubeadm internally for the actual cluster bootstrapping. This means Kubespray handles everything that kubeadm does not — the OS preparation, container runtime installation, CNI deployment, and post-installation configuration.

Think of it as layers:

Manual installation: You do everything kubeadm: Automates cluster bootstrapping, you do the rest Kubespray: Automates everything, you configure variables

The Kubespray repository is organized as a standard Ansible project.

Each directory under roles contains the logic for one piece of the cluster. The bootstrap-os role configures the operating system. The container-engine role installs containerd. The etcd role sets up the etcd cluster. And so on.

The Double-Edged Sword

Here is the uncomfortable truth about automation: the more it does for you, the less you understand about what is actually happening.

When you run cluster.yml, Kubespray executes hundreds of tasks across dozens of roles. It loads kernel modules, writes configuration files, downloads binaries, generates certificates, creates systemd units, applies Kubernetes manifests, and configures networking. All of this happens behind a progress bar showing task names scrolling by.

If the playbook completes successfully, you get a working cluster. You can run kubectl get nodes and see your machines. You can deploy applications. Everything works.

But what happens when something goes wrong six months later?

A pod is not scheduling. You check the events, and they mention resource pressure. You look at the node, and something is consuming all the memory. Is it the kubelet configuration? The container runtime? A kernel parameter? Where do you even look?

Network traffic between pods is failing. Packets are being dropped somewhere. Is it the CNI plugin? An iptables rule? A kernel module that is not loaded? Which configuration file controls this behavior?

Certificates have expired. The API server is refusing connections. How do you renew them? Where are they stored? What tool generated them in the first place?

If you installed everything manually, you would know the answers because you typed every command yourself. If you used kubeadm with understanding, you would know that kubeadm certs renew exists and where the certificates live.

But if you just ran ansible-playbook cluster.yml without understanding what it does, you are now staring at a broken cluster with no idea how to fix it.

This is the black box problem. The automation that saved you hours during installation now costs you days during troubleshooting.

The Upgrade Trap

Kubernetes moves fast. A new minor version releases every four months. Security patches come out regularly. Running an outdated cluster is a security risk and eventually a compatibility problem as client tools and container images stop supporting old API versions.

Kubespray provides an upgrade playbook:

ansible-playbook -i inventory/mycluster/inventory.ini upgrade-cluster.yml -e kube_version=v1.31.0

In theory, you run this command and your cluster upgrades. In practice, upgrades are where black box automation fails hardest.

The upgrade playbook modifies your cluster in place. It drains nodes, updates binaries, restarts services, and hopes everything comes back up. If something fails partway through, you have a partially upgraded cluster in an inconsistent state.

Without understanding what the playbook is doing, you cannot:

• Diagnose why a particular step failed
• Manually complete the upgrade if automation cannot
• Roll back to the previous state safely
• Verify that the upgrade completed correctly

The result is that people who do not understand their automation become afraid to upgrade. They skip versions. They fall behind on security patches. The tool that was supposed to help them maintain their cluster becomes the reason they cannot maintain it.

Kubespray Itself Changes

This problem compounds because Kubespray is also evolving. New versions add roles, rename variables, change default values, and restructure configuration files.

If you learned Kubespray 2.20 and then try to use 2.28, you will find that:

• Some variable names have changed
• New configuration options exist that you have never seen
• Default behaviors are different
• The role structure may have been reorganized

Your muscle memory and mental model from the old version no longer apply. You need to relearn the tool, but you never really learned it deeply in the first place — you just ran the playbook.

The Temptation of “Just Make It Work”

Every developer has done this at some point. A deadline is approaching. Something needs to be running by tomorrow. You find a tool that claims to solve your problem, you run it, and it works. You move on.

For short-lived systems, this is fine. But Kubernetes clusters are not short-lived. They run for years. They get upgraded. They have nodes added and removed. They suffer hardware failures and network partitions. They need certificates renewed and configurations tuned.

If you do not understand how your cluster was built, you do not really own it. You are at the mercy of the tool. When the tool cannot help you, you are stuck.

I have seen this happen. At a previous job, we used Ansible extensively for infrastructure automation. Nobody on the team really understood Ansible deeply. We copied playbooks from examples, modified them until they worked, and ran them. When something broke, we spent hours debugging YAML indentation and variable precedence because we had never learned how Ansible actually evaluates variables.

The playbooks worked, but we did not understand why they worked. That meant we also did not understand why they failed.

The Philosophy of This Tutorial

This brings us to why this tutorial exists and how it approaches Kubespray.

We are not going to treat Kubespray as a magic button that produces clusters. We are going to open it up, look inside, and understand what it does.

When we run cluster.yml, we will trace through the plays and roles to see what tasks execute. When we configure variables, we will understand where those variables come from and how they affect the final cluster. When we deploy a cluster, we will verify that each component is running correctly and know where to look if it is not.

This takes longer than just running the playbook and hoping for the best. But it means that six months from now, when something breaks at 3 AM, you will have a chance of fixing it.

The goal is not to memorize every variable and task in Kubespray. That would be impossible — the kubespray_defaults role alone has over 800 lines of variables, and there are dozens of other roles. The goal is to understand the structure well enough that you know where to look when you need to find something.

Think of it like learning a new city. You do not memorize every street. But you learn the main neighborhoods, the major roads, and how the transit system works. Then when you need to get somewhere specific, you can figure it out.

What Kubespray Automates

To understand Kubespray, it helps to see exactly what manual work it replaces. If you have done Kubernetes The Hard Way, you will recognize these steps:

Kubernetes The Hard Way had you:

1. Provision compute resources (VMs or physical machines)
2. Generate TLS certificates for every component with OpenSSL
3. Create kubeconfig files for kubelet, controller-manager, scheduler, and admin
4. Generate a data encryption configuration
5. Bootstrap the etcd cluster with systemd units
6. Bootstrap the control plane components as systemd services
7. Configure kubelet and kube-proxy on worker nodes
8. Set up pod networking routes manually

kubeadm automated steps 2 through 6 into kubeadm init and step 8 into kubeadm join. But you still had to:

• Prepare the OS (disable swap, load kernel modules, set kernel parameters)
• Install the container runtime (containerd)
• Install kubeadm, kubelet, and kubectl
• Configure the container runtime for Kubernetes
• Install a CNI plugin
• Set up high availability if you wanted it

Kubespray automates all of this. When you run cluster.yml, these roles execute in sequence across your inventory hosts. The entire process typically takes 15–30 minutes depending on network speed and the number of nodes.

The Comparison That Matters

Here is a concrete comparison of deploying a basic cluster with each approach:

With Kubernetes The Hard Way, you run approximately 200 commands across multiple machines. You SSH into each node, copy files, edit configurations, and start services. The process takes several hours if you are careful and understand what you are doing.

With kubeadm, you still prepare each machine manually, but the cluster bootstrapping is reduced to:

On the first control plane node
kubeadm init - config=kubeadm-config.yaml

On each worker node
kubeadm join 192.168.10.10:6443 - token <token> - discovery-token-ca-cert-hash sha256:<hash>

You save significant time, but you still need to visit each machine.

With Kubespray, you run a single command from your Ansible control node:

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b

Kubespray connects to all machines simultaneously, configures them in parallel where possible, and handles the sequencing of operations automatically.

The time savings are dramatic. But remember: you are trading understanding for convenience. This tutorial exists to help you get both.

Chapter 2: Kubespray Core Concepts and Features

Let me start by reading you the official description from the Kubespray GitHub repository. The tagline says “Deploy a Production Ready Kubernetes Cluster.” This is not just marketing speak. It means Kubespray is designed to give you a cluster that you can actually run workloads on immediately, not a toy setup that needs hours of additional configuration.

The official documentation describes Kubespray as “a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks.” Pay attention to that phrase “domain knowledge.” This is what separates Kubespray from a simple shell script that runs kubeadm commands. The Kubespray maintainers have spent years figuring out what works in production environments, what kernel parameters matter, which CNI configurations cause problems, and how to handle edge cases. All of that accumulated wisdom is baked into the playbooks.

Kubespray is maintained by the Kubernetes SIG Cluster Lifecycle group, which means it is an official Kubernetes project. This is not some random GitHub repository that might disappear tomorrow.

One thing that confuses people is how Kubespray relates to kubeadm. They are not competing tools. Since version 2.3, Kubespray has used kubeadm internally for the actual cluster bootstrapping. The official documentation explains this decision:

“Kubespray has started using kubeadm internally for cluster creation since v2.3 in order to consume life cycle management domain knowledge from it and offload generic OS configuration things from it, which hopefully benefits both sides.”

So when you run Kubespray, it is not reinventing the wheel. It is using kubeadm to handle certificate generation, etcd configuration, and control plane bootstrapping. What Kubespray adds on top is everything else that kubeadm does not handle.

Think about what you still need to do manually when using kubeadm alone.

• Machine provisioning OS pre-configuration like time synchronization
• SELinux settings, swap disable, kernel parameters Installing containerd and kubelet
• Installing CNI plugins Setting up load balancers for HA configuration

Kubespray automates all of this. When you run the cluster.yml playbook, you get a complete cluster, not a half-finished one that requires more work.

Kubespray is not just for initial deployment. It handles the entire lifecycle of your cluster through different playbooks.

For creating a new cluster: ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b

For upgrading the cluster to a new Kubernetes version: ansible-playbook -i inventory/mycluster/inventory.ini upgrade-cluster.yml -b

For adding new worker nodes: ansible-playbook -i inventory/mycluster/inventory.ini scale.yml -b

For removing specific nodes: ansible-playbook -i inventory/mycluster/inventory.ini remove-node.yml -b -e node=worker-3

For completely resetting the cluster back to a clean state: ansible-playbook -i inventory/mycluster/inventory.ini reset.yml -b

Each of these playbooks is idempotent thanks to Ansible. If your deployment fails halfway through, you can run the same command again and Ansible will skip the tasks that already succeeded, continuing from where it left off. This is enormously valuable when you are dealing with flaky networks or transient failures.

One of Kubespray’s strongest features is its ability to deploy clusters anywhere. The same inventory structure and playbooks work whether you are deploying to AWS, GCP, Azure, a VMware vSphere environment, or bare metal servers in your data center.

For public cloud environments, the typical pattern is to use Terraform for infrastructure provisioning and then Kubespray for Kubernetes deployment:

terraform apply ansible-playbook cluster.yml

The Kubespray repository includes Terraform samples for various cloud providers if you want to see how this integration works.

For air-gapped environments where there is no internet access, Kubespray provides offline deployment support. This requires pre-downloading container images and binaries, then hosting them on an internal registry and file server.

The group_vars files include settings for configuring registry mirrors and custom download URLs:

In group_vars/all/containerd.yml
containerd_registries_mirrors:
prefix: docker.io mirrors:
host: https://registry.internal.company.com capabilities: ["pull", "resolve"]

The only hard requirements are that your Ansible control node can reach the target machines via SSH and that Python is installed on those machines.

Since Ansible is agentless and uses SSH, you do not need to install any agents or daemons on the cluster nodes before running Kubespray.

Production clusters need high availability. Kubespray supports HA configurations for both the control plane and etcd.

For control plane HA, you simply list multiple nodes in your inventory:

[kube_control_plane] controller-0 controller-1 controller-2
[etcd] controller-0 controller-1 controller-2
[kube_node] worker-0 worker-1 worker-2

Kubespray will configure all three control plane nodes with the API server, controller manager, and scheduler. The controller manager and scheduler use leader election, so only one instance is active at a time while the others stand by.

For etcd, Kubespray deploys a proper cluster using the Raft consensus protocol. You should always use an odd number of etcd nodes, typically three or five, to ensure quorum can be reached even if one node fails.

Now here is something important to understand about load balancing. Kubespray automates client-side load balancing but not external load balancing. What does this mean?

Client-side load balancing means each worker node runs a local nginx or haproxy that proxies requests to the available API servers. This is configured automatically by Kubespray. The kubelet on each worker connects to localhost, and the local proxy distributes requests across the control plane nodes.

External load balancing means having a load balancer in front of your control plane that external clients like kubectl on your laptop can connect to. Kubespray does not automatically configure this because external load balancing is completely different depending on your environment:

On AWS you would use an ELB or NLB On GCP you would use a GCP Load Balancer On bare metal you might use HAProxy with keepalived and a virtual IP In some environments you might use MetalLB or kube-vip

Kubespray is a tool for configuring software on top of your operating system. It is not an infrastructure provisioning tool. Configuring virtual IPs, DNS entries, and cloud load balancers is outside its scope.

The expectation is that you handle external load balancing through Terraform or manual configuration, and Kubespray handles everything from the OS level up.

That said, Kubespray does support kube-vip as an option for control plane VIP, which can provide external load balancing without requiring separate infrastructure. You can enable it in your addons configuration:

In group_vars/k8s_cluster/addons.yml
kube_vip_enabled: true

When you deploy a cluster with Kubespray, you get production-grade defaults without having to think about them. Remember all those manual steps from the kubeadm documentation about preparing your nodes? Kubespray handles them automatically.

Time synchronization is configured using chrony or ntp. In a distributed system like Kubernetes, clock skew between nodes causes all sorts of subtle problems with certificates, leases, and log correlation.

Kernel parameters are set correctly for container networking:

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1

The necessary kernel modules are loaded.

overlay br_netfilter: Swap is disabled, which is a hard requirement for kubelet to run properly.

These settings are defined in the kubernetes/preinstall role and applied automatically during cluster deployment. You do not need to SSH into each node and run sysctl commands manually.

When you run cluster.yml, here is what gets installed and configured on your nodes.

On all nodes:

• Kernel modules and sysctl parameters for container networking
• Container runtime (containerd by default)
• kubelet and kubectl binaries
• CNI plugins

On control plane nodes:

• kube-apiserver as a static pod
• kube-controller-manager as a static pod
• kube-scheduler as a static pod
• etcd (either as a systemd service or static pod depending on configuration)

On worker nodes:

• kube-proxy (as a DaemonSet)
• Client-side load balancer for API server access (nginx or haproxy)

Cluster-wide:

• CoreDNS for cluster DNS
• CNI plugin DaemonSet (Calico, Flannel, etc.)
• Optional addons like metrics-server, ingress controllers, helm

The configuration for all of this lives in your inventory directory. Once you have a working inventory, deploying the same cluster configuration repeatedly is just a matter of running the playbook against new infrastructure. This is infrastructure as code applied to Kubernetes deployment.

Chapter 3: Project Structure Overview

When you first clone the Kubespray repository and run ls, you're greeted with a wall of files and directories that can feel overwhelming. But here's the thing — Kubespray follows standard Ansible project conventions, so once you understand how Ansible organizes things, the whole structure clicks into place. Let me walk you through it piece by piece.

First, let’s get a bird’s eye view. Clone the repository and take a look:

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
git checkout release-2.28
ls -la

You’ll see something like this at the root level:

kubespray/
├── ansible.cfg
├── cluster.yml
├── reset.yml
├── scale.yml
├── upgrade-cluster.yml
├── remove-node.yml
├── recover-control-plane.yml
├── inventory/
├── roles/
├── playbooks/
├── library/
├── docs/
└── ...

Those YAML files sitting at the root — they’re your entry points. Each one represents a major cluster operation, and you’ll be running these directly with ansible-playbook.

The most important one is cluster.yml. This is what you run to create a new cluster from scratch. When you execute ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml, Kubespray kicks off the entire deployment process — from configuring the operating system to installing containerd, setting up etcd, deploying the Kubernetes control plane, joining worker nodes, installing CNI, and deploying addons like CoreDNS. We'll dissect this playbook in detail in Chapter 7.

Then there’s reset.yml. This one tears everything down. It's the nuclear option — it removes all Kubernetes components, wipes etcd data, and leaves your nodes in a clean state. I've used this more times than I'd like to admit during testing when something went sideways and I just needed a fresh start.

scale.yml is for adding new nodes to an existing cluster. Say you deployed with three workers and now you need five. You update your inventory file with the new nodes, then run scale.yml. It's smart enough to skip all the work that's already done on existing nodes and only configure the new ones.

remove-node.yml does the opposite. When you need to decommission a node — maybe the hardware is failing or you're downsizing — this playbook handles the graceful removal. It drains the node, removes it from the cluster, and cleans up.

upgrade-cluster.yml handles version upgrades. Kubernetes releases new versions every few months, and running this playbook walks your cluster through the upgrade process node by node, respecting the proper order (etcd first, then control plane, then workers).

recover-control-plane.yml is your emergency playbook. If a control plane node dies and you need to restore it, this is what you reach for.

Now, here’s something that might confuse you at first. You’ll notice some files exist in both hyphen and underscore versions:

ls -la | grep -E "remove|scale"

remove-node.yml
remove_node.yml
scale.yml

Both remove-node.yml and remove_node.yml exist. This isn't a mistake — it's backward compatibility. At some point, Kubespray standardized on kebab-case (hyphens), but they kept the old snake_case versions around so existing scripts and documentation wouldn't break. If you're writing new automation, use the hyphen versions.

The inventory Directory

This is where you define what your cluster looks like. Kubespray ships with a sample inventory that you’re meant to copy and customize:

ls inventory/

local/
sample/

The local/ directory is for single-node testing on your local machine. The sample/ directory is the template you'll copy for real deployments. Let's look inside:

tree inventory/sample/

inventory/sample/
├── inventory.ini
└── group_vars/
    ├── all/
    │   ├── all.yml
    │   ├── aws.yml
    │   ├── azure.yml
    │   ├── containerd.yml
    │   ├── coreos.yml
    │   ├── cri-o.yml
    │   ├── docker.yml
    │   ├── etcd.yml
    │   ├── gcp.yml
    │   ├── hcloud.yml
    │   ├── huaweicloud.yml
    │   ├── oci.yml
    │   ├── offline.yml
    │   ├── openstack.yml
    │   ├── upcloud.yml
    │   └── vsphere.yml
    └── k8s_cluster/
        ├── addons.yml
        ├── k8s-cluster.yml
        ├── k8s-net-calico.yml
        ├── k8s-net-cilium.yml
        ├── k8s-net-custom-cni.yml
        ├── k8s-net-flannel.yml
        ├── k8s-net-kube-ovn.yml
        ├── k8s-net-kube-router.yml
        ├── k8s-net-macvlan.yml
        └── kube_control_plane.yml

The inventory.ini file is where you list your actual hosts and assign them to groups. A typical production inventory might look like:

[all]
node1 ansible_host=10.0.1.10 ip=10.0.1.10
node2 ansible_host=10.0.1.11 ip=10.0.1.11
node3 ansible_host=10.0.1.12 ip=10.0.1.12
node4 ansible_host=10.0.1.20 ip=10.0.1.20
node5 ansible_host=10.0.1.21 ip=10.0.1.21

[kube_control_plane]
node1
node2
node3

[etcd:children]
kube_control_plane

[kube_node]
node4
node5

[k8s_cluster:children]
kube_control_plane
kube_node

Notice the [etcd:children] syntax. This is Ansible's way of creating nested groups. Instead of listing hosts directly under [etcd], we're saying "the etcd group contains all hosts from the kube_control_plane group." This pattern is convenient when your etcd nodes and control plane nodes are the same machines, which is the common "stacked etcd" topology.

The group_vars/ directory is where the magic happens. Ansible automatically loads variables from files in this directory based on which groups a host belongs to. Files under group_vars/all/ apply to every host. Files under group_vars/k8s_cluster/ apply only to hosts in the k8s_cluster group.

This is why you see so many YAML files in there. containerd.yml has container runtime settings. etcd.yml has etcd-specific configuration. k8s-cluster.yml has core Kubernetes settings like the network plugin, service CIDR, and pod CIDR. The cloud provider files (aws.yml, gcp.yml, azure.yml) contain settings that only matter if you're deploying to those environments — they're mostly commented out by default.

The roles Directory

Here’s where the actual work gets defined. Kubespray is essentially a collection of Ansible roles, and each role handles a specific piece of the cluster setup:

ls roles/

adduser/
bootstrap_os/
bootstrap-os/
container-engine/
download/
etcd/
etcdctl_etcdutl/
kubernetes/
kubernetes-apps/
kubespray_defaults/
kubespray-defaults/
network_plugin/
recover_control_plane/
remove-node/
reset/
upgrade/
win_nodes/
...

Let me explain the key ones.

bootstrap-os handles the initial operating system configuration. It installs Python (which Ansible needs), updates packages, sets up required kernel modules like overlay and br_netfilter, configures sysctl parameters for networking, and disables swap. All that tedious pre-work you'd do manually before running kubeadm — this role automates it.

container-engine installs the container runtime. By default that's containerd, but Kubespray supports Docker (via cri-dockerd) and CRI-O as well. This role downloads the binaries, creates the configuration files, sets up systemd units, and ensures the runtime is running.

etcd deploys the etcd cluster. Depending on your settings, it either installs etcd as a systemd service directly on the host or lets kubeadm manage it as a static pod. It handles certificate generation, cluster membership, and health checking.

kubernetes is actually a directory containing multiple sub-roles:

ls roles/kubernetes/

control-plane/
kubeadm/
node/
node-label/
node-taint/
preinstall/

The preinstall sub-role does Kubernetes-specific preparation — things like creating the kubernetes user, setting up directories, and configuring kubelet. The node sub-role installs kubelet, kubectl, and kubeadm on all nodes. The control-plane sub-role runs the kubeadm init process on control plane nodes. The kubeadm sub-role handles both the initial cluster bootstrap and joining additional nodes.

network_plugin installs your chosen CNI. Look inside and you'll see sub-directories for each supported CNI:

ls roles/network_plugin/

calico/
cilium/
flannel/
kube-ovn/
kube-router/
macvlan/
multus/
weave/
...

When you set kube_network_plugin: calico in your group_vars, Kubespray runs the tasks in roles/network_plugin/calico/. Switch it to flannel, and it runs roles/network_plugin/flannel/ instead.

kubernetes-apps deploys the addons — CoreDNS for cluster DNS, metrics-server for resource metrics, helm if you enable it, ingress controllers, cert-manager, and so on.

download is a critical role that handles fetching all the binaries and container images. It's designed to be idempotent and can work in both online and offline (air-gap) scenarios.

Now, about those confusingly similar names — kubespray_defaults versus kubespray-defaults. These are not duplicates. They serve different purposes, and understanding this distinction matters.

kubespray_defaults (with underscore) contains the actual variable definitions. Look at its structure:

tree roles/kubespray_defaults/

roles/kubespray_defaults/
├── defaults/
│   └── main/
│       ├── download.yml
│       └── main.yml
└── vars/
    └── main/
        ├── checksums.yml
        └── main.yml

The defaults/main/main.yml file alone is over 800 lines. It defines default values for nearly every configurable aspect of Kubespray — Kubernetes version, network settings, paths, timeouts, feature flags. The defaults/main/download.yml adds another 1100+ lines covering download URLs and version mappings. The vars/main/checksums.yml contains SHA256 checksums for all downloadable binaries — these are in the vars/ directory (higher priority) because you really shouldn't be changing checksums unless you know exactly what you're doing.

kubespray-defaults (with hyphen), on the other hand, contains tasks:

tree roles/kubespray-defaults/

roles/kubespray-defaults/
└── tasks/
    └── main.yml

This role’s job is to load and process those variables. It’s a pattern you’ll see throughout Kubespray — separate the data (variables) from the logic (tasks).

You’ll find similar underscore/hyphen pairs elsewhere. bootstrap_os holds variables while bootstrap-os holds tasks. It's a bit confusing at first, but once you recognize the pattern, it makes navigating the codebase easier.

The playbooks Directory

Remember those root-level playbooks like cluster.yml? They’re mostly thin wrappers that import the real playbooks from this directory:

ls playbooks/

ansible_version.yml
cluster.yml
facts.yml
install_etcd.yml
reset.yml
scale.yml
upgrade_cluster.yml
...

When you run the root cluster.yml, it imports playbooks/cluster.yml which contains the actual play definitions. This separation keeps the root directory clean while allowing the playbook logic to be more complex.

Let’s peek at what the root cluster.yml actually does:

cat cluster.yml | head -30

---
- name: Check Ansible version
  import_playbook: ansible_version.yml

- name: Add kube-master nodes to kube_control_plane
  # Backward compatibility
  import_playbook: legacy_groups.yml

- name: Common tasks for every playbooks
  import_playbook: boilerplate.yml

- name: Gather facts
  import_playbook: facts.yml

- name: Prepare for etcd install
  hosts: "{{ hostvars[groups['etcd'][0]]['etcd_retries'] | default(groups['etcd'][0]) }}:kube_control_plane"
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/preinstall, tags: preinstall }
    - { role: container-engine, tags: container-engine, when: deploy_container_engine }
    - { role: download, tags: download, when: "not skip_downloads" }

You can see it’s a sequence of import_playbook statements and play definitions. The structure is methodical — check Ansible version, handle legacy group names, run common boilerplate, gather facts, then proceed through each phase of cluster setup.

Here’s the complete picture of how these pieces connect when you run a deployment:

You run:
  ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml

Ansible reads:
  1. ansible.cfg (from current directory)
  2. inventory/mycluster/inventory.ini (hosts and groups)
  3. inventory/mycluster/group_vars/all/*.yml (global variables)
  4. inventory/mycluster/group_vars/k8s_cluster/*.yml (cluster variables)

cluster.yml imports and runs:
  → playbooks/ansible_version.yml (version check)
  → playbooks/boilerplate.yml (common setup)
  → playbooks/facts.yml (gather system info)
  → plays that execute roles:
      → roles/kubespray_defaults (load variables)
      → roles/bootstrap-os (OS setup)
      → roles/container-engine (containerd)
      → roles/etcd (etcd cluster)
      → roles/kubernetes/node (kubelet)
      → roles/kubernetes/control-plane (kubeadm init)
      → roles/network_plugin/calico (or flannel, etc.)
      → roles/kubernetes-apps (addons)

Each role follows the standard Ansible role structure:

roles/some-role/
├── defaults/
│   └── main.yml      # Default variable values (lowest priority)
├── vars/
│   └── main.yml      # Role variables (higher priority)
├── tasks/
│   └── main.yml      # The actual work
├── templates/
│   └── config.j2     # Jinja2 templates
├── handlers/
│   └── main.yml      # Handlers (restart services, etc.)
└── files/
    └── something     # Static files to copy

The roles execute in dependency order. You can’t install Kubernetes before containerd is running. You can’t join worker nodes before the control plane is up. Kubespray handles this orchestration through careful ordering in the playbooks.

For a quick reference, here’s how to list what’s in each major directory:

# Count files in each major area
echo "Roles:" && ls roles/ | wc -l
echo "Role defaults files:" && find roles -path "*/defaults/*.yml" | wc -l  
echo "Role vars files:" && find roles -path "*/vars/*.yml" | wc -l
echo "Sample group_vars/all:" && ls inventory/sample/group_vars/all/ | wc -l
echo "Sample group_vars/k8s_cluster:" && ls inventory/sample/group_vars/k8s_cluster/ | wc -l

On the 2.28 release, you’ll see something like:

Roles: 40+
Role defaults files: 77
Role vars files: 50
Sample group_vars/all: 16
Sample group_vars/k8s_cluster: 10

That’s a lot of moving parts, but they’re organized logically. When you need to understand or modify something specific — say, how containerd gets configured — you know to look in roles/container-engine/containerd/. When you want to change a default timeout or version, you check roles/kubespray_defaults/defaults/. When you want to customize your deployment, you edit files in inventory/mycluster/group_vars/.

The key insight is that Kubespray isn’t magic. It’s just a well-organized collection of Ansible automation following established patterns. Once you internalize the structure, you can navigate it confidently, troubleshoot issues effectively, and customize it for your environment without fear of breaking things you don’t understand.

Chapter 4: ansible.cfg Configuration Analysis

When you run ansible-playbook from the Kubespray directory, Ansible doesn’t just use its global defaults. It picks up a project-specific configuration file that Kubespray ships with. Understanding this file is important because it controls how Ansible behaves during cluster deployment — things like SSH connection handling, fact caching, output formatting, and performance optimizations.

Ansible looks for configuration files in a specific order. First, it checks if you’ve set the ANSIBLE_CONFIG environment variable. If that’s not set, it looks for ansible.cfg in your current working directory. Failing that, it checks your home directory for .ansible.cfg, and finally falls back to the system-wide /etc/ansible/ansible.cfg.

This is exactly why every Kubespray command in the documentation tells you to run ansible-playbook from the kubespray directory. If you run it from somewhere else, Ansible won’t find Kubespray’s ansible.cfg and will use different settings, which can lead to unexpected behavior or failures.

Let’s look at what Kubespray’s ansible.cfg actually contains:

[ssh_connection]
pipelining=True
ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
#control_path = ~/.ssh/ansible-%%r@%%h:%%p

[defaults]
force_valid_group_names = ignore
host_key_checking=False
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp
fact_caching_timeout = 86400
timeout = 300
stdout_callback = default
display_skipped_hosts = no
library = ./library
callbacks_enabled = profile_tasks
roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles:/usr/share/kubespray/roles
deprecation_warnings=False
inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo, .creds, .gpg

[inventory]
ignore_patterns = artifacts, credentials

That’s a lot to unpack. Let’s go through each section.

The SSH Connection Section

The [ssh_connection] section controls how Ansible connects to your target nodes. Kubespray deploys to multiple machines and runs hundreds of tasks, so SSH performance matters a lot here.

The first setting is pipelining=True. By default, when Ansible runs a task on a remote machine, it copies a Python script over, executes it, and closes the connection. With pipelining disabled, each task involves multiple round trips. With pipelining enabled, Ansible can send multiple commands through a single SSH session, reducing overhead significantly. For a playbook with 500+ tasks running across multiple nodes, this makes a real difference.

The ssh_args line packs several SSH client options together:

ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null

ControlMaster=auto enables SSH connection multiplexing. When you first connect to a host, SSH creates a master connection. Subsequent connections to the same host reuse this master connection instead of establishing new ones. This eliminates the authentication overhead for each connection.

ControlPersist=30m keeps that master connection alive for 30 minutes after the last session ends. During a Kubespray run, Ansible connects to the same nodes over and over. Keeping the master connection alive means those reconnections are nearly instant.

ConnectionAttempts=100 tells SSH to retry up to 100 times if a connection fails. This sounds excessive, but network hiccups happen, especially in cloud environments. You don’t want your 20-minute deployment to fail because of a momentary network blip.

The last one, UserKnownHostsFile=/dev/null, is worth understanding properly. Normally, SSH maintains a file called known_hosts in your ~/.ssh directory. The first time you connect to a server, SSH shows you a fingerprint and asks if you want to trust this host. If you say yes, it saves the fingerprint to known_hosts. On subsequent connections, SSH checks that the fingerprint matches. If it doesn’t match, SSH refuses to connect because this could indicate a man-in-the-middle attack.

Setting UserKnownHostsFile=/dev/null means SSH writes the fingerprint to /dev/null — the black hole device that discards everything written to it. The fingerprint is never saved, so every connection is treated as a first-time connection. Combined with host_key_checking=False (which we’ll see in the defaults section), this means Ansible never prompts you to verify host keys and never complains about changed fingerprints.

Why would Kubespray want this? In cloud and VM environments, you often tear down and recreate machines. The same IP address might point to completely different machines over time. If Ansible kept checking host keys, you’d constantly get “HOST KEY VERIFICATION FAILED” errors and have to manually clean up your known_hosts file. For automated deployment, this is a practical necessity.

That said, this does reduce security. In a production environment where hosts are long-lived and you want to detect if someone is intercepting your SSH traffic, you might want to reconsider this setting. But for most Kubespray use cases — spinning up clusters, tearing them down, trying different configurations — the convenience outweighs the security tradeoff.

The control_path line is commented out:

#control_path = ~/.ssh/ansible-%%r@%%h:%%p

This would specify where SSH stores its control socket files for connection multiplexing. The %% patterns expand to the remote user, hostname, and port. Kubespray leaves this commented because Ansible manages control paths automatically, and the default behavior works fine for most setups.

The Defaults Section

The [defaults] section contains most of the interesting configuration. Let’s break it down.

force_valid_group_names = ignore

This one has a story behind it. In Ansible 2.8, the developers added stricter validation for inventory group names. They decided that group names should follow Python variable naming rules — no hyphens, no dots. The reasoning was that group names sometimes get used in Jinja2 templates or as variable names, and special characters could cause problems.

The thing is, Kubernetes uses hyphens everywhere. Namespaces like kube-system, pod names like coredns-12345, node names with hyphens. If you’re using Kubespray with an inventory that reflects your Kubernetes naming conventions, you’ll have groups with hyphens in their names.

When Ansible 2.8 came out, people started getting warnings about invalid group names. Some people’s playbooks broke entirely. There was a GitHub issue (#56930) where users complained, pointing out that group names aren’t variable names — they’re just strings that happen to be used as dictionary keys. The Kubernetes ecosystem had already standardized on hyphen-separated names, and forcing everyone to use underscores instead wasn’t practical.

Setting force_valid_group_names = ignore tells Ansible to accept any group name without complaining. Kubespray needs this to work seamlessly with Kubernetes naming conventions.

host_key_checking=False

This complements the UserKnownHostsFile=/dev/null setting from the SSH section. When SSH connects to a new host, it normally asks “Are you sure you want to continue connecting?” and waits for you to type yes. With host_key_checking disabled, Ansible automatically accepts new host keys without prompting. Combined with the /dev/null known_hosts file, this makes SSH connections completely non-interactive.

gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp
fact_caching_timeout = 86400

These four lines work together to optimize fact gathering. Facts are system information that Ansible collects from each host — things like the operating system, IP addresses, memory size, CPU count. Collecting facts requires running commands on each host, which takes time.

With gathering = smart, Ansible only collects facts when it doesn’t already have them cached. The fact_caching = jsonfile setting tells Ansible to cache facts as JSON files. fact_caching_connection = /tmp specifies where to store those JSON files. And fact_caching_timeout = 86400 means cached facts are valid for 86400 seconds — that’s 24 hours.

During a Kubespray run, Ansible connects to the same hosts multiple times across different plays. Without caching, it would collect facts every single time, adding minutes to the total runtime. With smart gathering and caching, facts are collected once and reused throughout the run.

timeout = 300

This sets the SSH connection timeout to 300 seconds (5 minutes). If a host doesn’t respond within 5 minutes, Ansible gives up on that connection. This is pretty generous, but it accommodates slow networks or hosts that take a while to become available.

stdout_callback = default
display_skipped_hosts = no

These control how Ansible displays output. The default callback gives you the standard Ansible output format. You could change this to yaml or json if you preferred different formatting.

display_skipped_hosts = no is a nice quality-of-life setting. Kubespray has many tasks that only run on certain host types. A task that configures etcd will be skipped on worker nodes. A task that installs kubelet runs everywhere. With display_skipped_hosts enabled, you’d see endless “skipping: [worker-1]” messages cluttering your output. Disabling it keeps the output focused on what’s actually happening.

library = ./library

Ansible lets you write custom modules when the built-in ones don’t do what you need. Kubespray includes a custom module called kube.py in its library directory. This module wraps kubectl commands, letting Ansible tasks manage Kubernetes resources directly. The library = ./library setting tells Ansible to look in that directory for custom modules.

callbacks_enabled = profile_tasks

This enables the profile_tasks callback plugin. When your playbook finishes, you’ll see a summary showing how long each task took:

PLAY RECAP ****
Tuesday 28 January 2026  15:23:45 +0900 (0:00:02.456)

===============================================================================
Download containerd -------------------------------- 45.23s
Configure kubelet ---------------------------------- 12.67s
Download images ------------------------------------ 89.45s

This is incredibly useful for understanding where time goes during deployment. If your cluster deployment is taking too long, this output tells you which tasks are the bottleneck.

roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles:/usr/share/kubespray/roles

This tells Ansible where to look for roles, in order of priority. The paths are separated by colons. Ansible checks the first path first, then moves on if it doesn’t find the role.

The first path, roles, is the local roles directory in the Kubespray project. This is where all of Kubespray’s roles live, and it has the highest priority.

The paths containing $VIRTUAL_ENV are for when you install Kubespray as a Python package into a virtual environment. The $VIRTUAL_ENV variable gets set automatically when you activate a Python virtual environment. If you’re just running Kubespray from a git clone (which is the most common approach), these paths won’t match anything and get ignored.

The last path, /usr/share/kubespray/roles, is for system-wide installations.

deprecation_warnings=False

Ansible loves to warn you about deprecated features. While these warnings are useful when you’re developing playbooks, they add noise when you’re just running Kubespray. Kubespray’s maintainers keep the playbooks updated, so you can trust that deprecated features will be fixed in future releases.

inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo, .creds, .gpg

When Ansible scans your inventory directory, it ignores files with these extensions. This prevents backup files, compiled Python files, and credential files from being accidentally parsed as inventory.

The Inventory Section

[inventory]
ignore_patterns = artifacts, credentials

Similar to inventory_ignore_extensions, this tells Ansible to skip directories matching these patterns when scanning for inventory files. The artifacts directory typically contains deployment outputs, and credentials contains sensitive files. Neither should be treated as inventory sources.

Customizing ansible.cfg

Kubespray’s default ansible.cfg works well for most situations, but you might want to adjust it for your environment. The key is understanding that you can override settings without modifying the original file.

One approach is to use the ANSIBLE_CONFIG environment variable:

ANSIBLE_CONFIG=ansible.cfg.custom ansible-playbook cluster.yml

You could maintain different configuration files for different environments:

kubespray/
├── ansible.cfg           # default
├── ansible.cfg.dev       # development (more verbose, shorter timeouts)
└── ansible.cfg.prod      # production (stricter security settings)

Some settings you might want to change:

For a more secure environment where you do want host key checking:

[defaults]
host_key_checking = True

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=30m
# Removed UserKnownHostsFile=/dev/null

For CI/CD pipelines where you need machine-parseable output:

[defaults]
stdout_callback = json

For debugging, to see which hosts are being skipped:

[defaults]
display_skipped_hosts = yes

For large clusters where you want more parallelism:

[defaults]
forks = 20  # default is 5

The forks setting controls how many hosts Ansible manages simultaneously. The default of 5 is conservative. If you have the network bandwidth and your control machine has enough resources, increasing this can speed up deployments to large clusters significantly.

If you modify ansible.cfg, keep a few things in mind. First, always keep a copy of the original so you can revert if something breaks. Second, document your changes with comments explaining why you made them. Third, if you’re version-controlling your Kubespray setup, include your modified ansible.cfg so team members get the same behavior.

The ansible.cfg file might seem like a minor detail compared to the actual playbooks and roles, but these settings affect every single task Ansible runs. Understanding them helps you troubleshoot problems, optimize performance, and adapt Kubespray to your specific environment.

Chapter 5: Variable System and Precedence Strategy

If you have ever spent hours debugging why a variable in Kubespray does not behave the way you expected, you are not alone. Kubespray has hundreds of variables spread across dozens of files, and understanding where to look and what takes priority over what is essential before you start customizing anything. This chapter will walk you through Ansible’s variable precedence system and show you exactly how Kubespray leverages it to create a maintainable, layered configuration architecture.

The fundamental principle is simple. Ansible has a 22-level priority system, and Kubespray deliberately places its variables at specific levels so that you, the operator, can override them without touching the core codebase. Once you understand this design, everything clicks into place.

Ansible evaluates variables from multiple sources and applies them in a strict order. When the same variable appears in multiple places, the one with higher priority wins. Here is the complete list, numbered from lowest to highest priority:

1.  command line values (for example, -u my_user)
2.  role defaults (roles/*/defaults/main.yml)
3.  inventory file or script group vars
4.  inventory group_vars/all
5.  playbook group_vars/all
6.  inventory group_vars/*
7.  playbook group_vars/*
8.  inventory file or script host vars
9.  inventory host_vars/*
10. playbook host_vars/*
11. host facts / cached set_facts
12. play vars
13. play vars_prompt
14. play vars_files
15. role vars (roles/*/vars/main.yml)
16. block vars
17. task vars
18. include_vars
19. set_facts / registered vars
20. role parameters
21. include parameters
22. extra vars (-e) (always win)

The magic number to remember is 22. Extra vars passed via the command line with -e always win. The second thing to remember is that role defaults sit at priority 2, the absolute bottom of the hierarchy. This is not an accident. Ansible designed it this way so that role authors could provide sensible defaults while making it trivial for users to override them from almost anywhere.

If you visualize this as a stack, it looks like this:

┌─────────────────────────┐
│   extra_vars (-e)       │  22 ← highest priority
├─────────────────────────┤
│   task vars             │  17
├─────────────────────────┤
│   role/vars/            │  15
├─────────────────────────┤
│   play vars             │  12
├─────────────────────────┤
│   host_vars/            │  9
├─────────────────────────┤
│   group_vars/*          │  6
├─────────────────────────┤
│   group_vars/all        │  4
├─────────────────────────┤
│   role/defaults/        │  2  ← lowest priority
└─────────────────────────┘

When you set kube_version: v1.30.0 in a role's defaults and then set kube_version: v1.31.0 in your inventory's group_vars, the group_vars value wins because priority 4 or 6 beats priority 2. If you then run the playbook with -e "kube_version=v1.32.0", that value wins over everything because priority 22 is king.

Mapping Precedence to Kubespray Files

Now let us connect these abstract priority levels to actual files in the Kubespray repository. This mapping is crucial for debugging and for knowing where to put your customizations.

At priority level 2, you have role defaults. In Kubespray, these live in paths like:

roles/kubespray_defaults/defaults/main/main.yml
roles/kubespray_defaults/defaults/main/download.yml
roles/etcd/defaults/main.yml
roles/kubernetes/node/defaults/main.yml

The kubespray_defaults role alone contains over 800 lines of variables in main.yml and over 1100 lines in download.yml. These are the sensible defaults that Kubespray provides out of the box.

At priority level 4, you have inventory group_vars/all. In a typical Kubespray setup, this maps to:

inventory/mycluster/group_vars/all/all.yml
inventory/mycluster/group_vars/all/etcd.yml
inventory/mycluster/group_vars/all/containerd.yml

At priority level 6, you have inventory group_vars for specific groups:

inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
inventory/mycluster/group_vars/k8s_cluster/addons.yml
inventory/mycluster/group_vars/k8s_cluster/k8s-net-calico.yml

At priority level 15, you have role vars. These are internal fixed values that Kubespray does not want you to accidentally override:

roles/kubespray_defaults/vars/main/checksums.yml
roles/kubespray_defaults/vars/main/main.yml

Let me show you how many files exist at each level. Run these commands from the Kubespray root directory:

find roles -type f -name "*.yml" | grep "/defaults/" | wc -l

This returns around 77 files. These are all the default variable files across every role.

find inventory/sample -path "*/group_vars/all/*.yml" | wc -l

This returns around 16 files in the sample inventory’s all directory.

find inventory/sample -path "*/group_vars/k8s_cluster/*.yml" | wc -l

This returns around 10 files for cluster-specific settings.

find roles -type f -name "*.yml" | grep "/vars/" | wc -l

This returns around 50 files containing internal fixed values.

Kubespray’s Intentional Design

Here is where it gets interesting. Kubespray deliberately places variables at specific priority levels to create a clean separation of concerns.

Role defaults at priority 2 contain the base configuration. These are values that work for most deployments. The Kubespray maintainers chose priority 2 because it is the lowest level, meaning you can override these values from almost anywhere.

Inventory group_vars at priority 4 and 6 are where you, the operator, put your customizations. When you copy the sample inventory to create your own cluster configuration, you are creating your override layer. Anything you define here beats the role defaults.

Role vars at priority 15 contain values that should not be overridden casually. The checksums.yml file is a perfect example. It contains SHA256 hashes for every binary that Kubespray downloads. If you accidentally override a checksum, the integrity verification fails and your deployment breaks. By placing checksums at priority 15, Kubespray ensures that your group_vars settings at priority 4–6 cannot accidentally clobber them.

The flow looks like this:

[role defaults] ──override──→ [inventory group_vars] ──override──→ [extra_vars]
    基本値                         your settings                    临时设定

Let me show you a concrete example. The variable override_system_hostname controls whether Kubespray sets the hostname on your nodes. In the role defaults, you will find:

# roles/bootstrap_os/defaults/main.yml
override_system_hostname: true

This is the default behavior. If you want to disable it for your cluster, you add this to your inventory:

# inventory/mycluster/group_vars/all/all.yml
override_system_hostname: false

Your value at priority 4 overrides the default at priority 2. You did not have to modify any Kubespray source code. When you upgrade Kubespray to a newer version, your customization remains intact in your inventory directory while the role defaults get updated.

The kubespray_defaults Role

The kubespray_defaults role deserves special attention because it is the single largest collection of variables in the entire project.

Look at its structure:

roles/kubespray_defaults/
├── defaults/main/
│   ├── main.yml        # 801 lines - core defaults
│   └── download.yml    # 1139 lines - download URLs, versions
└── vars/main/
    ├── main.yml        # internal paths
    └── checksums.yml   # binary integrity checksums

The defaults/main/main.yml file contains the core configuration for your cluster. Open it and you will see variables like:

kube_version: v1.31.0
container_manager: containerd
kube_network_plugin: calico
kube_proxy_mode: ipvs
etcd_deployment_type: host

The defaults/main/download.yml file contains download URLs and version mappings for every component. It tells Kubespray where to fetch containerd, runc, crictl, etcd, and dozens of other binaries.

The vars/main/checksums.yml file is massive. It contains checksums for every supported version of every component, organized by architecture.

A small excerpt looks like:

crictl_checksums:
  amd64:
    v1.29.0: sha256:abc123...
    v1.30.0: sha256:def456...
  arm64:
    v1.29.0: sha256:789abc...
    v1.30.0: sha256:012def...

The reason checksums live in vars/ instead of defaults/ is protection. At priority 15, they are harder to accidentally override. If you need to add a new version that Kubespray does not support yet, you would need to use extra_vars at priority 22 or modify the file directly. This friction is intentional.

Play Vars and Task Vars in Kubespray

Beyond role defaults and group_vars, Kubespray also uses play vars and task vars in specific situations. These are less common but worth understanding.

Play vars appear at priority 12. In Kubespray, they are used sparingly. I searched through the playbooks and found only four variables defined this way:

# playbooks/cluster.yml (around line 20-22)
- name: Install etcd
  vars:
    etcd_cluster_setup: true
    etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
  import_playbook: install_etcd.yml

The same pattern appears in upgrade_cluster.yml and scale.yml. Why does Kubespray define etcd_cluster_setup as a play var when there is already a default value in the role?

The answer lies in scale.yml. When you add a new node to an existing cluster, you do not want to reinstall etcd. The scale playbook sets etcd_cluster_setup: false to skip etcd installation. The cluster playbook sets it to true to perform the installation. By using play vars, Kubespray can reuse the same install_etcd.yml playbook with different behavior depending on which parent playbook invoked it.

# playbooks/scale.yml
- name: Install etcd
  vars:
    etcd_cluster_setup: false
    etcd_events_cluster_setup: false
  import_playbook: install_etcd.yml

The ansible_version.yml playbook uses play vars to define minimum and maximum supported Ansible versions:

# playbooks/ansible_version.yml
- name: Check Ansible version
  hosts: localhost
  vars:
    minimal_ansible_version: "2.14.0"
    maximal_ansible_version: "2.17.99"

Task vars at priority 17 appear more frequently but serve a narrow purpose. Most task vars in Kubespray are parameters passed to included tasks.

For example:

# roles/container-engine/containerd/tasks/main.yml
- name: Download containerd
  include_tasks: "../download/tasks/download_file.yml"
  vars:
    download: "{{ containerd_download }}"

The download variable is scoped to that specific task inclusion. It does not override any global variable because nothing else uses a variable simply named download at higher levels. This is a parameter-passing pattern, not a global override pattern.

I checked whether any task vars conflict with group_vars or defaults. They do not. The variable names used in task vars like download, iface, and etcd_peer_addresses are local parameters that exist only within the scope of that task.

Variable Categories from Documentation

The official Kubespray documentation at docs/ansible/vars.md categorizes variables into several groups. While I recommend reading the actual files rather than memorizing documentation, knowing these categories helps when you are searching for a specific setting.

Generic Ansible Variables are facts that Ansible collects automatically. The most commonly referenced one is:

ansible_default_ipv4.address

This is the IP address that Ansible detects as your node’s primary address. Kubespray uses this as a fallback when you do not specify the ip variable in your inventory. You can see how Ansible determines this by running:

ip -4 route get 8.8.8.8

On a typical VirtualBox VM, this returns the NAT interface (10.0.2.15), which is why you must explicitly set the ip variable in multi-NIC environments.

Common Vars are the variables you will modify most often:

kube_version: v1.31.0
kube_network_plugin: calico
kube_proxy_mode: ipvs
container_manager: containerd
etcd_version: v3.5.25

Container runtime variables depend on which runtime you choose. If you set container_manager: containerd, Kubespray uses containerd_version. If you set container_manager: docker, it uses docker_version and docker_containerd_version.

Addressing Variables control how nodes communicate:

ip: 192.168.10.10
access_ip: 192.168.10.10
loadbalancer_apiserver: 192.168.10.100

The ip variable is what the node uses for internal cluster communication. The access_ip is what other nodes use to reach this node. In most cases they are the same, but in environments with separate management and data networks, they might differ.

Calico-specific variables live in a separate category if you use Calico as your CNI:

calico_ipip_mode: Always
calico_vxlan_mode: Never
calico_network_backend: bird

The Customization Workflow

Now that you understand where variables live and how priority works, let me walk you through the workflow for customizing a Kubespray deployment.

First, recognize what you should and should not modify. This table is critical:

Location                              | Modify? | Reason
--------------------------------------|---------|----------------------------------
inventory/mycluster/group_vars/       | YES     | Your customization area
inventory/mycluster/host_vars/        | YES     | Host-specific overrides
-e command line option                | YES     | Temporary overrides
playbooks/*.yml                       | NO      | Kubespray code, gets overwritten
roles/*/defaults/                     | NO      | Kubespray code, gets overwritten
roles/*/vars/                         | NO      | Internal fixed values

When you clone Kubespray and run git pull or switch to a newer release, everything in playbooks/ and roles/ gets updated. But your inventory/mycluster/ directory is separate. It survives upgrades because you created it by copying from inventory/sample/.

The three-step workflow goes like this.

Step one: Copy the sample inventory.

cp -rfp inventory/sample inventory/mycluster

The sample inventory is a template. It contains some variables with values, some commented out as hints, and many variables not present at all (meaning they use role defaults).

Step two: Find the variable you want to change by checking role defaults.

grep -r "kube_version" roles/*/defaults/ | head -5

This shows you where kube_version is defined and what its default value is:

roles/kubespray_defaults/defaults/main/main.yml:kube_version: v1.31.0

Step three: Override it in your group_vars.

# inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
kube_version: v1.30.0

You only define the variables you want to change. Everything else uses the defaults. This keeps your configuration minimal and makes upgrades easier because you are not duplicating values that you did not need to modify.

Let me show you a complete example. Suppose you want to change the Kubernetes version from the default v1.31.0 to v1.30.0.

First, verify the default:

grep "kube_version" roles/kubespray_defaults/defaults/main/main.yml

Output:

kube_version: v1.31.0

Next, check if the sample inventory already has this variable:

grep "kube_version" inventory/sample/group_vars/k8s_cluster/k8s-cluster.yml

Output:

# kube_version: v1.31.0

The sample has it commented out, which means it is using the default. The comment serves as documentation showing you that this variable exists and can be changed.

Now add your override:

# inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
kube_version: v1.30.0

When you run ansible-playbook cluster.yml, Ansible loads the role default (v1.31.0) at priority 2, then loads your group_vars (v1.30.0) at priority 6, and the higher priority wins. Kubernetes 1.30.0 gets installed.

If you need to temporarily test a different version without modifying files, use extra vars:

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml \
  -e "kube_version=v1.32.0"

Priority 22 beats everything. The cluster deploys with v1.32.0 regardless of what your files say.

Key Files You Will Modify

Let me give you a practical reference for the files you will interact with most often.

The inventory/mycluster/group_vars/all/all.yml file contains global settings that apply to every node, including etcd nodes. Common settings here include:

bin_dir: /usr/local/bin
loadbalancer_apiserver_port: 6443
ntp_enabled: true
ntp_servers:
  - "0.pool.ntp.org iburst"
  - "1.pool.ntp.org iburst"
unsafe_show_logs: false

The inventory/mycluster/group_vars/all/etcd.yml file controls etcd deployment:

etcd_data_dir: /var/lib/etcd
etcd_deployment_type: host

The etcd_deployment_type is particularly important. Setting it to host means etcd runs as a systemd service outside of Kubernetes. Setting it to kubeadm means etcd runs as a static pod managed by kubeadm. The host option gives you more control and independence from kubeadm's lifecycle management.

The inventory/mycluster/group_vars/all/containerd.yml file configures the container runtime. Most options are commented out in the sample, meaning you use defaults:

# containerd_storage_dir: "/var/lib/containerd"
# containerd_state_dir: "/run/containerd"
# containerd_oom_score: 0
# containerd_default_runtime: "runc"

The inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml file is the most important one. This is where you configure the Kubernetes cluster itself:

kube_config_dir: /etc/kubernetes
kube_network_plugin: calico
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
kube_proxy_mode: ipvs
dns_mode: coredns
enable_nodelocaldns: true
container_manager: containerd
auto_renew_certificates: false

The inventory/mycluster/group_vars/k8s_cluster/addons.yml file controls which add-ons get installed:

helm_enabled: false
metrics_server_enabled: false
ingress_nginx_enabled: false
cert_manager_enabled: false
metallb_enabled: false
argocd_enabled: false

Every addon has additional configuration options below its enable flag. For example, if you enable metrics_server, you can configure its resource requests, replica count, and other settings in the same file.

The network plugin files like k8s-net-calico.yml and k8s-net-flannel.yml contain CNI-specific settings. The file that gets used depends on your kube_network_plugin value. If you set kube_network_plugin: flannel, then k8s-net-flannel.yml becomes relevant.

Understanding Scope: all vs k8s_cluster

A subtle but important detail is the difference between group_vars/all/ and group_vars/k8s_cluster/.

Variables in group_vars/all/ apply to every host in your inventory. This includes etcd nodes, control plane nodes, and worker nodes.

Variables in group_vars/k8s_cluster/ apply only to hosts that belong to the k8s_cluster group. Looking at a typical inventory:

[kube_control_plane]
k8s-ctr

[etcd]
k8s-ctr

[kube_node]
k8s-ctr

[k8s_cluster:children]
kube_control_plane
kube_node

The k8s_cluster group is defined using the :children suffix, which means it includes all hosts from kube_control_plane and kube_node. In this single-node example, k8s-ctr belongs to all groups. But in a multi-node setup where you have dedicated etcd nodes that are not part of the Kubernetes cluster, the etcd nodes would be in the etcd group but not in k8s_cluster.

This distinction matters because some settings only make sense for Kubernetes nodes. The kube_proxy_mode variable, for example, should only apply to nodes running kube-proxy. Putting it in group_vars/k8s_cluster/ ensures it does not get applied to standalone etcd nodes.

The priority difference also matters. Remember that group_vars/all is priority 4 and group_vars/k8s_cluster is priority 6. If you somehow define the same variable in both places, the k8s_cluster value wins for nodes that belong to that group.

One practical question that comes up is how to know which Kubernetes versions Kubespray supports. The answer is in the checksums file.

cat roles/kubespray_defaults/vars/main/checksums.yml | grep -A 20 "kubelet_checksums:"

This shows you the kubelet checksums, and each version listed is a supported version. If you try to set kube_version to a value that does not have a corresponding checksum, the download task will fail because Kubespray cannot verify the binary integrity.

When a new Kubernetes version comes out, the Kubespray maintainers add its checksums to this file. Until they do, you cannot use that version unless you bypass the checksum verification (not recommended) or add the checksums yourself using extra_vars at priority 22.

When a variable does not behave as expected, here is how to debug it.

First, check what value Ansible is actually using. Add a debug task to your playbook or run:

ansible -i inventory/mycluster/inventory.ini all -m debug -a "var=kube_version"

This shows you the resolved value of kube_version for each host.

Second, trace where the value comes from. Ansible does not have a built-in “show me which file set this variable” feature, but you can search:

grep -r "kube_version" inventory/mycluster/
grep -r "kube_version" roles/*/defaults/
grep -r "kube_version" roles/*/vars/

Compare the search results with the priority list. The highest priority source wins.

Third, check for typos. A common mistake is defining kube_versions (plural) when the variable is kube_version (singular). Ansible silently ignores variables it does not recognize, so your setting has no effect.

Fourth, verify the file is being loaded. If you created a new file in group_vars but named it without the .yml extension, Ansible ignores it. The file must end in .yml or .yaml.

Chapter 6: Group Variables Deep Dive

I explained how Kubespray organizes its variable system and why the precedence matters. Now we get into the actual files you will be editing. When you copy inventory or sample to inventory/mycluster, you inherit a carefully structured set of configuration files that control every aspect of your cluster. Understanding what each file does and which variables live where saves you from the frustrating experience of changing something in the wrong place and wondering why nothing happened.

The group_vars Directory Structure

After copying the sample inventory, your mycluster directory looks like this:

inventory/mycluster/
├── group_vars
│   ├── all
│   │   ├── all.yml
│   │   ├── aws.yml
│   │   ├── azure.yml
│   │   ├── containerd.yml
│   │   ├── coreos.yml
│   │   ├── cri-o.yml
│   │   ├── docker.yml
│   │   ├── etcd.yml
│   │   ├── gcp.yml
│   │   ├── hcloud.yml
│   │   ├── huaweicloud.yml
│   │   ├── oci.yml
│   │   ├── offline.yml
│   │   ├── openstack.yml
│   │   ├── upcloud.yml
│   │   └── vsphere.yml
│   └── k8s_cluster
│       ├── addons.yml
│       ├── k8s-cluster.yml
│       ├── k8s-net-calico.yml
│       ├── k8s-net-cilium.yml
│       ├── k8s-net-custom-cni.yml
│       ├── k8s-net-flannel.yml
│       ├── k8s-net-kube-ovn.yml
│       ├── k8s-net-kube-router.yml
│       ├── k8s-net-macvlan.yml
│       └── kube_control_plane.yml
└── inventory.ini

Two directories exist under group_vars: all and k8s_cluster. This split is not arbitrary. It follows Ansible’s group variable system where variables defined in a directory apply only to hosts belonging to that group.

How Ansible Group Variables Work

The group_vars/all directory contains variables that apply to every single host in your inventory. When Ansible runs a play against any host, it loads all the YAML files from group_vars/all and makes those variables available.

The group_vars/k8s_cluster directory is more specific. It only applies to hosts that belong to the k8s_cluster group. If you look at a typical Kubespray inventory, you will see something like this:

[kube_control_plane]
k8s-ctr

[etcd]
k8s-ctr

[kube_node]
k8s-ctr

[k8s_cluster:children]
kube_control_plane
kube_node

The k8s_cluster group is defined using the :children suffix, which means it includes all hosts from kube_control_plane and kube_node groups. So any host that is either a control plane node or a worker node gets the variables from group_vars/k8s_cluster.

This distinction matters for etcd. If you run etcd on separate dedicated nodes that are not part of kube_control_plane or kube_node, those etcd hosts would get variables from group_vars/all but not from group_vars/k8s_cluster. The separation allows you to configure etcd-specific settings without accidentally applying Kubernetes cluster settings to standalone etcd nodes.

Variable Precedence Between Groups

Ansible has a rule: more specific groups override less specific ones. The group_vars/all directory has priority 4 in Ansible’s precedence system, while group_vars/k8s_cluster has priority 6. This means if the same variable is defined in both places, the k8s_cluster version wins for hosts that belong to that group.

In practice, you rarely define the same variable in both places. Kubespray’s sample files are organized so that truly global settings go in all and cluster-specific settings go in k8s_cluster. But knowing this precedence helps when you need to override something for just the Kubernetes nodes without affecting standalone etcd hosts.

Conditional File Application

Not every file in group_vars gets used in every deployment. Many files are conditional based on other variable values.

For cloud provider files like aws.yml, azure.yml, and gcp.yml, the contents only matter if you set cloud_provider to that specific provider. If you are running on bare metal or a different cloud, these files are essentially ignored even though they exist.

Similarly, the container runtime files are conditional. The containerd.yml settings apply when container_manager is set to containerd. The cri-o.yml settings apply when you choose CRI-O instead. The docker.yml file is there for legacy Docker support through cri-dockerd, though this path is deprecated since Kubernetes 1.24 removed native Docker support.

The network plugin files in k8s_cluster follow the same pattern. The k8s-net-calico.yml file only matters when kube_network_plugin is calico. Switch to flannel and suddenly k8s-net-flannel.yml becomes relevant while the Calico file is ignored.

This conditional system is elegant because you can see all available options in your inventory without them interfering with each other. You pick your choices in k8s-cluster.yml and the corresponding configuration files automatically become active.

The group_vars/all Directory

This directory contains settings that every node needs regardless of its role.

all.yml — Global Settings

The all.yml file is where you configure settings that genuinely apply everywhere. Running grep to see only the non-commented lines:

grep "^[^#]" inventory/mycluster/group_vars/all/all.yml

You get output like:

---
bin_dir: /usr/local/bin
loadbalancer_apiserver_port: 6443
loadbalancer_apiserver_healthcheck_port: 8081
no_proxy_exclude_workers: false
kube_webhook_token_auth: false
kube_webhook_token_auth_url_skip_tls_verify: false
ntp_enabled: false
ntp_manage_config: false
ntp_servers:
  - "0.pool.ntp.org iburst"
  - "1.pool.ntp.org iburst"
  - "2.pool.ntp.org iburst"
  - "3.pool.ntp.org iburst"
unsafe_show_logs: false
allow_unsupported_distribution_setup: false

The bin_dir setting tells Kubespray where to install binaries like kubectl, kubelet, and kubeadm. The default /usr/local/bin works for most systems. Change this if your organization has a different standard location for locally installed binaries.

The loadbalancer_apiserver_port defaults to 6443, which is the standard Kubernetes API server port. In HA setups with an external load balancer, this is the port your load balancer listens on.

The ntp_enabled setting is false by default, which might surprise you. Time synchronization is critical for Kubernetes, especially for certificate validation and etcd consistency. Kubespray assumes your nodes already have NTP configured through your base OS provisioning. If you want Kubespray to manage NTP, set this to true and it will configure chrony or ntpd with the servers listed in ntp_servers.

The unsafe_show_logs setting controls whether sensitive information appears in Ansible output. Keep this false in production. Setting it true during debugging can help you see what values are being used, but you risk exposing secrets in your logs.

The allow_unsupported_distribution_setup is a safety valve. Kubespray has a list of tested and supported operating systems. If you try to deploy on something not on that list, it fails with an error. Setting this to true bypasses that check. Use this only if you know what you are doing and accept that things might break in unexpected ways.

etcd.yml — etcd Configuration

The etcd configuration is surprisingly minimal in the sample file:

grep "^[^#]" inventory/mycluster/group_vars/all/etcd.yml

---
etcd_data_dir: /var/lib/etcd
etcd_deployment_type: host

The etcd_data_dir is where etcd stores its database. This directory contains the entire state of your Kubernetes cluster. Losing this directory means losing your cluster. Make sure this path is on reliable storage and included in your backup strategy.

The etcd_deployment_type setting is more interesting. It has two possible values: host and kubeadm.

When set to host, Kubespray installs etcd as a systemd service running directly on the host. This is independent of Kubernetes itself. The etcd binary runs under the etcd user, managed by systemd, with its own certificates in /etc/ssl/etcd/ssl. This approach keeps etcd completely separate from Kubernetes, which some operators prefer because etcd can be managed, backed up, and recovered independently.

When set to kubeadm, etcd runs as a static pod managed by kubelet. This is what you get with a standard kubeadm init. The etcd container runs in the kube-system namespace, and kubeadm handles certificate generation and renewal. This approach is simpler but couples etcd lifecycle to Kubernetes.

Kubespray defaults to host because it provides more operational flexibility, especially for etcd upgrades and disaster recovery scenarios.

containerd.yml — Container Runtime Settings

Opening the containerd configuration file reveals mostly comments:

cat inventory/mycluster/group_vars/all/containerd.yml

---
# Please see roles/container-engine/containerd/defaults/main.yml 
# for more configuration options

# containerd_storage_dir: "/var/lib/containerd"
# containerd_state_dir: "/run/containerd"
# containerd_oom_score: 0

# containerd_default_runtime: "runc"
# containerd_snapshotter: "native"
...

The file is almost entirely commented out because the defaults work for most deployments. This is a pattern you will see throughout Kubespray: the sample files show you what options exist without forcing you to understand them all upfront.

If you need to change containerd’s storage location, perhaps because /var/lib is on a small root partition and you want container images on a larger disk, uncomment containerd_storage_dir and set your preferred path.

The comment at the top pointing to roles/container-engine/containerd/defaults/main.yml is valuable. When you need to customize something not shown in the sample file, that is where you look to find the variable name and its default value.

Cloud Provider Files

The group_vars/all directory contains configuration files for various cloud providers: aws.yml, azure.yml, gcp.yml, openstack.yml, vsphere.yml, and several others.

These files configure cloud-specific integrations like:

• Cloud controller manager settings
• CSI driver configurations for persistent volumes
• Load balancer integration
• Node metadata services

For example, if you deploy on AWS and want to use EBS volumes for persistent storage, you would configure the AWS CSI driver settings in aws.yml. If you deploy on bare metal, these files sit unused.

Most of the content in these files is commented out. The comments serve as documentation showing what options are available. When you need a specific cloud integration, you uncomment and configure the relevant settings.

docker.yml — Legacy Docker Support

The docker.yml file exists for deployments that still use Docker as the container runtime. Since Kubernetes 1.24 removed the built-in dockershim, using Docker now requires cri-dockerd as a shim layer.

docker_container_storage_setup: false
docker_dns_servers_strict: false
docker_daemon_graph: "/var/lib/docker"
docker_iptables_enabled: "false"
docker_log_opts: "--log-opt max-size=50m --log-opt max-file=5"
docker_bin_dir: "/usr/bin"
docker_rpm_keepcache: 1

Unless you have a specific reason to use Docker, stick with containerd. It is the default and the direction the Kubernetes ecosystem has moved.

offline.yml — Air-Gap Deployment

The offline.yml file contains settings for deploying Kubernetes in environments without internet access. Air-gap deployments require pre-downloading all container images and binaries, then serving them from internal repositories.

This file is where you configure your internal registry mirrors and binary download locations. In a connected environment, you can ignore this file entirely.

The group_vars/k8s_cluster Directory

This directory contains the real meat of your cluster configuration. The settings here define what kind of Kubernetes cluster you are building.

k8s-cluster.yml — The Heart of Your Cluster Configuration

This is the most important configuration file in Kubespray. It controls fundamental cluster characteristics that are difficult or impossible to change after deployment.

Running grep to see active settings:

grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

The output is substantial:

---
kube_config_dir: /etc/kubernetes
kube_script_dir: "/kubernetes-scripts"
kube_manifest_dir: "/manifests"
kube_cert_dir: "/ssl"
kube_token_dir: "/tokens"
kube_api_anonymous_auth: true
local_release_dir: "/tmp/releases"
retry_stagger: 5
kube_owner: kube
kube_cert_group: kube-cert
kube_log_level: 2
credentials_dir: "/credentials"
kube_network_plugin: calico
kube_network_plugin_multus: false
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
kube_network_node_prefix: 24
kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
kube_network_node_prefix_ipv6: 120
kube_apiserver_ip: ""
kube_apiserver_port: 6443
kube_proxy_mode: ipvs
kube_proxy_strict_arp: false
kube_encrypt_secret_data: false
cluster_name: cluster.local
ndots: 2
dns_mode: coredns
enable_nodelocaldns: true
enable_nodelocaldns_secondary: false
nodelocaldns_ip: 169.254.25.10
resolvconf_mode: host_resolvconf
deploy_netchecker: false
dns_domain: ""
container_manager: containerd
kata_containers_enabled: false
k8s_image_pull_policy: IfNotPresent
kubernetes_audit: false
volume_cross_zone_attachment: false
persistent_volumes_enabled: false
event_ttl_duration: "1h0m0s"
auto_renew_certificates: false
kubeadm_patches_dir: "/patches"
kubeadm_patches: []
remove_anonymous_access: false

Directory Settings

The first group of settings defines where Kubernetes stores its files:

kube_config_dir: /etc/kubernetes
kube_script_dir: "/kubernetes-scripts"
kube_manifest_dir: "/manifests"
kube_cert_dir: "/ssl"
kube_token_dir: "/tokens"

These paths are relative to kube_config_dir, so the actual manifest directory ends up at /etc/kubernetes/manifests. This is where static pod manifests for control plane components live.

You rarely need to change these unless your organization has specific filesystem layout requirements.

Network Settings

The network configuration is critical and mostly immutable after cluster creation:

kube_network_plugin: calico
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
kube_network_node_prefix: 24

The kube_network_plugin setting chooses your CNI provider. Calico is the default and provides excellent functionality including network policies. Other options include flannel for simplicity, cilium for eBPF-based networking, and several others.

The kube_service_addresses CIDR defines the IP range for Kubernetes Services. When you create a Service of type ClusterIP, it gets an IP from this range. The default 10.233.0.0/18 provides about 16,000 service IPs.

The kube_pods_subnet CIDR is where pod IPs come from. With 10.233.64.0/18, you have roughly 16,000 pod IPs. The kube_network_node_prefix of 24 means each node gets a /24 subnet (256 IPs) from this range for its pods.

These CIDR ranges must not overlap with your existing network infrastructure. If your corporate network uses 10.x.x.x addressing, you need to carefully plan these ranges to avoid conflicts. Changing these after deployment requires recreating the cluster.

Proxy Mode

The kube_proxy_mode setting determines how Kubernetes implements service networking:

kube_proxy_mode: ipvs

IPVS (IP Virtual Server) is the default and provides better performance than the older iptables mode, especially in clusters with many services. IPVS uses hash tables for service lookup, giving O(1) performance regardless of how many services exist. The iptables mode uses linear rule chains, so performance degrades as service count grows.

For most deployments, stick with ipvs. The iptables mode still works and might be preferred in very small clusters or when debugging networking issues since iptables rules are more familiar to many operators.

DNS Configuration

dns_mode: coredns
enable_nodelocaldns: true
nodelocaldns_ip: 169.254.25.10
cluster_name: cluster.local

CoreDNS is the standard Kubernetes DNS server and the only real option these days. The dns_mode setting exists for historical reasons when kube-dns was an alternative.

The enable_nodelocaldns setting deploys NodeLocal DNSCache, which runs a DNS caching agent on every node. Pods make DNS queries to the nodelocaldns_ip address (169.254.25.10, a link-local address), and the local agent either answers from cache or forwards to CoreDNS. This significantly reduces DNS latency and load on CoreDNS pods.

The cluster_name becomes your cluster’s DNS domain. Services are accessible at servicename.namespace.svc.cluster.local.

Container Runtime

container_manager: containerd

This chooses between containerd and CRI-O. Containerd is the dominant choice in the Kubernetes ecosystem now. It is what Docker uses internally, it is the default for most managed Kubernetes services, and it has excellent tooling support.

Certificate Management

auto_renew_certificates: false

Kubernetes component certificates expire after one year by default. When auto_renew_certificates is true, Kubespray configures a systemd timer to automatically renew certificates monthly using kubeadm certs renew.

The sample defaults to false, which means you need to manually renew certificates or implement your own automation. For any cluster running longer than a few months, enabling this is strongly recommended. Expired certificates cause control plane components to stop communicating, effectively bringing down your cluster.

addons.yml — Cluster Add-ons

The addons file controls which additional components get deployed:

grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/addons.yml

---
helm_enabled: false
registry_enabled: false
metrics_server_enabled: false
local_path_provisioner_enabled: false
local_volume_provisioner_enabled: false
gateway_api_enabled: false
ingress_nginx_enabled: false
ingress_publish_status_address: ""
ingress_alb_enabled: false
cert_manager_enabled: false
metallb_enabled: false
metallb_speaker_enabled: ""
metallb_namespace: "metallb-system"
argocd_enabled: false
kube_vip_enabled: false
node_feature_discovery_enabled: false

Each boolean controls whether that component gets installed. The defaults are conservative, deploying a minimal cluster. You enable what you need.

For most production clusters, you probably want:

helm_enabled: true
metrics_server_enabled: true

Helm is the de facto package manager for Kubernetes applications. Metrics Server is required for kubectl top commands and Horizontal Pod Autoscaler functionality.

For bare metal clusters without a cloud load balancer, MetalLB provides LoadBalancer service support:

metallb_enabled: true

For clusters that need ingress routing:

ingress_nginx_enabled: true

The file contains many commented-out sections with detailed configuration options for each addon. When you enable an addon, scroll through those comments to see what customization options exist.

kube_control_plane.yml — Control Plane Resource Reservation

This file is entirely commented out in the sample:

cat inventory/mycluster/group_vars/k8s_cluster/kube_control_plane.yml

# Reservation for control plane kubernetes components
# kube_memory_reserved: 512Mi
# kube_cpu_reserved: 200m
# kube_ephemeral_storage_reserved: 2Gi
# kube_pid_reserved: "1000"

# Reservation for control plane host system
# system_memory_reserved: 256Mi
# system_cpu_reserved: 250m
# system_ephemeral_storage_reserved: 2Gi
# system_pid_reserved: "1000"

These settings reserve resources on control plane nodes for Kubernetes components and the host operating system. Without reservations, pods could consume all available resources, starving critical system processes.

In production, especially on nodes that run both control plane components and workload pods, configuring these reservations prevents resource exhaustion scenarios. The commented values are reasonable starting points.

Network Plugin Configuration Files

The k8s_cluster directory contains configuration files for each supported CNI plugin. Only the file matching your kube_network_plugin choice gets used.

For Calico (k8s-net-calico.yml):

calico_ipip_mode: Always
calico_vxlan_mode: Never
calico_network_backend: bird

These control Calico’s encapsulation mode. IPIP mode wraps pod traffic in IP-in-IP packets, which works across most networks. VXLAN is an alternative encapsulation. The bird backend uses BGP for routing, which can peer with your physical network infrastructure for advanced deployments.

For Flannel (k8s-net-flannel.yml):

flannel_backend_type: vxlan

Flannel is simpler than Calico. It uses VXLAN overlay networking by default. The main thing you might configure is flannel_interface if you have multiple network interfaces and need Flannel to use a specific one.

For Cilium (k8s-net-cilium.yml):

Cilium has extensive configuration options for its eBPF-based networking. The defaults work for getting started, but Cilium’s advanced features like transparent encryption, Hubble observability, and service mesh capabilities all have configuration options here.

Checking Supported Kubernetes Versions

The group_vars files define what you want, but Kubespray can only install versions it knows about. The roles/kubespray_defaults/vars/main/checksums.yml file contains SHA256 checksums for all supported binaries.

To see what Kubernetes versions are available:

cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40

If a version does not have checksums in this file, Kubespray cannot install it. This is a security feature ensuring binary integrity. When Kubernetes releases a new version, Kubespray maintainers add the checksums in a subsequent release.

This is why you cannot simply set kube_version to any arbitrary version. You need a Kubespray release that includes that version’s checksums.

Practical Workflow

When setting up a new cluster, the typical workflow is:

First, copy the sample inventory:

cp -rfp inventory/sample inventory/mycluster

Second, edit inventory.ini with your hosts and groups.

Third, review k8s-cluster.yml and change the settings that matter for your environment:

• Choose your CNI plugin
• Verify the CIDR ranges do not conflict with your network
• Enable certificate auto-renewal
• Adjust any other settings based on requirements

Fourth, review addons.yml and enable the add-ons you need.

Fifth, if using a specific cloud provider, configure the corresponding file in group_vars/all.

Variables you do not change keep their defaults from roles/*/defaults/main.yml. You do not need to understand every variable to deploy a working cluster. Start with the defaults, deploy, and then iterate as you learn what needs customization.

The group_vars files in your inventory are yours to modify freely. Kubespray updates will not overwrite them because they live in your custom inventory directory, not in the sample directory or the roles themselves.

Chapter 7: cluster.yml Playbook Flow Analysis

When you run ansible-playbook cluster.yml, you are kicking off a carefully orchestrated sequence of operations that transforms a bunch of Linux machines into a functioning Kubernetes cluster.

Understanding this flow is not optional if you want to operate Kubespray in production. Let me walk you through the entire cluster.yml playbook, explaining what happens at each stage and why it is structured the way it is.

The cluster.yml file sits at the root of the Kubespray repository. It is the main entry point for cluster deployment. But here is the thing that confused me when I first looked at it: the file itself is surprisingly short. Most of the actual work happens in sub-playbooks and roles that get imported.

The overall flow looks like this:

Common tasks (boilerplate.yml)
↓
Fact gathering (internal_facts.yml)
↓
etcd installation preparation (preinstall, container-engine, download)
↓
etcd installation (install_etcd.yml)
↓
Kubernetes node installation (kubernetes/node)
↓
Control plane installation (kubernetes/control-plane)
↓
kubeadm execution and CNI installation (kubernetes/kubeadm, network_plugin)
↓
Calico Route Reflector (optional)
↓
Windows node patching (optional)
↓
Kubernetes apps installation (kubernetes-apps)
↓
resolv.conf finalization

Each of these stages corresponds to one or more plays in the playbook. Let me show you the actual structure.

The Playbook Structure

Here is the cluster.yml file in its entirety. I am including the whole thing because it is worth reading through:

---
- name: Common tasks for every playbooks
  import_playbook: boilerplate.yml

- name: Gather facts
  import_playbook: internal_facts.yml

- name: Prepare for etcd install
  hosts: "{{ groups['etcd'] | default([]) | union(groups['k8s_cluster'] | default([])) }}"
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/preinstall, tags: preinstall }
    - { role: container-engine, tags: container-engine, when: deploy_container_engine }
    - { role: download, tags: download, when: "not skip_downloads" }

- name: Install etcd
  vars:
    etcd_cluster_setup: true
    etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
  import_playbook: install_etcd.yml

- name: Install Kubernetes nodes
  hosts: k8s_cluster
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/node, tags: node }

- name: Install the control plane
  hosts: kube_control_plane
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/control-plane, tags: control-plane }

- name: Invoke kubeadm and install a CNI
  hosts: k8s_cluster
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/kubeadm, tags: kubeadm }
    - { role: kubernetes/node-label, tags: node-label }
    - { role: kubernetes/node-taint, tags: node-taint, when: node_taints is defined }
    - { role: network_plugin, tags: network }

- name: Install Calico Route Reflector
  hosts: calico_rr
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: network_plugin/calico/rr, tags: ['network', 'calico_rr'] }

- name: Patch Kubernetes for Windows
  hosts: kube_control_plane[0]
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: win_nodes/kubernetes_patch, tags: k8s-windows }
- name: Install Kubernetes apps
  hosts: kube_control_plane
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes-apps, tags: apps }
- name: Apply resolv.conf changes now that cluster DNS is up
  hosts: k8s_cluster
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf, dns_late: true }

Play 1 and 2: Boilerplate and Fact Gathering

The first two plays handle common setup tasks:

- name: Common tasks for every playbooks
  import_playbook: boilerplate.yml

- name: Gather facts
  import_playbook: internal_facts.yml

The boilerplate.yml playbook does things like validating your Ansible version, checking that your inventory is properly configured, and setting up bastion host SSH config if you are using one. It is the “sanity check before we do anything destructive” phase.

The internal_facts.yml playbook runs Ansible’s fact gathering across all nodes. This collects information about each machine: what operating system it runs, what IP addresses it has, how much memory, what network interfaces exist, and so on. This information becomes available as variables that later plays can use.

Notice that these use import_playbook rather than defining roles directly. This is a common pattern in Kubespray. The import_playbook directive pulls in another playbook file and executes it as if it were part of the current file.

Play 3: Preparing for etcd Installation

This play is where things start getting interesting:

- name: Prepare for etcd install
  hosts: "{{ groups['etcd'] | default([]) | union(groups['k8s_cluster'] | default([])) }}"
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/preinstall, tags: preinstall }
    - { role: container-engine, tags: container-engine, when: deploy_container_engine }
    - { role: download, tags: download, when: "not skip_downloads" }

Let me explain each line.

The hosts directive uses Jinja2 templating to target both the etcd group and the k8s_cluster group. The union filter combines these two groups, and the default([]) ensures the playbook does not fail if either group is undefined. This means the play runs on all nodes that will either run etcd or be part of the Kubernetes cluster.

The gather_facts: false setting might seem strange. Why would we skip fact gathering? The answer is that we already gathered facts in the previous play (internal_facts.yml), and Kubespray caches these facts. Look at the ansible.cfg settings:

gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp
fact_caching_timeout = 86400

The gathering = smart setting tells Ansible to use cached facts if they exist and are still valid. Since we gathered facts just moments ago and they are cached in /tmp as JSON files, there is no need to gather them again. This saves time, especially when you have many nodes.

The any_errors_fatal setting deserves attention. When set to true, if any single host fails a task, the entire playbook stops immediately. This is critical for cluster deployment because a half-configured cluster is worse than no cluster at all. If etcd fails to install on one node, you do not want the playbook to continue and try to bootstrap Kubernetes against a broken etcd cluster.

The environment directive sets environment variables for all tasks in this play. The proxy_disable_env variable is defined in roles/kubespray_defaults/defaults/main/main.yml:

proxy_disable_env:
  http_proxy: ""
  HTTP_PROXY: ""
  https_proxy: ""
  HTTPS_PROXY: ""
  no_proxy: ""
  NO_PROXY: ""

This clears all proxy-related environment variables. Cluster components communicate directly with each other, and you do not want that traffic going through an HTTP proxy.

Understanding Role Import Syntax

Look at how roles are imported:

roles:
  - { role: kubespray_defaults }
  - { role: kubernetes/preinstall, tags: preinstall }
  - { role: container-engine, tags: container-engine, when: deploy_container_engine }
  - { role: download, tags: download, when: "not skip_downloads" }

This is YAML shorthand syntax. It is equivalent to writing:

roles:
  - role: kubespray_defaults
  
  - role: kubernetes/preinstall
    tags: preinstall
  
  - role: container-engine
    tags: container-engine
    when: deploy_container_engine
  
  - role: download
    tags: download
    when: "not skip_downloads"

The curly brace syntax is more compact but can be harder to read. Kubespray uses it throughout.

Each role can have:

• A name (the path under the roles/ directory)
• Tags for selective execution
• Conditions (when clauses) that determine if the role runs

The kubespray_defaults role always runs first and has no tags or conditions. This is intentional. It loads all the default variable values that other roles depend on. If you skip it, nothing else will work.

Why kubespray_defaults Runs First in Every Play

You will notice that every single play in cluster.yml starts with:

roles:
  - { role: kubespray_defaults }

This pattern is not accidental. The kubespray_defaults role lives at roles/kubespray_defaults/ and contains:

roles/kubespray_defaults/
├── defaults/main/
│   ├── main.yml        # 800+ lines of default values
│   └── download.yml    # 1100+ lines of download-related defaults
└── vars/main/
    ├── main.yml
    └── checksums.yml   # Binary checksums for integrity verification

This role does not run any tasks. It just loads variables. Every other role in Kubespray depends on these variables being available. Things like kube_version, container_manager, kube_network_plugin, and hundreds of other settings are defined here.

When Ansible runs a play, it loads role defaults at the beginning. By putting kubespray_defaults first, Kubespray ensures that all default values are loaded before any other role tries to use them.

Tag-Based Selective Execution

Tags are powerful. They let you run only specific parts of the playbook. Look at the tags assigned to each role:

- { role: kubernetes/preinstall, tags: preinstall }
- { role: container-engine, tags: container-engine, when: deploy_container_engine }
- { role: download, tags: download, when: "not skip_downloads" }

If you want to reinstall just the network plugin without rerunning the entire playbook, you can do:

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml --tags network

This runs only the roles and tasks tagged with “network”. Everything else gets skipped.

Here is a quick reference of the major tags in cluster.yml:

preinstall       → kubernetes/preinstall role
container-engine → container-engine role
download         → download role
node             → kubernetes/node role
control-plane    → kubernetes/control-plane role
kubeadm          → kubernetes/kubeadm role
network          → network_plugin role
apps             → kubernetes-apps role

You can also combine tags:

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml --tags preinstall,container-engine

This runs only the preinstall and container-engine stages.

Play 4: Installing etcd

- name: Install etcd
  vars:
    etcd_cluster_setup: true
    etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
  import_playbook: install_etcd.yml

This play imports the install_etcd.yml playbook and passes two variables using the vars directive.

Here is something subtle but important. The etcd_cluster_setup variable is set to true here. But wait, is this variable not already defined somewhere? Yes, it is. In roles/etcd_defaults/defaults/main.yml, you will find:

etcd_cluster_setup: true

So why set it again in the play?

The answer becomes clear when you look at scale.yml:

- name: Install etcd
  vars:
    etcd_cluster_setup: false
    etcd_events_cluster_setup: false
  import_playbook: install_etcd.yml

In scale.yml, when adding new nodes to an existing cluster, etcd_cluster_setup is set to false. The etcd cluster already exists; we do not want to reinitialize it. By setting this variable at the play level, Kubespray controls how install_etcd.yml behaves differently in cluster.yml versus scale.yml.

The etcd_events_cluster_enabled variable controls whether to set up a separate etcd cluster for Kubernetes events. This is an advanced feature for large clusters where event traffic can overwhelm the main etcd cluster. Most deployments leave this disabled.

Play 5: Installing Kubernetes Nodes

- name: Install Kubernetes nodes
  hosts: k8s_cluster
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/node, tags: node }

This play targets the k8s_cluster group, which includes both control plane nodes and worker nodes. The kubernetes/node role installs kubelet, kubectl, and kubeadm on all nodes that will be part of the cluster.

Notice the pattern repeating: gather_facts: false (using cached facts), any_errors_fatal (fail fast), environment (no proxy), and kubespray_defaults first.

Play 6: Installing the Control Plane

- name: Install the control plane
  hosts: kube_control_plane
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/control-plane, tags: control-plane }

This play targets only kube_control_plane nodes. The kubernetes/control-plane role prepares these nodes to run the Kubernetes control plane components: kube-apiserver, kube-controller-manager, and kube-scheduler.

Play 7: Running kubeadm and Installing CNI

- name: Invoke kubeadm and install a CNI
  hosts: k8s_cluster
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/kubeadm, tags: kubeadm }
    - { role: kubernetes/node-label, tags: node-label }
    - { role: kubernetes/node-taint, tags: node-taint, when: node_taints is defined }
    - { role: network_plugin, tags: network }

This is where the cluster actually comes together. The kubernetes/kubeadm role runs kubeadm init on the first control plane node, then kubeadm join on all other nodes.

After the cluster is bootstrapped, the node-label role applies any custom labels you have defined. The node-taint role applies taints, but only if you have defined node_taints in your variables.

Finally, the network_plugin role installs your chosen CNI plugin. If you set kube_network_plugin: calico in your variables, this role installs Calico. If you set kube_network_plugin: flannel, it installs Flannel. The role reads your configuration and acts accordingly.

Play 8: Calico Route Reflector

- name: Install Calico Route Reflector
  hosts: calico_rr
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: network_plugin/calico/rr, tags: ['network', 'calico_rr'] }

This play only runs if you have a calico_rr group defined in your inventory. Calico Route Reflector is an advanced feature for large clusters using BGP routing. Most deployments do not use this, and if the calico_rr group is empty or undefined, this play does nothing.

Notice the tags syntax here: tags: [‘network’, ‘calico_rr’]. This assigns two tags to the role, so you can target it with either — tags network or — tags calico_rr.

Play 9: Windows Node Support

- name: Patch Kubernetes for Windows
  hosts: kube_control_plane[0]
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: win_nodes/kubernetes_patch, tags: k8s-windows }

This play applies patches needed for Windows node support. It targets only the first control plane node (kube_control_plane[0]) because it just needs to apply some manifests to the cluster.

If you are not running Windows workers, this play effectively does nothing. The win_nodes/kubernetes_patch role checks whether Windows support is enabled and skips itself if not.

Play 10: Installing Kubernetes Apps

- name: Install Kubernetes apps
  hosts: kube_control_plane
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes-apps, tags: apps }

This play installs cluster addons: CoreDNS, metrics-server, Helm, Ingress controllers, and whatever else you have enabled in your addons.yml file. It runs on control plane nodes because it needs kubectl access to apply manifests.

Play 11: Finalizing DNS Configuration

- name: Apply resolv.conf changes now that cluster DNS is up
  hosts: k8s_cluster
  gather_facts: false
  any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
  environment: "{{ proxy_disable_env }}"
  roles:
    - { role: kubespray_defaults }
    - { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf, dns_late: true }

This final play updates /etc/resolv.conf on all cluster nodes now that CoreDNS is running. The kubernetes/preinstall role is called again, but this time with dns_late: true. This variable changes the role’s behavior, making it update DNS configuration rather than doing general preinstall tasks.

The when clause ensures this only runs if you are using coredns (dns_mode != ‘none’) and managing the host’s resolv.conf (resolvconf_mode == ‘host_resolvconf’).

Mapping to Kubernetes The Hard Way

If you have gone through Kubernetes The Hard Way, you might wonder how Kubespray’s stages correspond to those manual steps. Here is the mapping:

Kubespray Role/Play              | The Hard Way Equivalent
---------------------------------|----------------------------------
kubernetes/preinstall            | OS configuration (sysctl, modules)
container-engine/containerd      | Containerd installation
download                         | Downloading binaries
etcd                             | Bootstrapping etcd cluster
kubernetes/node                  | Worker node setup (kubelet)
kubernetes/control-plane         | Control plane bootstrapping
kubernetes/kubeadm               | kubeadm init and join
network_plugin                   | Pod network configuration
kubernetes-apps                  | DNS and addon installation

The certificate generation that took an entire chapter in The Hard Way? Kubespray delegates that to kubeadm, which handles it automatically during kubeadm init.

The kubeconfig file generation? Also handled by kubeadm.

The data encryption configuration? Kubespray has options for that in the variables, and kubeadm configures it.

This is the power of Kubespray. It takes all those manual steps and encodes them into Ansible roles that execute reliably and repeatably.

Conditional Variables That Control Behavior

Several variables act as switches that change what cluster.yml does:

The deploy_container_engine variable defaults to true. If you set it to false, Kubespray skips container runtime installation entirely, assuming you have already installed containerd or another runtime yourself.

- { role: container-engine, tags: container-engine, when: deploy_container_engine }

The skip_downloads variable does what it says. Set it to true if you have pre-downloaded all binaries and images (useful for air-gap deployments):

- { role: download, tags: download, when: "not skip_downloads" }

The etcd_events_cluster_enabled variable controls whether to create a separate etcd cluster for Kubernetes events:

etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"

The dns_mode variable determines which cluster DNS to install. The default is coredns. Setting it to none skips DNS installation:

when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'"

The resolvconf_mode variable controls how Kubespray manages DNS resolution on nodes. The default host_resolvconf means Kubespray will modify /etc/resolv.conf.

Running the Playbook

Now that you understand the structure, running the playbook is straightforward:

cd ~/kubespray

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b -v

The -i flag specifies your inventory file. The -b flag enables become (sudo). The -v flag enables verbose output.

For a full deployment, expect this to take 15–30 minutes depending on your network speed and hardware. The download stage often takes the longest as it pulls container images and binaries.

If something fails, you can usually fix the issue and rerun the same command. Ansible’s idempotency means tasks that already completed successfully will show as “ok” and run quickly, while the failed tasks retry.

To run only specific stages:

# Just reinstall the network plugin
ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b -v --tags network

# Reinstall apps only
ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b -v --tags apps

# Run everything except downloads (useful after fixing a download issue)
ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b -v --skip-tags download

The — skip-tags option is the inverse of — tags. It runs everything except the specified tags.

Watching the Execution

When you run cluster.yml, you will see output like this:

PLAY [Common tasks for every playbooks] ****************************************

TASK [Check that python netaddr is installed] **********************************
ok: [localhost]
PLAY [Gather facts] ************************************************************
TASK [Gather minimal facts] ****************************************************
ok: [k8s-ctr]
PLAY [Prepare for etcd install] ************************************************
TASK [kubespray_defaults : Load kubespray defaults] ****************************
ok: [k8s-ctr]
TASK [kubernetes/preinstall : Set facts] ***************************************
ok: [k8s-ctr]
...

Each PLAY corresponds to a play in the playbook. Each TASK corresponds to a task within a role. The output shows which host the task runs on and whether it changed anything.

At the end, you get a PLAY RECAP:

PLAY RECAP *********************************************************************
k8s-ctr    : ok=523  changed=147  unreachable=0  failed=0  skipped=892  rescued=0  ignored=2

This tells you:

• ok: Tasks that completed without making changes
• changed: Tasks that made changes to the system
• unreachable: Hosts that could not be contacted
• failed: Tasks that failed
• skipped: Tasks that were skipped due to conditions
• rescued: Tasks that failed but were rescued by a rescue block
• ignored: Tasks that failed but had ignore_errors: true

A successful run has failed=0 and unreachable=0.

The profile_tasks callback enabled in ansible.cfg also shows timing information:

Tuesday 28 January 2026  15:23:45 +0900 (0:00:02.456)

===============================================================================
download : Download_container | Download image if required ----------- 48.84s
download : Download_container | Download image if required ----------- 33.23s
kubernetes/kubeadm : Join to cluster if needed ----------------------- 15.97s
container-engine : Containerd | Unpack containerd archive ------------- 8.34s
...

This helps identify which tasks take the longest, useful for optimization or debugging slow deployments.

Chapter 8: Lab Environment Setup

Before we can deploy a Kubernetes cluster with Kubespray, we need machines to deploy it on. In this chapter, we will set up a lab environment using Vagrant and VirtualBox. By the end of this chapter, you will have a Rocky Linux virtual machine ready to become a single-node Kubernetes cluster.

Prerequisites

You will need the following software installed on your host machine:

VirtualBox version 7.2.4 or later. Earlier versions have compatibility issues with newer Linux kernels, particularly Rocky Linux 10 which ships with kernel 6.x. If you are running an older version of VirtualBox, upgrade it before proceeding.

Vagrant version 2.4.x or later. Vagrant will orchestrate the virtual machine lifecycle and handle the initial provisioning.

Your host machine should have at least 8 GB of RAM because we will allocate 4 GB to the virtual machine. You also need approximately 20 GB of free disk space for the VM disk image and downloaded artifacts.

To verify your installed versions, run these commands:

VBoxManage --version
vagrant --version

If you see version numbers that meet the requirements, you are ready to proceed.

Windows Users: Disable Hyper-V First

If you are running Windows, there is a critical step you must complete before anything else. Windows has a feature called Hyper-V which is a Type-1 hypervisor built into the operating system. When Hyper-V is enabled, it takes exclusive control of the CPU’s hardware virtualization features (Intel VT-x or AMD-V). This prevents VirtualBox from using native virtualization, forcing it to fall back to a compatibility mode called NEM (Native Execution Mode).

The problem is that Rocky Linux 10 uses a modern 6.x kernel that does not play well with NEM mode. When VirtualBox tries to boot Rocky Linux under NEM, you will see a kernel panic within seconds of boot:

[    1.511197] ---[ end trace 0000000000000000 ]---
[    1.513571] RIP: 0010:wait_for_xmitr+0x61/0xc0
[    1.559573] Kernel panic - not syncing: Fatal exception
[    1.563919] ---[ end Kernel panic - not syncing: Fatal exception ]---

I spent an embarrassing amount of time staring at this error before discovering the root cause. The VirtualBox log file contained a telltale message: “HM: HMR3Init: Attempting fall back to NEM: VT-x is not available” which indicated that Hyper-V was claiming the virtualization hardware.

You might wonder why Hyper-V was enabled in the first place. Several Windows features depend on it: WSL2 (Windows Subsystem for Linux 2), Docker Desktop when using the WSL2 backend, and Windows Sandbox. If you have used any of these features, Hyper-V is probably active on your system.

To disable Hyper-V and its related components, open PowerShell as Administrator and run these commands:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All -NoRestart
Disable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart
Disable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform -NoRestart

Then disable the hypervisor launch at boot:

bcdedit /set hypervisorlaunchtype off

Finally, restart your computer:

Restart-Computer

After the reboot, verify that Hyper-V is truly disabled. Run this command which should produce no output if Hyper-V is off:

Get-WindowsOptionalFeature -Online | Where-Object {$_.FeatureName -like "*Hyper-V*" -and $_.State -eq "Enabled"}

Also verify the boot configuration:

bcdedit /enum | Select-String hypervisorlaunchtype

The output should show “hypervisorlaunchtype Off”. If it shows “Auto” or anything else, the disable did not take effect and you need to troubleshoot further.

Be aware that disabling Hyper-V means WSL2 and Docker Desktop (with WSL2 backend) will stop working. You can re-enable everything after completing this lab, which I will explain at the end of the chapter.

Creating the Vagrantfile

Create a new directory for this lab and create a file named Vagrantfile with the following content:

BOX_IMAGE = "bento/rockylinux-10.0"
BOX_VERSION = "202510.26.0"

Vagrant.configure("2") do |config|
  config.vm.define "k8s-ctr" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/Kubespray-Lab"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "k8s-ctr"
      vb.cpus = 4
      vb.memory = 4096
      vb.linked_clone = true
    end
    subconfig.vm.host_name = "k8s-ctr"
    subconfig.vm.network "private_network", ip: "192.168.10.10"
    subconfig.vm.network "forwarded_port", guest: 22, host: "60100", auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "init_cfg.sh"
  end
end

Let me explain what each section does.

The BOX_IMAGE and BOX_VERSION variables specify the Vagrant box to use. We are using the Bento project’s Rocky Linux 10.0 image, which is a well-maintained community box. Pinning the version ensures reproducibility.

The vm.provider block configures VirtualBox-specific settings. The — groups option organizes the VM into a folder called “Kubespray-Lab” in the VirtualBox Manager GUI. The — nicpromisc2 option enables promiscuous mode on the second network adapter, which is necessary for some CNI plugins to function correctly. We allocate 4 CPU cores and 4096 MB of memory, which is sufficient for a single-node cluster. The linked_clone option saves disk space by creating a copy-on-write clone rather than a full copy of the base image.

The network configuration creates two network interfaces. VirtualBox always creates a NAT interface as the first adapter, which provides outbound internet connectivity. We add a private_network with a static IP of 192.168.10.10, which creates a Host-Only network that allows the host machine and VM to communicate directly. The forwarded_port directive maps port 60100 on the host to port 22 on the guest, providing an alternative way to SSH into the VM.

The synced_folder line disables the default Vagrant folder synchronization. We do not need it and disabling it avoids potential permission issues.

Finally, the provision directive tells Vagrant to run a shell script called init_cfg.sh after the VM boots for the first time.

The Initialization Script

Create a file named init_cfg.sh in the same directory as your Vagrantfile:

#!/usr/bin/env bash

echo ">>>> Initial Config Start <<<<"
echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul
echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
echo "[TASK 3] Disable and turn off SWAP & Delete swap partitions"
swapoff -a
sed -i '/swap/d' /etc/fstab
sfdisk --delete /dev/sda 2 >/dev/null 2>&1
partprobe /dev/sda >/dev/null 2>&1
echo "[TASK 4] Config kernel & module"
cat << EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay >/dev/null 2>&1
modprobe br_netfilter >/dev/null 2>&1
cat << EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system >/dev/null 2>&1
echo "[TASK 5] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
cat << EOF >> /etc/hosts
192.168.10.10 k8s-ctr
EOF
echo "[TASK 6] Delete default routing - Secondary NIC"
SECONDARY_NIC=$(ip -o -4 addr show | grep "192.168.10" | awk '{print $2}')
if [ -n "$SECONDARY_NIC" ]; then
  echo "Found secondary NIC: $SECONDARY_NIC, disabling default route..."
  nmcli connection modify "$SECONDARY_NIC" ipv4.never-default yes 2>/dev/null || true
  nmcli connection up "$SECONDARY_NIC" 2>/dev/null || true
else
  echo "No secondary NIC found, skipping..."
fi
echo "sudo su -" >> /home/vagrant/.bashrc
echo ">>>> Initial Config End <<<<"

This script performs essential pre-configuration that Kubernetes requires. Let me walk through each task in detail.

Task 1 sets the timezone and configures the system clock. The timedatectl set-local-rtc 0 command tells the system to use UTC for the hardware clock, which is the recommended setting for servers. Change Asia/Seoul to your preferred timezone. Accurate time is important for Kubernetes because certificate validation, log correlation, and distributed coordination all depend on synchronized clocks.

Task 2 disables the firewall and sets SELinux to permissive mode. In a production environment, you would configure proper firewall rules and SELinux policies. For a lab environment, disabling them eliminates potential networking issues that could complicate troubleshooting. The firewalld service is stopped and disabled. SELinux is set to permissive mode, which logs policy violations without enforcing them. The sed command makes this change persistent across reboots by modifying the SELinux configuration file.

Task 3 disables swap. Kubernetes requires swap to be disabled because the kubelet is not designed to handle swap memory. When swap is active, it can cause unpredictable latency spikes and interfere with resource management. The swapoff -a command immediately disables all swap. The sed command removes any swap entries from /etc/fstab so swap does not re-enable after a reboot. The sfdisk command attempts to delete the swap partition entirely, though this may fail depending on the disk layout.

Task 4 loads kernel modules and sets sysctl parameters that container networking requires. The overlay module enables OverlayFS, which containerd uses for efficient container image layer management. The br_netfilter module enables netfilter to process traffic traversing network bridges, which is necessary for iptables rules to work correctly with container networks. The sysctl parameters enable IP forwarding and bridge netfilter processing for both IPv4 and IPv6.

Task 5 adds a hosts file entry so the VM can resolve its own hostname to the correct IP address. This is important because Kubernetes components need to resolve node names to IP addresses.

Task 6 is where things get interesting and where I encountered problems on my first attempt. VirtualBox VMs have two network interfaces: the NAT interface (usually eth0 or enp0s3) and the Host-Only interface (usually eth1 or enp0s8). By default, both interfaces might try to set a default route, and the NAT interface typically wins. This causes problems later when Kubespray tries to determine the node’s IP address because it might pick up the NAT IP (10.0.2.15) instead of the Host-Only IP (192.168.10.10).

The original version of this script hardcoded the interface name as enp0s9, but interface names can vary depending on the VirtualBox version and the order of network adapter configuration. I modified the script to dynamically detect the interface by looking for the one that has an IP address in the 192.168.10.0/24 range:

SECONDARY_NIC=$(ip -o -4 addr show | grep "192.168.10" | awk '{print $2}')

The ip -o -4 addr show command outputs IPv4 address information in a single-line format. We grep for our known IP range and extract the interface name with awk. Once we have the interface name, we tell NetworkManager to never use this interface as the default route with the ipv4.never-default option. This ensures that outbound traffic to the internet goes through the NAT interface while still allowing direct communication over the Host-Only network.

The last line adds “sudo su -” to the vagrant user’s bashrc so you automatically become root when you SSH into the VM. This is purely a convenience for lab environments.

Launching the Virtual Machine

With both files in place, launch the VM:

vagrant up k8s-ctr

Vagrant will download the Rocky Linux box if it is not already cached, create the VM, and run the initialization script. The output will show each task being executed. When you see “Initial Config End” followed by the Vagrant completion message, the VM is ready.

Verify the VM is running:

vagrant status

You should see:

Current machine states:
k8s-ctr                   running (virtualbox)

SSH into the VM to verify everything is working:

vagrant ssh k8s-ctr

You should be logged in as the vagrant user and immediately switched to root due to the bashrc modification. Verify the network configuration:

ip addr show

You should see at least two interfaces with IP addresses. One will have 10.0.2.15 (the NAT interface) and another will have 192.168.10.10 (the Host-Only interface). Verify the hostname resolves correctly:

hostname
ping -c 1 k8s-ctr

Both commands should work without errors.

Windows Troubleshooting: VERR_ALREADY_EXISTS Error

If you are on Windows and encounter an error containing VERR_ALREADY_EXISTS when running vagrant up a second time or after a failed first attempt, the problem is leftover VM directories. The — groups setting causes VirtualBox to move the VM into a folder, and if that folder already exists from a previous attempt, VirtualBox throws an error.

To fix this, destroy the VM and clean up the directories:

vagrant destroy -f k8s-ctr

Then remove the leftover directories in PowerShell:

Remove-Item -Recurse -Force "C:\Users\$env:USERNAME\VirtualBox VMs\Kubespray-Lab" -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force "C:\Users\$env:USERNAME\VirtualBox VMs\k8s-ctr" -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force .vagrant -ErrorAction SilentlyContinue

Now try vagrant up again. If the problem persists, you can work around it by removing the — groups line from the Vagrantfile:

# vb.customize ["modifyvm", :id, "--groups", "/Kubespray-Lab"]

The — groups option is purely cosmetic. It organizes VMs into folders in the VirtualBox Manager UI but has no effect on VM operation.

Windows Troubleshooting: Unknown Connection Error

If the init_cfg.sh script fails with “Error: unknown connection ‘enp0s9’” or a similar message, it means the original script’s hardcoded interface name does not match your system. Make sure you are using the modified version of the script that dynamically detects the interface name using the IP address pattern:

SECONDARY_NIC=$(ip -o -4 addr show | grep "192.168.10" | awk '{print $2}')

This approach works regardless of what VirtualBox names the interfaces on your particular system.

Setting Up SSH Keys for Ansible

Kubespray uses Ansible, which connects to target machines via SSH. For password-less authentication, we need to set up SSH key-based access from the Ansible control node to the target nodes. In this single-node lab, the VM will act as both the Ansible control node and the target node, so it needs to be able to SSH to itself.

First, generate an SSH key pair if you do not already have one. From your host machine or from within the VM (depending on where you plan to run Ansible from):

ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519

The -N “” option creates a key without a passphrase. In production, you would use a passphrase and an SSH agent, but for a lab environment, a passphrase-less key simplifies automation.

Copy the public key to the VM. If you are running Ansible from your host machine, use the forwarded port:

ssh-copy-id -o StrictHostKeyChecking=no -p 60100 vagrant@127.0.0.1

Enter “vagrant” when prompted for the password.

If you plan to run Ansible from within the VM itself (which is what we will do in the deployment chapter), SSH into the VM and set up local SSH access:

vagrant ssh k8s-ctr

Then generate a key and copy it to localhost:

ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.10

Enter the root password when prompted. On this Vagrant box, the root password is typically “vagrant”.

Verify password-less SSH works:

ssh root@192.168.10.10 "hostname"

If this prints “k8s-ctr” without asking for a password, SSH key authentication is working correctly.

Re-enabling Hyper-V After the Lab

If you disabled Hyper-V on Windows and want to restore WSL2 and Docker Desktop functionality after completing this lab, run these commands in an Administrator PowerShell:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All -NoRestart
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart
Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform -NoRestart
bcdedit /set hypervisorlaunchtype auto

Then restart your computer:

Restart-Computer

Your WSL2 distributions and their data will still be intact. Disabling and re-enabling Hyper-V does not affect the stored data.

Verifying the Environment

Before moving on, let us verify that the VM is correctly configured for Kubernetes:

Check that swap is disabled:

swapon --show

This should produce no output. If it shows swap partitions, the disable did not work and you need to troubleshoot.

Check that the kernel modules are loaded:

lsmod | grep -E "overlay|br_netfilter"

You should see both overlay and br_netfilter in the output.

Check the sysctl parameters:

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

All three should show a value of 1.

Check the SELinux status:

getenforce

This should show “Permissive”.

Check that the firewall is disabled:

systemctl status firewalld

This should show “inactive (dead)”.

If all these checks pass, your lab environment is ready for Kubespray. In the next chapter, we will configure the Kubespray inventory and customize the cluster variables before deploying Kubernetes.

Chapter 9: Inventory Configuration and Variable Modification

With your lab environment up and running, you now have a Rocky Linux VM sitting at 192.168.10.10, waiting to become a Kubernetes node. But Kubespray does not know anything about your infrastructure yet. You need to tell it what machines exist, what roles they should play, and how the cluster should be configured. This is where inventory configuration comes in.

Kubespray ships with a sample inventory that serves as both a template and a learning resource. The sample contains sensible defaults and commented examples that show you what options are available. Your first step is to copy this sample and make it your own.

cp -rfp /root/kubespray/inventory/sample /root/kubespray/inventory/mycluster

The flags here matter. The -r flag copies recursively since the inventory is a directory containing subdirectories. The -f flag forces the copy without prompting. The -p flag preserves the original file attributes including permissions, ownership, and timestamps. This last flag is often overlooked but can save you from mysterious permission issues later.

After running this command, you have a complete inventory structure at /root/kubespray/inventory/mycluster that mirrors the sample. Now you need to edit it to match your actual infrastructure.

The Inventory File

The heart of any Ansible-based deployment is the inventory file. This file tells Ansible which machines to manage and how to connect to them. In Kubespray, the inventory also determines which machines become control plane nodes, which become workers, and which host etcd.

Open the inventory file and replace its contents with your cluster definition:

cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
k8s-ctr ansible_host=192.168.10.10 ip=192.168.10.10

[kube_control_plane]
k8s-ctr

[etcd:children]
kube_control_plane

[kube_node]
k8s-ctr
EOF

This inventory defines a single-node cluster where one machine plays all roles. Let me walk through each part because understanding this structure will save you hours of debugging later.

The first line defines a host:

k8s-ctr ansible_host=192.168.10.10 ip=192.168.10.10

Here, k8s-ctr is the hostname that Ansible will use internally. It is an alias, a convenient name you can reference throughout your playbooks and inventory. The ansible_host variable tells Ansible the actual IP address or hostname to use when establishing SSH connections. When Ansible needs to connect to k8s-ctr, it will SSH to 192.168.10.10.

The ip variable is specific to Kubespray, not a standard Ansible variable. Kubespray uses this to determine which IP address the Kubernetes components should bind to and advertise. This distinction between ansible_host and ip becomes critical in environments with multiple network interfaces.

You might wonder why both variables are set to the same value. In simple environments, they often are identical. But consider a scenario where you have a bastion host or jump server. You might SSH through one IP address but want Kubernetes to communicate over a different internal network. Or consider our VirtualBox environment where each VM has multiple network interfaces.

The VirtualBox NAT IP Trap

If you have been following this tutorial series from the beginning, you have encountered this problem before. VirtualBox creates a NAT interface as the first network adapter on each VM. This interface gets the IP address 10.0.2.15 on every single VM. It is designed for outbound internet access from the VM, not for inter-VM communication.

When Ansible gathers facts about a host, it collects information about all network interfaces. Kubespray then tries to determine which IP address to use for Kubernetes components. If you do not explicitly set the ip variable, Kubespray might pick 10.0.2.15 because it appears first in the interface list. The result is a cluster where the API server advertises itself at 10.0.2.15, and every worker node tries to contact that address, which is their own localhost NAT interface. Nothing works.

I have made this mistake more times than I care to admit. Even after documenting it thoroughly in previous tutorials, I still forget to set the ip variable when setting up new test clusters. The symptom is always the same: kubeadm init succeeds on the control plane, but kubeadm join fails on workers with connection refused errors pointing to 10.0.2.15.

By explicitly setting ip=192.168.10.10, you tell Kubespray to use the Host-Only network interface for all Kubernetes communication. This is the interface where your VMs can actually reach each other.

Understanding Ansible Groups

The inventory file defines several groups using the bracket notation. Each group serves a specific purpose in Kubespray.

[kube_control_plane]
k8s-ctr

This group lists all nodes that should run Kubernetes control plane components. These are the nodes that will run kube-apiserver, kube-controller-manager, and kube-scheduler. In a production cluster, you would list three or five nodes here for high availability. For this single-node lab, only k8s-ctr appears in the list.

[kube_node]
k8s-ctr

This group lists all nodes that should run workloads. In Kubernetes terminology, these are the worker nodes. They run kubelet and kube-proxy and can schedule pods. Notice that k8s-ctr appears here as well. This means your control plane node will also accept workloads. In production, you often want control plane nodes dedicated to cluster management, but for a learning environment, having one node do everything is perfectly fine.

The etcd group is more interesting:

[etcd:children]
kube_control_plane

This uses Ansible’s nested group feature. Instead of listing hosts directly, the :children suffix tells Ansible that the entries below are names of other groups, not hosts. This line says “the etcd group contains all hosts that are members of kube_control_plane.”

This pattern is elegant because it creates an automatic relationship. If you add a node to kube_control_plane, it automatically becomes an etcd member too. For the common case where etcd runs on control plane nodes, this reduces duplication and prevents inconsistencies.

But this is not the only way to configure etcd. In production environments with strict performance requirements, you might want dedicated etcd nodes that do nothing else. In that case, you would define the etcd group directly:

[etcd]
etcd-1 ansible_host=192.168.10.20
etcd-2 ansible_host=192.168.10.21
etcd-3 ansible_host=192.168.10.22

Kubespray supports both patterns. The sample inventory uses the children approach because it matches the common deployment model where etcd is colocated with control plane nodes.

The Implicit all Group

You might have noticed that the host definition line appears outside any group:

k8s-ctr ansible_host=192.168.10.10 ip=192.168.10.10

In Ansible, any host defined outside of explicit groups automatically belongs to the special all group. This group is reserved by Ansible and always contains every host in the inventory. Kubespray uses this property extensively. Variables defined in group_vars/all/ apply to every single node, regardless of what other groups they belong to.

Verifying Your Inventory

Before running any playbooks, verify that Ansible interprets your inventory correctly. Mistakes in inventory files often produce confusing errors during playbook execution. It is much easier to catch them early.

Check the etcd group:

ansible -i /root/kubespray/inventory/mycluster/inventory.ini etcd --list-hosts

The output should show:

hosts (1):
    k8s-ctr

Check the control plane group:

ansible -i /root/kubespray/inventory/mycluster/inventory.ini kube_control_plane --list-hosts

hosts (1):
    k8s-ctr

Check the worker node group:

ansible -i /root/kubespray/inventory/mycluster/inventory.ini kube_node --list-hosts

hosts (1):
    k8s-ctr

And verify the all group contains your host:

ansible -i /root/kubespray/inventory/mycluster/inventory.ini all --list-hosts

hosts (1):
    k8s-ctr

All four groups resolve to the same single host, which is exactly what we expect for a single-node cluster. If any of these commands show unexpected results, go back and check your inventory file syntax carefully. A misplaced bracket or typo can cause hosts to end up in the wrong groups.

Modifying Cluster Configuration

With the inventory defining your infrastructure, the next step is configuring how Kubespray should build the cluster. These settings live in the group_vars directory, which contains YAML files that set Ansible variables.

Kubespray provides sensible defaults for everything, but you will almost always want to customize some settings. For this tutorial, we will make several changes to k8s-cluster.yml, the file that controls core Kubernetes settings.

The changes we need to make are:

First, switch the CNI plugin from Calico to Flannel. Calico is the default and is an excellent choice for production, but Flannel is simpler and easier to understand when learning:

sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' \
  inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

Second, switch kube-proxy from IPVS mode to iptables mode. IPVS offers better performance at scale, but iptables is the traditional mode and is easier to debug:

sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' \
  inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

Third, disable NodeLocal DNSCache. This feature improves DNS performance by running a caching DNS server on each node, but it adds complexity that we do not need for learning:

sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' \
  inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

Fourth, enable automatic certificate renewal. Kubernetes certificates expire after one year, and this setting ensures they are renewed automatically:

sed -i 's|auto_renew_certificates: false|auto_renew_certificates: true|g' \
  inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

Fifth, uncomment the certificate renewal schedule. The setting exists in the file but is commented out by default:

sed -i 's|# auto_renew_certificates_systemd_calendar|auto_renew_certificates_systemd_calendar|g' \
  inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

After running all these commands, verify the changes took effect:

grep -iE 'kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:|^auto_renew_certificates' \
  inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

You should see:

kube_network_plugin: flannel
kube_proxy_mode: iptables
enable_nodelocaldns: false
auto_renew_certificates: true
auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:00:00"

The calendar expression for certificate renewal looks cryptic at first glance. It follows the systemd timer calendar format. “Mon — 1,2,3,4,5,6,7 03:00:00” means “every Monday that falls on the 1st through 7th of any month, at 3:00 AM.” In practice, this triggers certificate renewal once a month on the first Monday, at a time when the cluster is likely idle.

Configuring Flannel

Since we switched from Calico to Flannel, we need to configure Flannel-specific settings. Kubespray maintains separate configuration files for each CNI plugin. The Flannel settings live in k8s-net-flannel.yml.

One critical setting for VirtualBox environments is specifying which network interface Flannel should use for VXLAN traffic. Without this setting, Flannel might pick the wrong interface, just like Kubespray picking the wrong IP address when the ip variable is missing.

Check the current contents of the Flannel configuration:

cat inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml

The file contains various Flannel settings, mostly commented out because the defaults work for typical deployments. Add the interface specification:

echo "flannel_interface: enp0s8" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml

The interface name enp0s8 corresponds to the Host-Only network adapter in VirtualBox. This is where your VMs can communicate with each other. If you used different network settings in your Vagrantfile, you might need to adjust this value. You can check available interfaces on your VM by running ip addr.

Verify the setting was added:

grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml

This grep command shows only non-comment lines, filtering out the extensive documentation that Kubespray includes in its configuration files.

What These Settings Actually Do

Let me explain why we chose these particular settings, beyond just simplicity for learning.

Flannel versus Calico represents a fundamental tradeoff in Kubernetes networking. Calico uses BGP routing and supports NetworkPolicy enforcement, making it the preferred choice for production clusters that need fine-grained network security. Flannel uses simpler VXLAN encapsulation and does not support NetworkPolicy at all. For a single-node learning cluster where you are not going to test network policies, Flannel’s simplicity is an advantage. There is less to go wrong, and the networking behavior is easier to understand.

The kube-proxy mode setting affects how Kubernetes implements Service load balancing. In iptables mode, kube-proxy programs iptables rules that distribute traffic across pod endpoints. Every packet traverses the iptables rules, which works fine for moderate traffic loads. In IPVS mode, kube-proxy uses the Linux kernel’s IPVS (IP Virtual Server) module, which is designed for load balancing and handles high connection rates more efficiently. For a learning cluster, iptables mode is easier to inspect and debug. You can see the rules with iptables -L -t nat, which is helpful when troubleshooting service connectivity.

NodeLocal DNSCache improves DNS performance by running a DNS caching agent on every node. Instead of sending all DNS queries to CoreDNS pods that might be running on different nodes, queries go to the local cache first. This reduces latency and network traffic. However, it adds another moving part to the cluster, and for learning purposes, the extra complexity is not worth the performance gain.

Certificate auto-renewal is something you absolutely want in production. Kubernetes uses many certificates: the API server certificate, the kubelet client certificates, the etcd certificates, and more. These certificates typically have a one-year validity period. If they expire, your cluster stops working. The auto-renewal feature uses a systemd timer to run kubeadm certs renew automatically before expiration. The monthly schedule gives plenty of margin since certificates are valid for a full year.

The Variable Hierarchy Reminder

Remember from earlier chapters that Kubespray uses Ansible’s variable precedence system. The files you just modified in group_vars/k8s_cluster/ have a priority of 6 in Ansible’s hierarchy. They will override the default values defined in roles/kubespray_defaults/defaults/ (priority 2) but can themselves be overridden by host_vars files (priority 9) or command-line extra-vars (priority 22).

This means if you ever need to test a different setting without permanently changing your inventory, you can use the -e flag:

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml \
  -e "kube_network_plugin=cilium"

This would deploy with Cilium instead of Flannel, overriding the inventory setting just for this run. The inventory files remain unchanged, so subsequent runs without -e would use Flannel again.

This hierarchy is intentional. Kubespray defaults represent best practices that work for most deployments. Your inventory customizations represent your organization’s requirements. Command-line overrides let you experiment without changing anything permanently.

Preparing for Deployment

At this point, your inventory is configured and validated. You have defined which machine will be your cluster node, specified the correct IP address to avoid VirtualBox networking pitfalls, and customized the cluster configuration to use Flannel with iptables-mode kube-proxy.

The configuration is complete. In the next chapter, we will execute the deployment and watch Kubespray transform a bare Rocky Linux VM into a functioning Kubernetes cluster. Before proceeding, it is worth double-checking that SSH connectivity still works:

ansible -i inventory/mycluster/inventory.ini all -m ping

If this returns success, you are ready to deploy. If it fails, troubleshoot SSH connectivity before moving on. No amount of correct inventory configuration will help if Ansible cannot reach your nodes.

Chapter 10: Kubernetes The Kubespray Way — A Comparative Tutorial

This chapter walks through deploying a Kubernetes cluster using Kubespray, deliberately structured to mirror the steps from Kubernetes The Hard Way. The goal here is not just to get a cluster running, but to understand exactly which manual steps Kubespray automates for you. By the end, you will have a working cluster and a clear mental map of what happens behind that single ansible-playbook command.

The tutorial follows the official Kubespray documentation “Setting up your first cluster,” which was itself inspired by Kubernetes The Hard Way. The difference is that instead of manually executing hundreds of commands across multiple nodes, you will define your desired state in inventory files and let Ansible do the work.

Prerequisites

In Kubernetes The Hard Way, the prerequisites chapter covered installing VirtualBox and Vagrant, verifying host system requirements, and planning the network topology. The same applies here, but Kubespray adds a few more requirements on the control node side.

For Kubespray, you need a machine that will run Ansible. This is called the Ansible Control Node, and it sits outside the cluster. From this machine, you will SSH into all the target nodes and orchestrate the installation. The target nodes themselves only need SSH access and Python installed.

Here is the requirement summary:

Ansible Control Node: Linux or Mac with Python 3 installed
Target Nodes: SSH accessible, Python installed
Network: Nodes can communicate with each other, internet access for downloading images
Privileges: Root or sudo access on all target nodes

The original documentation uses Google Cloud Platform for the infrastructure. This tutorial uses Vagrant instead, for several reasons. Vagrant is free, runs locally without network latency, and you have been using it throughout the previous tutorials. There is no reason to change now.

Provisioning Compute Resources

In Kubernetes The Hard Way, you created four VMs: a Jumpbox for running commands, a Server for the control plane, and two worker nodes. The Kubespray setup follows a similar pattern, but the Jumpbox becomes the Ansible Control Node.

Here is the node layout for this tutorial:

controller (192.168.10.10) — Ansible Control Node, equivalent to the Jumpbox
controller-0 (192.168.10.100) — Control Plane node, equivalent to the Server
worker-0 (192.168.10.101) — Worker node
worker-1 (192.168.10.102) — Worker node

Create a directory for this lab and add the following Vagrantfile:

BOX_IMAGE = "bento/rockylinux-10.0" BOX_VERSION = "202510.26.0"
Vagrant.configure("2") do |config|
config.vm.define "controller" do |subconfig| subconfig.vm.box = BOX_IMAGE subconfig.vm.box_version = BOX_VERSION subconfig.vm.provider "virtualbox" do |vb| vb.customize ["modifyvm", :id, " - groups", "/Kubespray-Lab"] vb.customize ["modifyvm", :id, " - nicpromisc2", "allow-all"] vb.name = "controller" vb.cpus = 2 vb.memory = 2048 vb.linked_clone = true end subconfig.vm.host_name = "controller" subconfig.vm.network "private_network", ip: "192.168.10.10" subconfig.vm.synced_folder "./", "/vagrant", disabled: true end
config.vm.define "controller-0" do |subconfig| subconfig.vm.box = BOX_IMAGE subconfig.vm.box_version = BOX_VERSION subconfig.vm.provider "virtualbox" do |vb| vb.customize ["modifyvm", :id, " - groups", "/Kubespray-Lab"] vb.customize ["modifyvm", :id, " - nicpromisc2", "allow-all"] vb.name = "controller-0" vb.cpus = 2 vb.memory = 2048 vb.linked_clone = true end subconfig.vm.host_name = "controller-0" subconfig.vm.network "private_network", ip: "192.168.10.100" subconfig.vm.synced_folder "./", "/vagrant", disabled: true end
config.vm.define "worker-0" do |subconfig| subconfig.vm.box = BOX_IMAGE subconfig.vm.box_version = BOX_VERSION subconfig.vm.provider "virtualbox" do |vb| vb.customize ["modifyvm", :id, " - groups", "/Kubespray-Lab"] vb.customize ["modifyvm", :id, " - nicpromisc2", "allow-all"] vb.name = "worker-0" vb.cpus = 2 vb.memory = 2048 vb.linked_clone = true end subconfig.vm.host_name = "worker-0" subconfig.vm.network "private_network", ip: "192.168.10.101" subconfig.vm.synced_folder "./", "/vagrant", disabled: true end
config.vm.define "worker-1" do |subconfig| subconfig.vm.box = BOX_IMAGE subconfig.vm.box_version = BOX_VERSION subconfig.vm.provider "virtualbox" do |vb| vb.customize ["modifyvm", :id, " - groups", "/Kubespray-Lab"] vb.customize ["modifyvm", :id, " - nicpromisc2", "allow-all"] vb.name = "worker-1" vb.cpus = 2 vb.memory = 2048 vb.linked_clone = true end subconfig.vm.host_name = "worker-1" subconfig.vm.network "private_network", ip: "192.168.10.102" subconfig.vm.synced_folder "./", "/vagrant", disabled: true end
end

Bring up all four VMs:

vagrant up

This takes a few minutes. Once complete, verify all machines are running:

vagrant status

You should see all four VMs in the “running” state.

Configuring SSH Access

Just like in Kubernetes The Hard Way where you set up SSH from the Jumpbox to the other nodes, Ansible needs passwordless SSH access to all target nodes. Log into the controller node:

vagrant ssh controller

Generate an SSH key pair:

ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa

Now copy the public key to each target node. The default password for Vagrant boxes is “vagrant”:

ssh-copy-id root@192.168.10.100 ssh-copy-id root@192.168.10.101 ssh-copy-id root@192.168.10.102

Test that passwordless SSH works:

ssh root@192.168.10.100 "hostname" ssh root@192.168.10.101 "hostname" ssh root@192.168.10.102 "hostname"

Each command should return the hostname without prompting for a password. If this works, you are ready to set up Kubespray.

Setting Up Kubespray

This is where things diverge dramatically from The Hard Way. In The Hard Way, the next several hours would be spent manually generating certificates, creating kubeconfig files, writing systemd unit files, and bootstrapping etcd. With Kubespray, you will spend the next few minutes configuring inventory files, and then a single command handles the rest.

Still on the controller node, create a Python virtual environment. Ansible is a Python application, and using a virtual environment keeps dependencies isolated:

python3 -m venv venv source venv/bin/activate

Clone the Kubespray repository and check out a stable release:

git clone https://github.com/kubernetes-sigs/kubespray.git cd kubespray git checkout release-2.28

Install the required Python packages:

pip install -r requirements.txt

You will see Ansible and its dependencies being installed. The output ends with something like:

Successfully installed MarkupSafe-3.0.3 PyYAML-6.0.3 ansible-9.13.0 ansible-core-2.16.15 …

Configuring the Inventory

Kubespray uses Ansible inventory files to define which hosts belong to which groups. Copy the sample inventory to create your own.

cp -rfp inventory/sample inventory/mycluster

Now edit the inventory file to match your node layout. Open inventory/mycluster/inventory.ini and replace its contents with:

[all] controller-0 ansible_host=192.168.10.100 ip=192.168.10.100 worker-0 ansible_host=192.168.10.101 ip=192.168.10.101 worker-1 ansible_host=192.168.10.102 ip=192.168.10.102
[kube_control_plane] controller-0
[etcd] controller-0
[kube_node] worker-0 worker-1
[calico_rr]
[k8s_cluster:children] kube_control_plane kube_node calico_rr

Let me explain what each section does.

The [all] section lists every host with its connection details. The ansible_host variable tells Ansible which IP address to SSH to. The ip variable is critical and often misunderstood. This tells Kubespray which IP address to use for Kubernetes internal communication. If you omit ip, Kubespray will try to auto-detect it, and on VirtualBox it often picks the NAT interface (10.0.2.15) instead of the host-only network. This causes the cluster to fail because nodes cannot reach each other on 10.0.2.15.

The [kube_control_plane] group contains nodes that will run the Kubernetes control plane components: kube-apiserver, kube-controller-manager, and kube-scheduler.

The [etcd] group contains nodes that will run the etcd cluster. In this setup, etcd runs on the same node as the control plane, which is called “stacked etcd.” For production, you might run etcd on dedicated nodes.

The [kube_node] group contains worker nodes where your application pods will run.

The [calico_rr] group is for Calico Route Reflectors, used in large-scale deployments. Leave it empty for now.

The [k8s_cluster:children] group is a meta-group that includes all Kubernetes cluster members. The :children suffix means this group inherits members from the listed child groups.

Notice how the sample inventory uses [etcd:children] with kube_control_plane listed under it. This is a shorthand that says “the etcd group consists of all hosts in the kube_control_plane group.” In this tutorial, we list controller-0 directly under [etcd] for clarity, but both approaches produce the same result.

Understanding the Default Configuration

Before deploying, take a moment to look at the default configuration. The most important file is inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml. You can view the active settings with:

grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml | head -30

Some key defaults you will notice:

kube_network_plugin: calico — The CNI plugin for pod networking
kube_service_addresses: 10.233.0.0/18 — The CIDR range for Kubernetes services
kube_pods_subnet: 10.233.64.0/18 — The CIDR range for pods
container_manager: containerd — The container runtime
kube_proxy_mode: ipvs — How kube-proxy handles service routing

For this tutorial, keep the defaults. In production, you would carefully review and customize these settings before deployment.

One optional but recommended setting is enabling the Metrics Server. Edit inventory/mycluster/group_vars/k8s_cluster/addons.yml and find the metrics_server_enabled line. Change it to:

metrics_server_enabled: true

The Metrics Server provides resource usage data for commands like kubectl top and is required for Horizontal Pod Autoscaler.

Deploying the Cluster

Everything up to this point has been preparation. Now comes the moment of truth. From the kubespray directory, run:

ansible-playbook -i inventory/mycluster/inventory.ini 
 -u root -b -v 
 - private-key=~/.ssh/id_rsa 
 cluster.yml

Let me break down this command:

-i inventory/mycluster/inventory.ini specifies the inventory file
-u root tells Ansible to connect as the root user
-b enables "become" mode for privilege escalation (though root does not need it)
-v enables verbose output so you can see what is happening
--private-key specifies the SSH key to use
cluster.yml is the main playbook that deploys everything

Press enter and watch the output scroll by. This takes 15 to 30 minutes depending on your hardware and internet speed. You will see hundreds of tasks execute across multiple plays.

If everything succeeds, the final output looks like:

PLAY RECAP ********************************************************************* controller-0 : ok=XXX changed=XX unreachable=0 failed=0 … worker-0 : ok=XXX changed=XX unreachable=0 failed=0 … worker-1 : ok=XXX changed=XX unreachable=0 failed=0 …

The key metric is failed=0 for all hosts.

Troubleshooting: Inventory Path Issues

If you see warnings like “Unable to parse inventory” and all plays show “skipping: no hosts matched,” check your inventory path. The official documentation uses a directory path:

-i inventory/mycluster/

However, this does not work reliably in all environments. Specify the file directly:

-i inventory/mycluster/inventory.ini

Troubleshooting: USERNAME Variable

If you get an error about expecting an argument for -u, or if the ansible-playbook help is displayed instead of running the playbook, the $USERNAME environment variable is not set. The official documentation assumes you ran:

USERNAME=$(whoami)

Either set this variable or replace $USERNAME with root directly in the command.

Troubleshooting: VirtualBox NAT IP Problem

This is the most common issue with VirtualBox environments, and you have encountered it before in previous tutorials. If the deployment fails during the worker node join phase with an error like:

error execution phase preflight: couldn't validate the identity of the API Server: Get "https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.0.2.15:6443: connect: connection refused

The problem is that Kubespray detected the wrong IP address. VirtualBox VMs have a NAT interface as their first network adapter, which gets the IP 10.0.2.15. This IP is not routable between VMs. The fix is to explicitly set the ip variable in your inventory, which you already did in the configuration above.

If you hit this error, verify your inventory.ini has ip= set for every host:

[all] controller-0 ansible_host=192.168.10.100 ip=192.168.10.100 worker-0 ansible_host=192.168.10.101 ip=192.168.10.101 worker-1 ansible_host=192.168.10.102 ip=192.168.10.102

Then run the playbook again. Thanks to Ansible’s idempotency, it will skip the already-completed tasks and continue from where it failed.

Troubleshooting: etcd Health Check Failure After IP Change

Here is a more insidious problem. Suppose you initially ran the playbook without the ip variable set, the control plane was configured with 10.0.2.15, then you fixed the inventory and ran again. This time you might see:

TASK [etcd : Configure | Wait for etcd cluster to be healthy] ********************* fatal: [controller-0]: FAILED! => { "cmd": "… /usr/local/bin/etcdctl endpoint - cluster health …", "stderr": "… dial tcp 192.168.10.100:2379: connect: connection refused …" }

The etcd service was already installed and configured to bind to 10.0.2.15. Changing the ip variable does not automatically reconfigure already-running services. Ansible’s idempotency means “if the service is running, don’t touch it.”

The solution is to reset the cluster and start fresh:

ansible-playbook -i inventory/mycluster/inventory.ini 
 -u root -b -v 
 - private-key=~/.ssh/id_rsa 
 reset.yml

When prompted with “Are you sure you want to reset cluster state?”, type yes. This removes all Kubernetes components and etcd data, returning the nodes to a clean state.

After the reset completes, run cluster.yml again:

ansible-playbook -i inventory/mycluster/inventory.ini 
 -u root -b -v 
 - private-key=~/.ssh/id_rsa 
 cluster.yml

This time, everything installs correctly with the right IP addresses.

Note: When you run cluster.yml after reset.yml, you might see an error during the etcd version check:

fatal: [controller-0]: FAILED! => {"msg": "[Errno 2] No such file or directory: b'/usr/local/bin/etcd'"}

This is expected. The reset removed the etcd binary, so the version check fails. Kubespray handles this gracefully and proceeds to install etcd. Check the final PLAY RECAP — if failed=0, everything is fine.

Accessing the Cluster

The cluster is deployed, but you still need kubectl access from outside the cluster. Kubespray created a kubeconfig file on the control plane node at /etc/kubernetes/admin.conf. You need to copy this to your Ansible control node.

Still on the controller node, create the kubectl configuration directory:

mkdir -p ~/.kube

Copy the kubeconfig from the control plane:

scp root@192.168.10.100:/etc/kubernetes/admin.conf ~/.kube/config

There is one more step. The kubeconfig file was generated for use on the control plane node itself, so it points to localhost:

cat ~/.kube/config | grep server

You will see server: https://127.0.0.1:6443. This does not work from the controller node because the API server is not running locally. Fix it by replacing 127.0.0.1 with the actual control plane IP:

sed -i ‘s/127.0.0.1/192.168.10.100/g’ ~/.kube/config

Now install kubectl. If your controller node does not have it already:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl sudo mv kubectl /usr/local/bin/

Test the connection:

kubectl get nodes

You should see all three nodes:

NAME STATUS ROLES AGE VERSION
controller-0 Ready control-plane 11m v1.32.11
worker-0 Ready <none> 10m v1.32.11 worker-1 Ready <none> 10m v1.32.11

The cluster is ready.

Smoke Tests

Run the same verification tests from Kubernetes The Hard Way to confirm everything works.

If you enabled Metrics Server, check that resource metrics are available:

kubectl top nodes

After a few minutes for metrics collection to start:

NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
 controller-0 98m 7% 1939Mi 66%
 worker-0 40m 2% 977Mi 99%
 worker-1 43m 3% 994Mi 101%

Network — Pod to Pod Communication

Open two terminal windows, both connected to the controller node.

In the first terminal, create a pod and note its IP:

kubectl run myshell1 -it - rm - image busybox - sh

Once inside the pod:

hostname -i

Note the IP address, something like 10.233.107.4.

In the second terminal, create another pod and ping the first one:

kubectl run myshell2 -it - rm - image busybox - sh

Inside the second pod:

ping 10.233.107.4

You should see successful ping responses:

64 bytes from 10.233.107.4: seq=0 ttl=62 time=0.686 ms 64 bytes from 10.233.107.4: seq=1 ttl=62 time=0.620 ms

This confirms pod-to-pod networking works. Calico is doing its job.

Exit both pods with exit or Ctrl-D.

Deployments

Create a simple deployment:

kubectl create deployment nginx — image=nginx

Watch the pod come up:

kubectl get pods -l app=nginx

Wait until STATUS shows Running.

Port Forwarding

Forward a local port to the nginx pod:

POD_NAME=$(kubectl get pods -l app=nginx -o jsonpath="{.items[0].metadata.name}") kubectl port-forward $POD_NAME 8080:80

In another terminal:

curl — head http://127.0.0.1:8080

You should see HTTP/1.1 200 OK and nginx headers.

Logs

Retrieve container logs:

kubectl logs $POD_NAME

You will see nginx startup logs and, if you ran the curl command, the access log entry.

Exec

Execute a command inside the container:

kubectl exec -ti $POD_NAME — nginx -v

Output: nginx version: nginx/1.x.x

Services — NodePort

Expose the deployment as a NodePort service:

kubectl expose deployment nginx — port 80 — type NodePort

Find the assigned node port:

NODE_PORT=$(kubectl get svc nginx -o jsonpath='{.spec.ports[0].nodePort}')
echo $NODE_PORT

Access the service through a worker node’s IP:

curl -I http://192.168.10.101:$NODE_PORT

You should get HTTP/1.1 200 OK.

Local DNS

Test cross-namespace DNS resolution. First, create a namespace and deploy nginx there:

kubectl create namespace dev
kubectl create deployment nginx - image=nginx -n dev
kubectl expose deployment nginx - port 80 - type ClusterIP -n dev

Now create a pod in the default namespace and access the service in the dev namespace by DNS name:

kubectl run curly -it — rm — image curlimages/curl:7.70.0 — /bin/sh

Inside the pod:

curl — head http://nginx.dev:80

This resolves nginx.dev to nginx.dev.svc.cluster.local and returns the nginx response. CoreDNS is working correctly.

Cleaning Up

When you are done experimenting, clean up the Kubernetes resources:

kubectl delete namespace dev
kubectl delete deployment nginx
kubectl delete svc nginx

To reset the cluster but keep the VMs (for example, to redeploy with different settings):

ansible-playbook -i inventory/mycluster/inventory.ini 
 -u root -b -v 
 - private-key=~/.ssh/id_rsa 
 reset.yml

To completely remove everything:

exit # Leave the controller VM vagrant destroy -f

Chapter 11: containerd Configuration File Explained

I kept running into the same problem. Every time I needed to configure containerd, I found myself searching “containerd config.toml” and piecing together information from five different sources. After the third time doing this in a month, I decided to sit down and actually understand the configuration file properly. This chapter is the result of that effort.

Understanding TOML Before We Start

containerd uses TOML for its configuration file. If you’ve never encountered TOML before, you might wonder why not just use JSON or YAML like everything else in the Kubernetes ecosystem. The answer comes down to readability and structure.

TOML stands for Tom’s Obvious, Minimal Language. It looks similar to INI files but has a more rigorous specification. Here’s what basic TOML looks like:

# Comments start with hash
key = "value"
number = 42
boolean = true

[section]
  nested_key = "nested_value"

[section.subsection]
  deep_key = "deep_value"

containerd chose TOML because its configuration is deeply hierarchical. Plugins have sub-configurations, which have their own sub-configurations. TOML handles this nesting elegantly while remaining readable.

JSON would work but becomes a mess of brackets. YAML works but its significant whitespace causes endless debugging sessions when someone mixes tabs and spaces.

The bracket notation in TOML defines what are called tables, which are essentially sections or namespaces. When you see [grpc], everything that follows belongs to the grpc configuration until the next table declaration. When you see [plugins."io.containerd.grpc.v1.cri"], that's a table with a quoted key because the key contains dots that would otherwise be interpreted as nested tables.

This distinction matters. Consider the difference:

# This creates nested tables: plugins.io.containerd.grpc.v1.cri
[plugins.io.containerd.grpc.v1.cri]

# This creates a single table with a key that happens to contain dots
[plugins."io.containerd.grpc.v1.cri"]

containerd uses the second form. The entire string “io.containerd.grpc.v1.cri” is the plugin identifier, not a hierarchy.

Where containerd Looks for Configuration

When containerd starts, it looks for configuration in this order. First, it checks if you passed a — config flag:

containerd --config /path/to/custom-config.toml

If you did, it uses that file and ignores everything else. If you didn’t pass — config, it looks at the default path:

/etc/containerd/config.toml

If that file doesn’t exist either, containerd runs with built-in defaults. You can see these defaults anytime:

containerd config default

This command dumps the complete default configuration to stdout. It’s incredibly useful when you’re trying to figure out what options exist. If you want to see what configuration containerd is actually running with right now, use:

containerd config dump

The difference matters. The default command shows what containerd would use if you had no config file. The dump command shows the merged result of your config file plus defaults for anything you didn’t specify.

Configuration File Versions

The config file has a version field, and this trips people up constantly. If you don’t specify a version, containerd assumes version 1, which is deprecated and removed in containerd 2.x. You should always explicitly set the version.

For containerd 1.x, use version 2:

version = 2

For containerd 2.x, use version 3:

version = 3

The version numbers refer to the configuration schema, not the containerd version. This naming is confusing but we’re stuck with it.

What changed between versions? Version 1 to 2 added the “io.containerd.” prefix to all plugin identifiers. Version 2 to 3 reorganized the CRI plugin structure and changed some default values.

containerd performs automatic migration when it starts. If you have a version 2 config and run containerd 2.x, it converts the config in memory to version 3 format. Your original file stays unchanged, but there’s a small performance cost at startup. If you want to avoid this, you can pre-migrate your config:

containerd config migrate /etc/containerd/config.toml > /etc/containerd/config.toml.new

Review the output before replacing your original. Migrated configs aren’t backward compatible, so you can’t easily roll back to an older containerd version.

Global Settings at the Top

The top of config.toml contains settings that apply to containerd as a whole, not to any specific plugin.

version = 3
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
oom_score = 0

The root directory is where containerd stores persistent data. Images, container metadata, snapshots — everything that should survive a reboot lives here. Don’t put this on a tmpfs.

The state directory holds runtime state. Sockets, PID files, information about running containers. This data is ephemeral and gets recreated on restart. That’s why it defaults to /run, which is typically a tmpfs.

The temp setting specifies where containerd creates temporary files. If empty, it uses the system default (usually /tmp).

The oom_score adjusts how the Linux OOM killer prioritizes containerd. Values range from -1000 to 1000. Lower values mean the process is less likely to be killed. A value of 0 means no adjustment from the default.

There’s also an imports field that lets you split configuration across multiple files:

version = 3
imports = ["/etc/containerd/conf.d/*.toml"]

This works like nginx’s conf.d pattern. You can drop additional config files into that directory and they’ll be merged. In version 3, the default imports path is /etc/containerd/conf.d/*.toml. Files are processed in glob order, and later files override earlier ones.

This is useful for managing configuration in production. Your base config stays in config.toml, while environment-specific settings go in conf.d. For example:

/etc/containerd/config.toml              # Base configuration
/etc/containerd/conf.d/50-registry.toml  # Private registry settings
/etc/containerd/conf.d/99-nvidia.toml    # GPU runtime configuration

The gRPC Section

containerd communicates with clients through gRPC. This is not an implementation choice — it’s a requirement. The Container Runtime Interface (CRI) that Kubernetes uses is defined as a gRPC service using Protocol Buffers. Any container runtime that wants to work with Kubernetes must speak gRPC.

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

The address is the Unix socket path where containerd listens. The kubelet connects to this socket to create and manage containers. The uid and gid control who owns the socket file. Setting both to 0 means root owns it, so only root can connect.

The message size limits are 16MB by default. You rarely need to change these unless you’re doing something unusual with very large container specs.

There’s also support for TCP with TLS if you need remote access:

[grpc]
  address = "/run/containerd/containerd.sock"
  tcp_address = "0.0.0.0:10000"
  tcp_tls_cert = "/etc/containerd/cert.pem"
  tcp_tls_key = "/etc/containerd/key.pem"
  tcp_tls_ca = "/etc/containerd/ca.pem"

Most people leave tcp_address empty because exposing containerd over the network is a significant security risk.

The ttrpc Section

You’ll also see a ttrpc section:

[ttrpc]
  address = ""
  uid = 0
  gid = 0

TTRPC is a lightweight RPC protocol that containerd developed for communication with shims. A shim is the process that sits between containerd and the actual container process. When containerd needs to talk to a shim, it uses TTRPC instead of gRPC because the overhead is lower.

In most deployments, you can ignore this section entirely. The default empty address means TTRPC is disabled, and containerd uses other mechanisms for shim communication.

Debug and Metrics

For troubleshooting, there’s a debug section:

[debug]
  address = "/run/containerd/debug.sock"
  uid = 0
  gid = 0
  level = "info"
  format = "text"

The level can be trace, debug, info, warn, error, fatal, or panic. During initial setup, setting this to debug helps immensely. In production, info or warn is appropriate.

The format can be text or json. Text is readable for humans watching logs. JSON is better when you’re feeding logs into a log aggregation system.

containerd can also expose Prometheus metrics:

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

By default, address is empty and metrics are disabled. Set an address to enable the metrics endpoint. The grpc_histogram option adds detailed gRPC latency histograms but increases cardinality significantly.

The Plugins Section

Now we get to the heart of containerd configuration. Almost everything in containerd is a plugin. The CRI implementation that Kubernetes talks to? A plugin. The snapshotter that manages image layers? A plugin. The runtime that actually runs containers? A plugin.

Each plugin has its own configuration section under [plugins]:

[plugins."io.containerd.grpc.v1.cri"]
  # CRI plugin configuration

[plugins."io.containerd.gc.v1.scheduler"]
  # Garbage collection scheduler

[plugins."io.containerd.snapshotter.v1.overlayfs"]
  # OverlayFS snapshotter

The plugin identifiers follow a pattern: io.containerd.<type>.<version>.<name>. For example, io.containerd.grpc.v1.cri is a gRPC service plugin, version 1, named cri.

Let me walk through the important plugins.

The garbage collection scheduler (io.containerd.gc.v1.scheduler) cleans up unused content:

[plugins."io.containerd.gc.v1.scheduler"]
  pause_threshold = 0.02
  deletion_threshold = 0
  mutation_threshold = 100
  schedule_delay = "0s"
  startup_delay = "100ms"

The thresholds control how aggressively garbage collection runs. Higher values mean more frequent collection. The startup_delay prevents GC from running immediately when containerd starts, giving the system time to stabilize.

The metadata plugin (io.containerd.metadata.v1.bolt) stores containerd’s metadata in a BoltDB database:

[plugins."io.containerd.metadata.v1.bolt"]
  content_sharing_policy = "shared"

The content_sharing_policy determines whether multiple containers can share the same content. Shared is the default and appropriate for most cases.

The overlayfs snapshotter (io.containerd.snapshotter.v1.overlayfs) manages container filesystem layers. On most Linux systems, this is the default snapshotter and usually needs no configuration.

The CRI Plugin in Detail

For Kubernetes integration, the CRI plugin is what matters most. In version 2 configs, everything is under one plugin:

[plugins."io.containerd.grpc.v1.cri"]

In version 3, it’s split into two:

[plugins."io.containerd.cri.v1.images"]
[plugins."io.containerd.cri.v1.runtime"]

I’ll focus on version 2 since that’s what most people are still using. The concepts translate directly to version 3.

The most commonly configured options:

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"
  max_concurrent_downloads = 3
  max_container_log_line_size = 16384

The sandbox_image is the pause container that Kubernetes uses to hold network namespaces for pods. If you’re in an air-gapped environment, you need to change this to point to your internal registry.

The max_concurrent_downloads limits how many image layers containerd downloads in parallel. Increase this if you have good network bandwidth and want faster image pulls. Decrease it if you’re overwhelming your registry or network.

Underneath the CRI plugin, there’s a containerd subsection for runtime configuration:

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  default_runtime_name = "runc"

The snapshotter setting determines which snapshotter plugin manages image layers. OverlayFS is the standard choice on modern Linux.

The default_runtime_name specifies which runtime to use when one isn’t explicitly requested. This points to a runtime defined in the runtimes subsection.

Configuring Container Runtimes

The runtimes subsection is where you define the OCI runtimes that containerd can use:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    BinaryName = ""
    SystemdCgroup = true

This defines a runtime named “runc”. The runtime_type specifies which shim to use. The io.containerd.runc.v2 shim is the standard choice for runc.

The options section contains runtime-specific settings. For runc, the important ones are:

BinaryName specifies the path to the runc binary. If empty, containerd searches PATH for “runc”. You can set an explicit path like “/usr/local/bin/runc” if you have multiple runc versions installed.

SystemdCgroup is critical for Kubernetes. When true, runc uses systemd to manage cgroups instead of the filesystem directly. This must match your kubelet configuration. If kubelet is configured with cgroupDriver: systemd (which is the default and recommended), then SystemdCgroup must be true. Mismatching these causes subtle and frustrating problems.

You can define multiple runtimes for different use cases:

[plugins."io.containerd.grpc.v1.cri".containerd]
  default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
    SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"

Here I’ve defined three runtimes. The default is runc for normal containers. There’s also an nvidia runtime for GPU workloads and a kata runtime for VM-isolated containers.

To use a non-default runtime in Kubernetes, you create a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

The handler field must match the runtime name in your containerd config. Then pods can request this runtime:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  runtimeClassName: nvidia
  containers:
  - name: cuda
    image: nvidia/cuda:12.0-base

CNI Configuration

The CRI plugin also manages CNI (Container Network Interface) settings:

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"

The bin_dir is where CNI plugin binaries live. The conf_dir is where CNI configuration files go. These are the standard paths that most CNI plugins expect.

In a Kubernetes cluster, you typically don’t configure CNI through containerd directly. Instead, you install a CNI plugin like Calico or Flannel, which drops its own configuration files into conf_dir. But understanding these paths helps when troubleshooting network issues.

Registry Configuration

Registry configuration changed significantly between versions. In version 2, there were two approaches. The old way used inline configuration:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]

The new way (recommended) uses a config_path:

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

With config_path, you create a directory structure:

/etc/containerd/certs.d/
├── docker.io/
│   └── hosts.toml
├── gcr.io/
│   └── hosts.toml
└── my-registry.example.com/
    ├── hosts.toml
    └── ca.crt

Each registry gets its own directory with a hosts.toml file:

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]
[host."https://mirror.example.com"]
  capabilities = ["pull", "resolve"]
  skip_verify = false

This approach is more flexible. You can add registries without editing the main config file, and you can include CA certificates alongside the host configuration.

The capabilities field specifies what operations are allowed. Pull means downloading images, resolve means looking up image metadata, push means uploading images. Most configurations only need pull and resolve.

The skip_verify option disables TLS certificate verification. Never set this to true in production. If you’re using a private CA, put the CA certificate in the same directory and reference it:

[host."https://my-registry.example.com"]
  capabilities = ["pull", "resolve", "push"]
  ca = "/etc/containerd/certs.d/my-registry.example.com/ca.crt"

A Complete Configuration Example

Let me put this all together into a complete version 2 configuration that you might use in production:

version = 2

root = "/var/lib/containerd"
state = "/run/containerd"
[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
[debug]
  level = "info"
[metrics]
  address = "127.0.0.1:1338"
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"
  max_concurrent_downloads = 3
  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true
  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

This configuration enables metrics, uses systemd cgroups (matching a typical kubelet configuration), and uses the hosts.toml approach for registry configuration.

K3s and containerd

If you use K3s, you’ll encounter a different configuration pattern. K3s embeds containerd and generates its own configuration file at:

/var/lib/rancher/k3s/agent/etc/containerd/config.toml

This file has a comment at the top:

# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.

K3s regenerates this file on every start. If you need to customize containerd in K3s, you create a template file:

/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

K3s uses Go templates, so you can include the base configuration and add your customizations:

version = 2

{{ template "base" . }}
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
    SystemdCgroup = true

The {{ template "base" . }} line includes K3s's default configuration. Your additions come after.

For simple registry configuration changes in K3s, there’s an easier approach. Edit /etc/rancher/k3s/registries.yaml:

mirrors:
  docker.io:
    endpoint:
      - "https://mirror.example.com"
configs:
  "my-registry.example.com":
    tls:
      ca_file: "/etc/rancher/k3s/certs/ca.crt"

K3s translates this YAML into the appropriate containerd configuration automatically.

Verifying Your Configuration

After changing containerd configuration, restart the service and verify everything is working:

systemctl restart containerd
systemctl status containerd

Check that the configuration loaded correctly:

containerd config dump | grep -A5 "runtimes.runc"

Test that you can pull images:

crictl pull docker.io/library/alpine:latest

If you’re running Kubernetes, verify that kubelet can communicate with containerd:

crictl info

This command returns JSON with containerd’s current state. Look for the runtime configuration to confirm your changes took effect.

When troubleshooting, the containerd logs are essential:

journalctl -u containerd -f

Watch these logs while performing operations to understand what containerd is doing and where failures occur.

The configuration might seem overwhelming at first, but most of it is things you’ll never touch. In practice, you’ll spend 90% of your configuration time in the CRI plugin section, dealing with runtimes and registries. The rest is set once and forgotten.

Chapter 12: High Availability Configuration

When you run a single control plane node, you have a single point of failure. The API server goes down, and suddenly kubectl stops working, new pods cannot be scheduled, and your cluster is effectively dead. In production, this is unacceptable. You need high availability.

Kubespray was built with HA in mind from the start. Unlike kubeadm where you have to manually configure load balancers, set up additional control plane nodes, and worry about etcd quorum, Kubespray handles most of this automatically. But “automatically” does not mean “magically.” You still need to understand what is happening under the hood, because when something breaks at 3 AM, you will be the one fixing it.

Let me walk you through how Kubespray implements HA, starting with the control plane.

Control Plane High Availability

A Kubernetes control plane consists of three main components: the API server, the controller manager, and the scheduler. Each of these has different HA characteristics.

The API server is stateless. It reads from and writes to etcd, but it does not maintain any state itself. This means you can run multiple API server instances simultaneously, and they will all work correctly. Every API server instance is active and can handle requests. This is called active-active configuration.

The controller manager and scheduler are different. They maintain internal state and make decisions that could conflict if multiple instances ran simultaneously. Imagine two controller managers both deciding to create a replacement pod for a failed one.

You would end up with two replacement pods instead of one. To prevent this, Kubernetes uses leader election. Only one instance is active at any time, while the others wait in standby. If the leader fails, one of the standby instances takes over. This is active-standby configuration.

When you configure multiple control plane nodes in Kubespray, here is what happens. On each control plane node, Kubespray deploys the API server, controller manager, and scheduler as static pods. The API servers all run and accept requests. The controller managers and schedulers all run, but only one of each is the leader. The others continuously try to acquire the leadership lease stored in etcd.

You can see the leader election in action by checking the lease objects:

kubectl get lease -n kube-system

You will see leases named kube-controller-manager and kube-scheduler. The holderIdentity field shows which node currently holds the leadership.

To configure an HA control plane with Kubespray, you simply list multiple nodes in the kube_control_plane group. Here is an example inventory for a three-node HA setup:

[all]
k8s-ctr1 ansible_host=192.168.10.11 ip=192.168.10.11
k8s-ctr2 ansible_host=192.168.10.12 ip=192.168.10.12
k8s-ctr3 ansible_host=192.168.10.13 ip=192.168.10.13
k8s-w1 ansible_host=192.168.10.21 ip=192.168.10.21
k8s-w2 ansible_host=192.168.10.22 ip=192.168.10.22

[kube_control_plane]
k8s-ctr1
k8s-ctr2
k8s-ctr3
[etcd]
k8s-ctr1
k8s-ctr2
k8s-ctr3
[kube_node]
k8s-w1
k8s-w2
[k8s_cluster:children]
kube_control_plane
kube_node

Notice that all three control plane nodes are also in the etcd group. This is a common pattern called stacked etcd, where etcd runs on the same nodes as the control plane. I will discuss the alternative, external etcd, later in this chapter.

With this inventory, Kubespray will deploy everything needed for HA. But there is one critical piece missing from this picture: how do clients know which API server to talk to?

The Load Balancing Problem

When you have three API servers running on three different nodes, you need some way to distribute traffic among them. If a client always connects to k8s-ctr1:6443 and that node goes down, the client cannot reach the cluster even though k8s-ctr2 and k8s-ctr3 are perfectly healthy.

There are two approaches to solving this: external load balancers and client-side load balancing.

External load balancers sit in front of your API servers and distribute incoming connections. In AWS, you would use an ELB or NLB. In GCP, you would use a GCP Load Balancer. On-premises, you might use HAProxy with keepalived and a virtual IP. The external load balancer provides a single endpoint that clients connect to, and it forwards requests to healthy API servers.

Kubespray does not automatically configure external load balancers. This is a deliberate design decision, not an oversight. The reason is simple: external load balancers are infrastructure-level components that vary dramatically between environments. An AWS NLB requires AWS API calls to provision. A GCP Load Balancer requires GCP API calls. An on-premises HAProxy setup requires virtual IP configuration, which involves network infrastructure that Kubespray cannot possibly know about.

Kubespray is a tool for configuring software on top of your operating system. It is not an infrastructure provisioning tool. That is what Terraform is for. The expected workflow in production is to use Terraform to provision your VMs and external load balancer, then use Kubespray to deploy Kubernetes on those VMs.

However, Kubespray does provide one option for external load balancing that works without infrastructure provisioning: kube-vip. This is a software-based solution that creates a virtual IP address using either ARP (for layer 2 networks) or BGP (for layer 3 networks). If you enable kube-vip in Kubespray, it will deploy as a static pod on your control plane nodes and manage a floating VIP that always points to a healthy API server.

To enable kube-vip, you set these variables in your group_vars:

kube_vip_enabled: true
kube_vip_arp_enabled: true
kube_vip_address: 192.168.10.100

The kube_vip_address should be an unused IP in your network that will become the virtual IP for your API server endpoint.

Client-Side Load Balancing

For internal cluster communication, Kubespray uses a different approach: client-side load balancing. Instead of routing traffic through an external load balancer, each worker node runs its own local load balancer that distributes requests across all API servers.

Here is how it works. On each worker node, Kubespray deploys nginx as a lightweight TCP load balancer. This nginx listens on localhost:6443. The kubelet on that worker node is configured to connect to https://localhost:6443 as its API server endpoint. When kubelet makes a request, nginx receives it and forwards it to one of the actual API servers.

This might seem like an unnecessary extra hop, but it has significant advantages. First, there is no single point of failure in the load balancing layer. If the nginx on one worker node crashes, only that worker node is affected. Other worker nodes continue operating normally. Second, you do not need to provision or maintain any external infrastructure. Third, the latency added by the local nginx proxy is negligible since it is all localhost communication.

The nginx configuration that Kubespray generates looks something like this:

stream {
    upstream kube_apiserver {
        least_conn;
        server 192.168.10.11:6443 max_fails=3 fail_timeout=30s;
        server 192.168.10.12:6443 max_fails=3 fail_timeout=30s;
        server 192.168.10.13:6443 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 127.0.0.1:6443;
        proxy_pass kube_apiserver;
        proxy_timeout 10m;
        proxy_connect_timeout 1s;
    }
}

The upstream block lists all your API servers. The least_conn directive tells nginx to send new connections to the server with the fewest active connections, which provides good load distribution. The max_fails and fail_timeout parameters control health checking. If an API server fails three times within 30 seconds, nginx temporarily removes it from the pool.

The server block listens only on 127.0.0.1, meaning only processes on the local machine can connect. This is intentional. The nginx proxy is not meant to be accessed from other machines.

The kubelet configuration on worker nodes points to this local proxy:

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://localhost:6443
  name: cluster.local

When you look at this kubeconfig, you might wonder how the kubelet can connect to localhost:6443 and reach the API server. The answer is nginx. Every request to localhost:6443 goes through nginx, which forwards it to a real API server.

Kubespray calls this approach “localhost load balancing” or “nginx proxy.” You can choose between nginx and haproxy as the local proxy by setting the loadbalancer_apiserver_type variable:

loadbalancer_apiserver_type: nginx

or

loadbalancer_apiserver_type: haproxy

Both work well. nginx is the default and is lighter weight. haproxy provides more sophisticated load balancing options if you need them.

One important detail: control plane nodes do not need this local proxy. They can connect directly to their local API server at localhost:6443 because the API server is running on the same machine. Kubespray only deploys the nginx proxy on nodes that are in kube_node but not in kube_control_plane.

If you have nodes that are both control plane and worker nodes (which is common in smaller clusters), Kubespray handles this correctly. The node uses its local API server directly rather than going through nginx.

etcd High Availability

The API server is stateless, but etcd is not. etcd stores all cluster state: every pod definition, every service, every secret, every configmap. If you lose etcd data, you lose your cluster. This makes etcd HA critically important.

etcd uses the Raft consensus algorithm to replicate data across multiple nodes. I will not go into the full details of Raft here, but the key concept you need to understand is quorum. A quorum is the minimum number of nodes that must agree for a write to be committed. In Raft, the quorum is a majority: more than half of the total nodes.

For a 3-node etcd cluster, the quorum is 2. This means writes require at least 2 nodes to acknowledge them before they are considered committed. If one node is down, the remaining 2 nodes can still form a quorum and the cluster continues operating. But if 2 nodes are down, the single remaining node cannot form a quorum, and the cluster stops accepting writes.

For a 5-node etcd cluster, the quorum is 3. The cluster can tolerate 2 node failures.

This is why etcd clusters should always have an odd number of nodes. Consider what happens with 4 nodes. The quorum is 3 (more than half of 4). The cluster can tolerate only 1 failure, same as a 3-node cluster. But you have to maintain an extra node. You get no additional fault tolerance for the extra operational burden.

Here is the fault tolerance for different cluster sizes:

Nodes    Quorum    Tolerated Failures
1        1         0
3        2         1
5        3         2
7        4         3

Notice that going from 1 to 3 nodes gives you the ability to survive 1 failure. Going from 3 to 5 nodes lets you survive 2 failures. In practice, 3 nodes is sufficient for most production environments. You use 5 nodes when you need to perform rolling maintenance on etcd nodes while still tolerating a failure.

Never run 2 or 4 nodes. With 2 nodes, quorum is 2, so a single node failure makes the cluster unavailable. You would be better off with a single node at that point, since the operational complexity of 2 nodes buys you nothing.

In the inventory example I showed earlier, all three control plane nodes were also etcd nodes:

[etcd]
k8s-ctr1
k8s-ctr2
k8s-ctr3

This is stacked etcd, where etcd runs on the same machines as the control plane. The alternative is external etcd, where etcd runs on dedicated machines separate from the control plane.

Stacked etcd is simpler to set up and requires fewer machines. For a minimum HA setup with stacked etcd, you need 3 nodes (which are both control plane and etcd). With external etcd, you would need 3 control plane nodes plus 3 etcd nodes, totaling 6 machines.

External etcd has advantages in larger environments. etcd can be resource-intensive under heavy load, and isolating it on dedicated machines ensures that control plane components do not compete with etcd for CPU and memory. External etcd also allows you to scale and maintain the etcd cluster independently from the control plane.

Kubespray supports both configurations. For stacked etcd, you put the same nodes in both kube_control_plane and etcd groups. For external etcd, you put different nodes in each group:

[kube_control_plane]
k8s-ctr1
k8s-ctr2
k8s-ctr3

[etcd]
k8s-etcd1
k8s-etcd2
k8s-etcd3

With this configuration, Kubespray will deploy the control plane on k8s-ctr1 through k8s-ctr3, and etcd on k8s-etcd1 through k8s-etcd3. The API servers on the control plane nodes will be configured to connect to the external etcd cluster.

etcd Deployment Types in Kubespray

Kubespray supports two ways to run etcd, controlled by the etcd_deployment_type variable.

When etcd_deployment_type is set to “host”, etcd runs as a systemd service directly on the host. Kubespray downloads the etcd binary, generates all necessary certificates, creates the systemd unit file, and starts the service. This is the default in Kubespray and what I have been describing throughout this chapter.

The systemd unit file looks like this:

[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
After=network.target

[Service]
User=etcd
Type=notify
EnvironmentFile=/etc/etcd.env
ExecStart=/usr/local/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target