This is an extension of my previous post: Learn Ansible and Terraform for Free: Provision Self-Hosted Virtual Machines with Custom Disk Images on Linux.
The Terraform configuration has been extended to provision 3 virtual machines, 1 control plane and 2 worker nodes. The Ansible playbook prepares each machine with the prerequisites needed to initialize the cluster with kubeadm and join the worker nodes.
Video guideStep 1 - Provision and Configure with Terraform and Ansible
After running terraform apply, 3 VMs are provisioned and the Ansible inventory is automatically written:
Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs:
vm_ips = {
"master" = "192.168.122.193"
"worker1" = "192.168.122.39"
"worker2" = "192.168.122.79"
}matthew@matthew-ThinkPad-E14-Gen-6:~/infra-lab/terraform$ cat ../ansible/inventory.ini
[master]
master ansible_host=192.168.122.193 ansible_user=matt ansible_python_interpreter=/usr/bin/python3
[workers]
worker1 ansible_host=192.168.122.39 ansible_user=matt ansible_python_interpreter=/usr/bin/python3
worker2 ansible_host=192.168.122.79 ansible_user=matt ansible_python_interpreter=/usr/bin/python3
[lab:children]
master
workersRun the Ansible playbook to configure all 3 nodes. It installs containerd, kubelet, kubeadm and kubectl, sets the required kernel parameters, disables swap, and populates /etc/hosts so all nodes can resolve each other by hostname:
matthew@matthew-ThinkPad-E14-Gen-6:~/infra-lab/ansible$ ansible-playbook -i inventory.ini playbook.yml
PLAY RECAP *********************************************************************
master : ok=21 changed=17 unreachable=0 failed=0 skipped=0
worker1 : ok=21 changed=17 unreachable=0 failed=0 skipped=0
worker2 : ok=21 changed=17 unreachable=0 failed=0 skipped=0Step 2 - Initialize the Control Plane
SSH onto the master node and run kubeadm init. We specify the Flannel pod network CIDR and the control plane endpoint hostname:
matthew@matthew-ThinkPad-E14-Gen-6:~$ ssh matt@192.168.122.193
matt@master:~$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=masterOn success:
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You can join any number of worker nodes by running the following on each as root:
kubeadm join master:6443 --token ib2x9p.dn0gozchs1covuhp \
--discovery-token-ca-cert-hash sha256:e1cd5dc4ac21ea516370f...Save the join command. If you lose it, regenerate it with kubeadm token create --print-join-command on the master.
Set up kubectl for your non-root user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configStep 3 - Install the CNI Plugin
Kubernetes delegates pod networking to CNI (Container Network Interface) plugins, which assign IP addresses to pods and enable cross-node communication. Install Flannel:
matt@master:~$ kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds createdVerify CoreDNS pods are running:
matt@master:~$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-589f44dc88-dzfql 1/1 Running 0 6m21s
coredns-589f44dc88-fr6p8 1/1 Running 0 6m21s
etcd-master 1/1 Running 0 6m28s
kube-apiserver-master 1/1 Running 0 6m28s
kube-controller-manager-master 1/1 Running 0 6m28s
kube-proxy-45lks 1/1 Running 0 6m21s
kube-scheduler-master 1/1 Running 0 6m28sStep 4 - Join the Worker Nodes
SSH onto each worker node and run the join command from Step 2. Use the version without --control-plane, that flag is only for adding additional control plane nodes in HA setups:
sudo kubeadm join master:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>[preflight] Running pre-flight checks
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[kubelet-check] The kubelet is healthy after 1.00176137s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.Confirm the complete cluster on the control plane:
matt@master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 14m v1.36.0
worker1 Ready <none> 6m46s v1.36.0
worker2 Ready <none> 5m15s v1.36.0Step 5 - Deploy a Test Workload
Create an nginx deployment and check which node it was scheduled on:
matt@master:~$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
matt@master:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-7f8fbb96d-cpvlv 1/1 Running 0 50s 10.244.2.2 worker2 <none> <none>Expose via NodePort and access from the host:
matt@master:~$ kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
matt@master:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17m
nginx NodePort 10.111.237.212 <none> 80:31760/TCP 15smatthew@matthew-ThinkPad-E14-Gen-6:~$ curl 192.168.122.79:31760
<!DOCTYPE html>
...
<h1>Welcome to nginx!</h1>
<p>If you see this page, nginx is successfully installed and working.</p>The pod is running on worker2, but the same NodePort works on any node in the cluster. The request is routed correctly regardless of which node you hit.
Extension: Persistent Storage
By default the cluster has no StorageClass, so PersistentVolumeClaims stay Pending. Install Rancher's local-path-provisioner to enable dynamic provisioning using node-local disk:
matt@master:~$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.36/deploy/local-path-storage.yaml
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
matt@master:~$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
matt@master:~$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 38sDemonstrate with a busybox pod that writes to a PVC:
# pvc-test.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: test
image: busybox
command: ["/bin/sh", "-c", "echo 'hello from persistent storage' > /data/test.txt && sleep 3600"]
volumeMounts:
- mountPath: /data
name: test-volume
volumes:
- name: test-volume
persistentVolumeClaim:
claimName: test-pvcmatt@master:~$ kubectl apply -f pvc-test.yaml
matt@master:~$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-pvc Bound pvc-52f23c03-6174-4eb1-bc29-1b8764853c19 1Gi RWO local-path 33s
matt@master:~$ kubectl exec test-pod -- cat /data/test.txt
hello from persistent storageExtension: Load Balancer with MetalLB
Without a cloud provider, LoadBalancer resources are not supported natively. MetalLB assigns real IPs from a pool on your local network, allowing access from the host without specifying a port number.
matt@master:~$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.3/config/manifests/metallb-native.yaml
matt@master:~$ kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-f59dc4bc7-9fvcs 1/1 Running 0 55s
speaker-585cf 1/1 Running 0 55s
speaker-8z8md 1/1 Running 0 55s
speaker-sklv7 1/1 Running 0 55sConfigure an IP pool from a range that won't conflict with your DHCP leases:
# metallb-config.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default-pool
namespace: metallb-system
spec:
addresses:
- 192.168.122.200-192.168.122.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-systemmatt@master:~$ kubectl apply -f metallb-config.yaml
matt@master:~$ kubectl expose deployment nginx --port=80 --type=LoadBalancer --name=nginx-lb
service/nginx-lb exposed
matt@master:~$ kubectl get svc nginx-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-lb LoadBalancer 10.103.212.226 192.168.122.200 80:32017/TCP 10sAn external IP is assigned from the pool. Access it directly on port 80 from the host:
matthew@matthew-ThinkPad-E14-Gen-6:~$ curl 192.168.122.200
<!DOCTYPE html>
...
<h1>Welcome to nginx!</h1>
<p>If you see this page, nginx is successfully installed and working.</p>You now have a self-hosted Kubernetes cluster with persistent storage and load balancing running on your laptop. If you want to go further, look into building a high-availability setup with 3 control plane nodes behind HAProxy using stacked etcd.
Thanks for reading!