TL;DR: Inspired by Kelsey Hightower’s kubernetes-the-hard-way, I am here putting the steps I did to setup K8s in my laptop with VirtualBox VMs, instead of Google Cloud resources(saving some bucks here).

Jump to part 2

Setup overview we will accomplish by the end of the post:

The IP addresses and Network requirements:

We will be dealing basically with 4 networks:

  1. 10.240.0.0/24 – VirtualBox Host Network, which will be used for interconnecting between the VMs. You may watch https://www.youtube.com/watch?v=m0ZWtcJT7fc for more info on how to set the host-only network in VirtualBox.
  2. 10.200.0.0/16 – POD CIDR that will be used to assign POD IP addresses
  3. 10.32.0.0/24 – Cluster Network, which will be used by K8s for all its internal components like service endpoints
  4. 192.168.33.0/24 – VirtualBox NAT network, which will be used for connecting to the Internet(eg: to install a package)

Therefore, the addressing in the cluster would be:

VMIP address
kube-master-01 10.240.0.100
kube-master-0210.240.0.101
kube-worker-0110.240.0.200
kube-worker-0210.240.0.201
LB-server10.240.0.50
POD CIDR subnet10.200.1.0/16 – for kube-worker-01
10.200.2.0/16 – for kube-worker-02
Cluster CIDR & Service cluster IP range 10.32.0.0/24

The diagram now looks something like this:

In case you are new to Kubernetes, I would suggest you read this awesome post by Nicolas Leiva: Kubernetes Networking: Behind the scenes

Extra: my network interface config

This might help you to compare your network setup, here is a copy of /etc/network/interfaces file:

 
source /etc/network/interfaces.d/* 

# The loopback network interface 
auto lo 
iface lo inet loopback 

# NAT Network 
auto enp0s3 
iface enp0s3 inet static 
  address 192.168.33.100 
  gateway 192.168.33.1 
  dns-nameservers 8.8.8.8 

# HOST-ONLY network 
auto enp0s8 
iface enp0s8 inet static 
  address 10.240.0.100 

Step 1: Installing the Client Tools

In this step, we will be installing tools like cfsslcfssljson, and kubectl. This can be done from any machine(mostly your workstation or laptop).

1.1 Install cfssl and cfssljson in Mac:

curl -o cfssl https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/darwin/cfssl

curl -o cfssljson https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/darwin/cfssljson
chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/local/bin/

1.2 Install cfssl and cfssljson in Linux:

wget -q --show-progress --https-only --timestamping \
  https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssl \
  https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssljson
chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/local/bin/

1.3 Verification cfssl and cfssljson:

cfssl version
cfssljson --version
cfssl version

Version: 1.3.4
Revision: dev
Runtime: go1.13
---
---
cfssljson --version

Version: 1.3.4
Revision: dev
Runtime: go1.13

1.4 Install kubectl in Mac:

curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/darwin/amd64/kubectl
chmod +x kubectl; sudo mv kubectl /usr/local/bin/

1.5 Install kubectl in Linux:

wget https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl
chmod +x kubectl; sudo mv kubectl /usr/local/bin/

1.6 Verification of kubectl:

kubectl version --client
kubectl version --client

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Step 2: Generate the CA configuration file, certificate, and private key(Certificate Authority)

Make a directory called ~/k8s-requirements/

mkdir -p ~/k8s-requirements/

cd ~/k8s-requirements/

2.1 Generate the CA configuration file, certificate, and private key(Certificate Authority).

To do copy and paste the below in your terminal/shell:

{
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Oregon"
    }
  ]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
}

Make sure you can see the below files are created:

ca-key.pem
ca.pem

2.2 Client and server certificates for each Kubernetes component and a client certificate for the Kubernetes admin user:

{
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:masters",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin
}

Make sure you can see the below files are created:

admin-key.pem
admin.pem

2.3 Generate a certificate and private key for each Kubernetes worker node

for instance in kube-worker-01 kube-worker-02; do
cat > ${instance}-csr.json <<EOF
{
  "CN": "system:node:${instance}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:nodes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

INTERNAL_IP_kube-worker-01=10.240.0.200
INTERNAL_IP_kube-worker-02=10.240.0.201

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${instance},${INTERNAL_IP}_${instance} \
  -profile=kubernetes \
  ${instance}-csr.json | cfssljson -bare ${instance}
done

Make sure you can see the below files are created:

kube-worker-01-key.pem
kube-worker-02-key.pem
kube-worker-01.pem
kube-worker-02.pem

2.4 Generate the kube-controller-manager client certificate and private key:

{
cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
}

Make sure you can see the below files are created:

kube-controller-manager.pem
kube-controller-manager-key.pem

2.5 Generate the kube-proxy client certificate and private key:

{

cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:node-proxier",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy

}

Make sure you can see the below files are created:

kube-proxy-key.pem
kube-proxy.pem

2.6 Generate the kube-scheduler client certificate and private key:

{

cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler

}

Make sure you can see the below files are created:

kube-scheduler-key.pem
kube-scheduler.pem

2.7 Kubernetes API Server Certificate:

Ensure to include the IP addresses of your nodes so that they can validate the certs by the remote clients. As you may notice, I have added few extra IP addresses just incase we need in future to add more nodes as master or worker.

{

IP_PRIVATE_ADDRESSES=10.32.0.1,10.240.0.50,10.240.0.100,10.240.0.101,10.240.0.102,10.240.0.200,10.240.0.201,10.240.0.202,10.240.0.203,10.240.0.204,10.240.0.205
KUBERNETES_HOSTNAMES=kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.svc.cluster.local

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${IP_PRIVATE_ADDRESSES},127.0.0.1,${KUBERNETES_HOSTNAMES} \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes

}

Make sure you can see the below files are created:

kubernetes.pem
kubernetes-key.pem

2.8 Generate the service-account certificate and private key for Kubernetes Controller Manager

{

cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

}

Make sure you can see the below files are created:

service-account-key.pem
service-account.pem

2.9 Transfer the files to respective servers:

Ensure that you change the username in the below commands and that you already create a directory named ‘~/k8s-requirements/’ in home directory of respective nodes, as mentioned in the earlier section.

#worker nodes:
scp kube-worker-01.pem kube-worker-01-key.pem nandan@10.240.0.200:~/k8s-requirements/
scp kube-worker-02.pem kube-worker-02-key.pem nandan@10.240.0.201:~/k8s-requirements/

#controller/master nodes:
scp service-account-key.pem service-account.pem kubernetes.pem kubernetes-key.pem ca-key.pem ca.pem nandan@10.240.0.100:~/k8s-requirements/

scp service-account-key.pem service-account.pem kubernetes.pem kubernetes-key.pem ca-key.pem ca.pem nandan@10.240.0.101:~/k8s-requirements/

I can understand that it is becoming a lengthy blog post, and getting super confusing. But hold on with me, and try to understand that we are building the K8s cluster from scratch. If you don’t understand any step, just have patience and follow along. I promise everything will be clear if you do hands-on. It took me 3 times during my initial days, to understand all the components.

We will continue to the 2nd part.

2 thoughts on “Setup highly available Kubernetes Cluster, the hard way in VirtualBox VMs MAC/Linux – part 1”

Leave a Reply

Your email address will not be published. Required fields are marked *