Praveen V

Praveen V

2019-10-11 12:42:20 UTC

How Kubernetes Communicates with Underlying Cloud Infrastructure to manage Load Balancers, Nodes etc.

Share:

Cloud Controller Manager

Cloud Controller Manager

CLOUD CONTROLLER MANAGER

Introduction:

When we are running a Kubernetes cluster in a cloud platform, an important point which can come into consideration is how Kubernetes is going to integrate with the cloud provider. This is important because Kubernetes need to get information about the nodes running in the cluster, provision and configure load balancers, persistent volume, etc. Even though all cloud platforms have some common features there will be differences in the way they are implemented for operations.

It is not an easy task to develop Kubernetes core to make it integrate with any cloud platform it is going to run on. Moreover, it will not be a best practice since the development of the Kubernetes project and cloud platform are at a different pace. To overcome these real-world issues, a daemon called Cloud Controller Manager CCM is introduced that embeds cloud-specific control loops. CCM can be linked to any cloud provider that satisfies CloudProviderInterface (CPI)

Functions Of Cloud Controller Manager (CCM)

CCM inherits its functionality from two cloud dependent components of Kubernetes:-

1. Kubernetes controller manager

The core control loops of the Kubernetes are embedded together in a daemon known as Kubernetes Controller Manager. Among these, the control loops that integrate with a cloud are the following:

  • Service controller
  • Node Controller
  • Route Controller

Service Controller

Responsible for service related events. This controller will listen to events like create, delete, update related to service and configures cloud load balancers to reflect the state of services in the Kubernetes.

Node Controller

The Node controller is responsible for initializing a node by obtaining information about the nodes running in the cluster from the cloud provider. The node controller performs the following functions:
- Initialize a node with cloud-specific zone/region labels.
- Initialize a node with cloud-specific instance details, for example, type and size
- Obtain the node’s network addresses and hostname.
- If the node is deleted from the cloud, delete the node object from Kubernetes

Route controller

This controller is applicable only for Google Compute Engine clusters and the responsibility is to configure routes in the cloud so that the pods in different nodes can communicate with each other.

These control loops are communicating with cloud using a common interface which we mentioned earlier CloudProviderInterface.

By implementing CCM we are splitting KubeControllerManager(KCM) into two:

  • KubeControllerManager itself, which will hold all the control loops which don't interact with the cloud.
  • CCM, which will hold all control loops that interact with the cloud.

The latter will be maintained by cloud providers ie any cloud provider can develop their own CCM, the only point is that it must be satisfied by CPI. Thus Cloud-controller-manager allows cloud vendors and Kubernetes core to evolve independent of each other.

2. Kubelet

Kubelet was responsible for initializing a node with cloud-specific details such as IP addresses, region/zone labels, and instance type information. The introduction of the CCM has moved this initialization operation from the Kubelet into the CCM. In the new model, Kubelet will initialize the nodes without cloud-specific data and will add a taint to nodes. This will make nodes unschedulable until CCM initialize the node with cloud-specific information

Requirements:

As explained in Kubernetes official Documentation, general requirement for cloud provider integration with Kubernetes core are:

Cloud authentication/authorization: your cloud may require a token or IAM rules to allow access to their APIs

Kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the --Kubernetes apiserver

High availability: like kube-controller-manager, you may want a high available setup for cloud controller manager using leader election (on by default).

Parameters for running CCM

--cloud-provider=external

For the successful implementation of CCM, all kubelets in the cluster MUST set the flag --cloud-provider=external. kube-apiserver and kube-controller-manager must NOT set the flag --cloud-provider, which will default them to use no cloud provider natively. By default, the parameter is not set.

Adding the parameter --cloud-provider=external will taint all nodes in a cluster with node.cloudprovider.kubernetes.io/uninitialized

Reason for taint:

Kubelets interact with cloud provider to collect details like region, zone on which that particular node is residing, etc. By adding a taint we are informing kube-apiserver that registering nodes are not yet complete and there are some more data to be added for completing the registration. This makes nodes not ready for scheduling pods. When CCM finds a node with the above taint it will process the nodes and remove taints after adding the required information.

Once CCM (Cloud Controller Manager) is deployed, cloud-related information about nodes in the cluster will no longer be retrieved using local metadata, but instead, all API calls to retrieve node information will go through Cloud Controller Manager.

Architecture of a Kubernetes cluster without the cloud controller manager


.

Without CCM, Kubernetes and the cloud provider are integrated through several different components.

  • Kubelet
  • Kubernetes Controller Manager
  • Kubernetes API server

By introducing CCM, we are creating a single point of integration with cloud.

Architecture with the CCM

Deploying CloudControllerManager for DigitalOcean Cloud

DigitalOcean cloud is currently not supported in Kubernetes core, so we are creating a CCM using DigitalOcean CCM Project. Details are here

Requirements:

  • 2 Virtual machines with Ubuntu 18.04 installed and sudo privileges.
  • 2 GB or more of RAM per machine.
  • 2 CPUs or more on all nodes.
  • Enable remote access on all nodes.
  • Enable Private networking on all VMs.
Create two DigitalOcean droplets, one master and one worker node.
  • Install the following components on both machines.(we are using the current latest version(v1.16) for this demo).
sudo apt-get update && sudo apt-get install -y apt-transport-https

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubectl kubelet kubeadm docker.io
  • On both master and worker node, add the flag --cloud-provider=external to kubelet config file.
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
------
Environment="KUBELET_KUBECONFIG_ARGS=--cloud-provider=external --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
--------
Setup Control Plane

Run the following command on the Control plane node.

 kubeadm init --pod-network-cidr=192.168.0.0/16
  • Join the worker node to cluster by using the token generated in master node, also copy kubeconfig file to location
Setup Kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Install the pod network addon using this detail

Here we selected calico pod so below command is used to install addons.

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
Setup Worker node:

Copy the kubeadm join command from the output of kubeadm init and run on the worker nodes to join them to the cluster.

Check the nodes
kubectl get nodes
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   160m   v1.16.0
worker   Ready    <none>   158m   v1.16.0
  • If we check the nodes, we can find a taint is added
kubectl get node master -o yaml
-----------------------
 taints:
  - effect: NoSchedule
    key: node.cloudprovider.kubernetes.io/uninitialized
    value: "true"
 ------------
  • At this point, some pods in namespace kube-system will be in pending state, which is waiting for CCM to be up and running.
kubectl get pod -n kube-system
NAME                                                     READY   STATUS    RESTARTS   AGE
calico-kube-controllers-564b6667d7-sjmmf                 0/1     Pending   0          6m20s
calico-node-68lrs                                        1/1     Running   0          6m20s
calico-node-sbjwf                                        1/1     Running   0          6m20s
coredns-5644d7b6d9-b8kjb                                 0/1     Pending   0          9m32s
coredns-5644d7b6d9-r82vc                                 0/1     Pending   0          9m32s
digitalocean-cloud-controller-manager-7fcdd94759-2ztkp   1/1     Running   0          76s
etcd-master                                              1/1     Running   0          8m27s
kube-apiserver-master                                    1/1     Running   0          8m36s
kube-controller-manager-master                           1/1     Running   0          8m53s
kube-proxy-fsvnb                                         1/1     Running   0          7m20s
kube-proxy-p5cdl                                         1/1     Running   0          9m32s
kube-scheduler-master                                    1/1     Running   0          8m45s

DigitalOcean Token

  • To run DigitalOcean-cloud-controller-manager, we need a DigitalOcean personal access token.
  • If you are already logged in, you can create one here
  • Ensure the token you create has both read and write access.
  • Create a Kubernetes Secret as a way for the cloud controller manager to access your token.
Secret.yaml will look like below.You can find template here
apiVersion: v1
kind: Secret
metadata:
  name: digitalocean
  namespace: kube-system
stringData:
  access-token: "**************"
  • Create the Secret
kubectl apply -f secret.yml
  • Check if the secret is created
kubectl -n kube-system get secrets | grep digitalocean
digitalocean                                     Opaque                                1      3h53m

Deploy Cloud Controller Manager

The latest version of CCM currently available in DO repository is v0.1.20. Before deploying we need to make some changes since the APIversion for deployment mentioned in this file is deprecated in current version of Kubernetes. The edited file will look like below:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: digitalocean-cloud-controller-manager
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: digitalocean-cloud-controller-manager
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: digitalocean-cloud-controller-manager
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      dnsPolicy: Default
      hostNetwork: true
      serviceAccountName: cloud-controller-manager
      tolerations:
        # this taint is set by all kubelets running `--cloud-provider=external`
        # so we should tolerate it to schedule the digitalocean ccm
        - key: "node.cloudprovider.kubernetes.io/uninitialized"
          value: "true"
          effect: "NoSchedule"
        - key: "CriticalAddonsOnly"
          operator: "Exists"
        # cloud controller manages should be able to run on masters
        - key: "node-role.kubernetes.io/master"
          effect: NoSchedule
      containers:
      - image: digitalocean/digitalocean-cloud-controller-manager:v0.1.20
        name: digitalocean-cloud-controller-manager
        command:
          - "/bin/digitalocean-cloud-controller-manager"
          - "--leader-elect=false"
        resources:
          requests:
            cpu: 100m
            memory: 50Mi
        env:
          - name: DO_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: digitalocean
                key: access-token

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  name: system:cloud-controller-manager
rules:
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - update
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services/status
  verbs:
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - get
  - list
  - watch
  - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:cloud-controller-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
- kind: ServiceAccount
  name: cloud-controller-manager
  namespace: kube-system
kubectl apply -f v0.1.20.yml

Verify the CCM implementation

Once CCM is deployed :

  • Check the Pod status
 root@master:~# kubectl get pod -n kube-system
NAME                                                     READY   STATUS    RESTARTS   AGE
calico-kube-controllers-564b6667d7-sjmmf                 1/1     Running   0          4h3m
calico-node-68lrs                                        1/1     Running   0          4h3m
calico-node-sbjwf                                        1/1     Running   0          4h3m
coredns-5644d7b6d9-b8kjb                                 1/1     Running   0          4h6m
coredns-5644d7b6d9-r82vc                                 1/1     Running   0          4h6m
digitalocean-cloud-controller-manager-7fcdd94759-2ztkp   1/1     Running   0          3h57m
etcd-master                                              1/1     Running   0          4h5m
kube-apiserver-master                                    1/1     Running   0          4h5m
kube-controller-manager-master                           1/1     Running   0          4h5m
kube-proxy-fsvnb                                         1/1     Running   0          4h4m
kube-proxy-p5cdl                                         1/1     Running   0          4h6m
kube-scheduler-master                                    1/1     Running   0          4h5m
  • Check if the taint node.cloudprovider.kubernetes.io/uninitialized is removed from the nodes.

  • Check if new labels are added

labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: s-2vcpu-2gb
    beta.kubernetes.io/os: linux
    failure-domain.beta.kubernetes.io/region: nyc3
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: master
    kubernetes.io/os: linux
    node-role.kubernetes.io/master: ""
  • Create a deployment and service of type load balancer
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-ctr
        image: nginx:1.15.4
        ports:
        - containerPort: 80
apiVersion: v1
kind: Service
metadata:
  name: nginx-lb
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: nginx
  • Verify Load Balancer
kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP        19m
nginx-lb     LoadBalancer   10.101.27.197   139.59.50.243   80:30021/TCP   5m29s
$ curl -s http://139.59.50.243/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Limitations:

Before using CCM in a production environment you must be aware of a few limitations :

  • Support Volumes:
    CCM does not implement volume controllers found in Kube-Controller-Manager, as the volume integrations also require coordination with kubelets as well. For more details, please check the k8s documentation.

  • Scalability:
    As per the new architecture, we fully rely on the cloud controller managers to retrieve information for all nodes. Previously node information was retrieved bykubelet using local metadata service. For larger clusters, there may be possible bottlenecks such as resource requirements and API rate limiting.

Reference:

Cloud Controller Manager

Digital Ocean CCM

Subscribe to Newsletter

Get updates on latest courses and tools.