Cloud Controller Manager

11 October 2019

How Kubernetes Communicates with Underlying Cloud Infrastructure to manage Load Balancers, Nodes etc.

Copy Code
Loading...
Copy Code
Loading...

On both master and worker node, add the flag --cloud-provider=external to kubelet config file.

Copy Code
Loading...
  • Install the following components on both machines.(we are using the current latest version(v1.16) for this demo).
Create two DigitalOcean droplets, one master and one worker node.
  • 2 Virtual machines with Ubuntu 18.04 installed and sudo privileges.
  • 2 GB or more of RAM per machine.
  • 2 CPUs or more on all nodes.
  • Enable remote access on all nodes.
  • Enable Private networking on all VMs.

Requirements:

DigitalOcean cloud is currently not supported in Kubernetes core, so we are creating a CCM using DigitalOcean CCM Project. Details are here

Deploying CloudControllerManager for DigitalOcean Cloud

Architecture with the CCM

By introducing CCM, we are creating a single point of integration with cloud.

  • Kubelet
  • Kubernetes Controller Manager
  • Kubernetes API server

Without CCM, Kubernetes and the cloud provider are integrated through several different components.

The architecture of a Kubernetes cluster without the cloud controller manager

Once CCM (Cloud Controller Manager) is deployed, cloud-related information about nodes in the cluster will no longer be retrieved using local metadata, but instead, all API calls to retrieve node information will go through Cloud Controller Manager.

Kubelets interact with a cloud provider to collect details like region, zone on which that particular node is residing, etc. By adding a taint we are informing kube-apiserver that registering nodes are not yet complete and there are some more data to be added for completing the registration. This makes nodes not ready for scheduling pods. When CCM finds a node with the above taint it will process the nodes and remove taints after adding the required information.

Reason for taint:

Adding the parameter --cloud-provider=external will taint all nodes in a cluster with node.cloudprovider.kubernetes.io/uninitialized

For the successful implementation of CCM, all kubelets in the cluster MUST set the flag --cloud-provider=externalkube-apiserver and kube-controller-manager must NOT set the flag --cloud-provider, which will default them to use no cloud provider natively. By default, the parameter is not set.

--cloud-provider=external

Parameters for running CCM

High availability: like kube-controller-manager, you may want a high available setup for cloud controller manager using leader election (on by default).

Kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the --Kubernetes API server

Cloud authentication/authorization: your cloud may require a token or IAM rules to allow access to its APIs

As explained in Kubernetes official Documentation, the general requirement for cloud provider integration with Kubernetes core are:

Requirements:

Kubelet was responsible for initializing a node with cloud-specific details such as IP addresses, region/zone labels, and instance type information. The introduction of the CCM has moved this initialization operation from the Kubelet into the CCM. In the new model, Kubelet will initialize the nodes without cloud-specific data and will add a taint to nodes. This will make nodes unschedulable until CCM initialize the node with cloud-specific information

2. Kubelet

The latter will be maintained by cloud providers ie any cloud provider can develop their own CCM, the only point is that it must be satisfied by CPI. Thus Cloud-controller-manager allows cloud vendors and Kubernetes core to evolve independently of each other.

  • KubeControllerManager itself, which will hold all the control loops which don't interact with the cloud.
  • CCM, which will hold all control loops that interact with the cloud.

By implementing CCM we are splitting KubeControllerManager(KCM) into two:

These control loops are communicating with the cloud using a common interface which we mentioned earlier CloudProviderInterface.

This controller is applicable only for Google Compute Engine clusters and the responsibility is to configure routes in the cloud so that the pods in different nodes can communicate with each other.

Route controller

The Node controller is responsible for initializing a node by obtaining information about the nodes running in the cluster from the cloud provider. The node controller performs the following functions:
- Initialize a node with cloud-specific zone/region labels.
- Initialize a node with cloud-specific instance details, for example, type and size
- Obtain the node’s network addresses and hostname.
- If the node is deleted from the cloud, delete the node object from Kubernetes

Node Controller

Responsible for service-related events. This controller will listen to events like create, delete, update related to service and configures cloud load balancers to reflect the state of services in the Kubernetes.

Service Controller

  • Service controller
  • Node Controller
  • Route Controller

The core control loops of the Kubernetes are embedded together in a daemon known as Kubernetes Controller Manager. Among these, the control loops that integrate with a cloud are the following:

1. Kubernetes controller manager

CCM inherits its functionality from two cloud dependent components of Kubernetes:-

Functions Of Cloud Controller Manager (CCM)

It is not an easy task to develop Kubernetes core to make it integrate with any cloud platform it is going to run on. Moreover, it will not be a best practice since the development of the Kubernetes project and cloud platform are at a different pace. To overcome these real-world issues, a daemon called Cloud Controller Manager CCM is introduced that embeds cloud-specific control loops. CCM can be linked to any cloud provider that satisfies CloudProviderInterface (CPI)

When we are running a Kubernetes cluster in a cloud platform, an important point which can come into consideration is how Kubernetes is going to integrate with the cloud provider. This is important because Kubernetes need to get information about the nodes running in the cluster, provision and configure load balancers, persistent volume, etc. Even though all cloud platforms have some common features there will be differences in the way they are implemented for operations.

Introduction: