Creating a custom operator from scratch in kuberentes can be tedious and involves dealing with Kubernetes API to create, watch, update objects etc. It involves a steep learning curve with lots of complexity to handle. Many tools and SDKs help automate tasks with the help of libraries like client-go and controller runtime. And one of the most popular tools is kubebuilder.

What is Kubebuilder?

Kubebuilder is an SDK for building Kubernetes APIs using operators and Custom Resource Definitions (CRDs). It helps in scaffolding project which uses controller runtime and controller tools as a subproject for creating custom controller, APIs, and CRD manifests. It is an open-source project written in GO and maintained by the kubernetes-sig-api-machinery group.

It helps generate a basic project structure with boilerplate code in an organized manner with Makefile. The Makefile has certain targets for generating CRDs, building, installing, and deploying operators in clusters with relevant RBACs permissions, and so on.

Kubebuilder Installation

Before installing Kubebuilder, make sure to install the following things as a prerequisite.

Prerequisites

• Install go and verify its installation

wget https://go.dev/dl/go1.20.4.linux-amd64.tar.gz

tar -C /usr/local -xvf go1.20.4.linux-amd64.tar.gz

echo "export PATH=$PATH:/usr/local/go/bin " >> ~/.bashrc
source ~/.bashrc

go version

• Verify docker installation

docker version

• Access to a kubernetes cluster

kubectl get nodes -o wide

• Now install kubebuilder binaries

curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)

chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

kubebuilder version

NOTE: Kubebuilder also need kustomize tool which will be installed with the help of makefile

Creating a custom resource and controller

In the following lab demo, we will create a custom resource that will be watched continuously by the custom controller and reconcile method and create an nginx pod as a result.

Scaffold Project and Bootstrap Operator

• Create an empty project directory

mkdir project && cd project

• Initialize the project directory and bootstrap our custom controller project

kubebuilder init --domain=demo.kcd.io --repo project

The domain is basically part of a group which helps in avoiding naming conflicts. For example, storage.k8s.io/v1 (Storage classes) has the domain k8s.io in it. Also, repo is used to define when a project is being built out of the GO path.

This will create generic folders and files with boilerplate code and basic dependencies.

• Create the api for CRDs, custom resource and custom controllers

kubebuilder create api --group demo --version v1 --kind DemoResource

Press y to create custom resources and controllers. This will create some more folders and files. Observe that it should have created a internal/controller folder by now. The above command will help us to create a custom resource of kind DemoResource and api group demo.demo.kcd.io and v1 as its api version.

These two commands have created the basic project structure for us to write our own custom operator with required CRD , RBAC manifests , Makefile and reconcile method boilerplate code.

Inspecting Project Structure

The Kubebuilder project has the following structure with folders and files

• api/ : This folder contains all information about the custom resource(DemoResource). It has demoresource_types.go, which contains api definition. In this file, we will define object spec and object status for reconcile loop. This will help Kubebuilder to generate CRD manifests based on the definition.
• bin/ : It contains some binaries like kustomize,controller-gen, which will help install CRDs and operators in the cluster.
• config/ : This folder contains all The YAML manifests related to CRDs, RBAC, sample custom resources etc.
• internal/controller/ : This folder contains the logic for the controller. In this, one can write the code for the reconciliation loop for a custom operator, which can be later attached to the “manager” wrapper.
• /cmd/main.go : This file is going to be starting point for the operator and contains metrics-server logic by controller-runtime to collect metrics.
• Makefile: This file contains all the make targets which will help generate CRD, install it in the cluster, and deploy the operator as well.
• Dockerfile: This file contains information to build a docker image of the manager and our custom operator, which can be used in the cluster.

NOTE:Install GO and YAML extensions for better understanding of code.

Building Custom Controller

For this demo, we will be going to build an operator which will have a custom resource DemoResource with name and namespace as its spec fields.The api definition will be defined inside /api/v1/demoresource_types.go file.

type DemoResourceSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of DemoResource. Edit demoresource_types.go to remove/update
	Name      string `json:"name,omitempty"`
	Namespace string `json:"namespace,omitempty"`
}

// DemoResourceStatus defines the observed state of DemoResource
type DemoResourceStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	Name string `json:"name,omitempty"`

}

We will use the name field to maintain the current and desired state.

Now, define the reconcile loop logic in the internal/controller/demoresource_controller.go file. This controller will watch the custom resource if its current and desired state doesn’t match. It will update it. And create a pod with the same custom resource name.

/*
Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controller

import (
	"context"

	"github.com/go-logr/logr"
	v1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"


	demov1 "project/api/v1"
)

// DemoResourceReconciler reconciles a DemoResource object
type DemoResourceReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=demo.demo.kcd.io,resources=demoresources,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=demo.demo.kcd.io,resources=demoresources/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=demo.demo.kcd.io,resources=demoresources/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the DemoResource object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile
func (r *DemoResourceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	l := log.FromContext(ctx)
	l.Info("Enter Reconcile", "req", req)

	testres := &demov1.DemoResource{}
	r.Get(ctx, types.NamespacedName{Name: req.Name, Namespace: req.Namespace}, testres)
	l.Info("Enter Reconcile", "spec", testres.Spec, "status", testres.Status)

	if testres.Spec.Name != testres.Status.Name {
		testres.Status.Name = testres.Spec.Name
		r.Status().Update(ctx, testres)
	}

	r.reconcilePOD(ctx, testres, l)

	return ctrl.Result{}, nil
}
func (r *DemoResourceReconciler) reconcilePOD(ctx context.Context, testres *demov1.DemoResource, l logr.Logger) error {
	name, namespace := testres.Name, testres.Namespace

	pod := &v1.Pod{
		ObjectMeta: metav1.ObjectMeta{
			Name:      name,
			Namespace: namespace,
		},
		Spec: v1.PodSpec{
			RestartPolicy: v1.RestartPolicyNever,
			Containers: []v1.Container{
				v1.Container{
					Name:  "nginx",
					Image: "nginx:latest",
				},
			},
		},
	}

	l.Info("Creating POD")
	return r.Create(ctx, pod)

}

// SetupWithManager sets up the controller with the Manager.
func (r *DemoResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&demov1.DemoResource{}).
		Complete(r)
}

Edit the main.go and change the metrics server address to 30000 or any other port if port 8080 is already occupied.

Test and Deploy Custom Operator

• Now, generate the CRDs and install them in the kubernetes cluster

make manifests

make install

• Verify that the custom resources definitions and its api resources are installed in the cluster

kubectl get crds | grep demo

kubectl api-resources | grep demo

• To create the custom resource, edit the /root/project/config/samples/demo_v1_demoresource.yaml file and change the spec field according to our API definition.

apiVersion: demo.demo.kcd.io/v1
kind: DemoResource
metadata:
  name: demoresource-sample
spec:
  # TODO(user): Add fields here
  name: app
  namespace: default

• Now run the operator locally to test it

make run

• Apply the custom resource and wait for the pod creation

kubectl apply -f /root/project/config/samples/demo_v1_demoresource.yaml

kubectl get demoresource

kubectl get pods

Observe the make run command output as well.

• Build the docker image of the controller and push it to the docker registry. Make sure to login into the docker registry.

make docker-build docker-push IMG=oshi36/controller:latest

• Deploy the docker image in the cluster

make deploy IMG=oshi36/controller:latest

References

• CNCF Operator Whitepaper
• Kubebuilder docs
• Kubecon Talk
• Controller-runtime