Kubernetes Operator series 1 — controller-runtime example controller

Introduction

If you start developing a Kubernetes operator, you’d start with kubebuilder or operator-sdk, both of which are tools to help you develop a Kubernetes operator.

This is the first episode of a series about controller-runtime, which is a tool to make it easy to develop a custom controller, used by kubebuilder and operator-sdk.

Getting Started

As you can see in the go doc of controller-runtime, we can start with the example controller.

The example controller monitors ReplicaSet and its Pods and if the controller receives create/update/delete event for them, it executes the reconciliation logic. Specifically, count the number of Pods for the ReplicaSet that is being reconciled and update the label pod-count with the value.

package main

import (
 "context"
 "fmt"
 "os"

 appsv1 "k8s.io/api/apps/v1"
 corev1 "k8s.io/api/core/v1"

 ctrl "sigs.k8s.io/controller-runtime"
 "sigs.k8s.io/controller-runtime/pkg/client"

 // since we invoke tests with -ginkgo.junit-report we need to import ginkgo.
 _ "github.com/onsi/ginkgo/v2"
)

func main() {
 var log = ctrl.Log.WithName("builder-examples")

 manager, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{})
 if err != nil {
  log.Error(err, "could not create manager")
  os.Exit(1)
 }

 err = ctrl.
  NewControllerManagedBy(manager). // Create the Controller
  For(&appsv1.ReplicaSet{}).       // ReplicaSet is the Application API
  Owns(&corev1.Pod{}).             // ReplicaSet owns Pods created by it
  Complete(&ReplicaSetReconciler{Client: manager.GetClient()})
 if err != nil {
  log.Error(err, "could not create controller")
  os.Exit(1)
 }

 if err := manager.Start(ctrl.SetupSignalHandler()); err != nil {
  log.Error(err, "could not start manager")
  os.Exit(1)
 }
}

// ReplicaSetReconciler is a simple Controller example implementation.
type ReplicaSetReconciler struct {
 client.Client
}

// Implement the business logic:
// This function will be called when there is a change to a ReplicaSet or a Pod with an OwnerReference
// to a ReplicaSet.
//
// * Read the ReplicaSet
// * Read the Pods
// * Set a Label on the ReplicaSet with the Pod count.
func (a *ReplicaSetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

 rs := &appsv1.ReplicaSet{}
 err := a.Get(ctx, req.NamespacedName, rs)
 if err != nil {
  return ctrl.Result{}, err
 }

 pods := &corev1.PodList{}
 err = a.List(ctx, pods, client.InNamespace(req.Namespace), client.MatchingLabels(rs.Spec.Template.Labels))
 if err != nil {
  return ctrl.Result{}, err
 }

 rs.Labels["pod-count"] = fmt.Sprintf("%v", len(pods.Items))
 err = a.Update(context.TODO(), rs)
 if err != nil {
  return ctrl.Result{}, err
 }

 return ctrl.Result{}, nil
}

Believe it or not, this short piece of code is all you need for an example controller.

main function

Let's take a look at the main function.

func main() {
 var log = ctrl.Log.WithName("builder-examples")

 manager, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{})
 if err != nil {
  log.Error(err, "could not create manager")
  os.Exit(1)
 }

 err = ctrl.
  NewControllerManagedBy(manager). // Create the Controller
  For(&appsv1.ReplicaSet{}).       // ReplicaSet is the Application API
  Owns(&corev1.Pod{}).             // ReplicaSet owns Pods created by it
  Complete(&ReplicaSetReconciler{Client: manager.GetClient()})
 if err != nil {
  log.Error(err, "could not create controller")
  os.Exit(1)
 }

 if err := manager.Start(ctrl.SetupSignalHandler()); err != nil {
  log.Error(err, "could not start manager")
  os.Exit(1)
 }
}

In the main function, there are three following steps.

1. Initialize a manager with NewManager.
2. Create a controller and register it to the manager with NewControllerManagedBy, configure which resources to trigger the controller with For and Owns, and then lastly complete with ReplicaSetReconciler.
3. Lastly, start the manager

At this point, you might have several questions popping up in your mind, such as ‘What is the manager?’ or ‘What is the relationship between the controller and manager?’

Additionally, you may be curious about the terms ‘For’ and ‘Owns’.

Those are the starting point of learning controller-runtime in details.

ReplicaSetReconciler

The other part of the code above is about ReplicaSetReconciler. As the name indicates, this Reconciler is the implementation of the reconciliation logic.

type ReplicaSetReconciler struct {
 client.Client
}

As you can see, ReplicaSetReconciler embeds client.Client, which is also a controller-runtime component. But for now, it can be thought of as a client to talk to the API server to get and update the target Kubernetes objects, Pod and ReplicaSet specifically in this case.

The reconciliation logic is implemented in the Reconcile function, which is the only, but very important, function.

func (a *ReplicaSetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

 rs := &appsv1.ReplicaSet{}
 err := a.Get(ctx, req.NamespacedName, rs)
 if err != nil {
  return ctrl.Result{}, err
 }

 pods := &corev1.PodList{}
 err = a.List(ctx, pods, client.InNamespace(req.Namespace), client.MatchingLabels(rs.Spec.Template.Labels))
 if err != nil {
  return ctrl.Result{}, err
 }

 rs.Labels["pod-count"] = fmt.Sprintf("%v", len(pods.Items))
 err = a.Update(context.TODO(), rs)
 if err != nil {
  return ctrl.Result{}, err
 }

 return ctrl.Result{}, nil
}

As you can see in the code snippet, the logic is pretty simple:

1. Get ReplicaSet with the NamespacedName given by the request.
2. Get all the Pods with the matching label.
3. Set the label pod-count to the length of the obtained pods.Items
4. Update the ReplicaSet

The Reconcile function is where we need to implement the core logic of the controller, called reconciliation logic, to develop a custom controller with kubebuilder or operator-sdk, both of which internally use controller-runtime.

We'll also cover the important components used in a reconciler, such as Reconciler, Reconcile function, Request, Result, and more.

Summary

We can write a Kubernetes operator with surprisingly small piece of code using controller-runtime. Behind the scenes, there are several components that help developers focus only on the core reconciliation logic. It’s very helpful but at the same time, it hides how the entire system works. So it’s also important to understand what each of the components does.

Series Index

1. Kubernetes Operator series 1 — controller-runtime example controller
2. Kubernetes Operator series 2 — Overview of controller-runtime
3. Kubernetes Operator series 3 — controller-runtime component — Manager
4. Kubernetes Operator series 4 — controller-runtime component — Builder
5. Kubernetes Operator series 5 — controller-runtime component — Reconciler
6. Kubernetes Operator series 6 — controller-runtime component — Controller