Kubernetes Operator series 3 — controller-runtime component — Manager

6 min readApr 24, 2023

Overview

In this series, we will be taking a deep dive into controller-runtime.

In this post, we will be focusing on the Manager component, which is a crucial part of controller-runtime.

If you haven’t had a chance to read our previous posts about the example controller with controller-runtime and the overview of controller-runtime, we recommend checking them out before diving into this post.

What’s exactly a Manager?

Package manager is required to create Controllers and provides shared dependencies such as clients, caches, schemes, etc. Controllers must be started by calling Manager.Start.
from Manager

In simple terms, the main role of the Manager is to manage the lifecycle of a set of controllers, which is why it’s called the Manager. But what does it mean to “manage controllers?” There are three main aspects to this:

Managing the lifecycle of controllers, including the registration of controllers, starting and stopping each controller, and more.
Providing access to shared resources that are used by the controllers, such as the Kubernetes API server client, cache, event recorder, and more.
Managing leader election for the controllers to ensure high availability.

Manager is an interface:

// Manager initializes shared dependencies such as Caches and Clients, and provides them to Runnables.
// A Manager is required to create Controllers.
type Manager interface {
 cluster.Cluster
 Add(Runnable) error
 Elected() <-chan struct{}
 AddMetricsExtraHandler(path string, handler http.Handler) error
 AddHealthzCheck(name string, check healthz.Checker) error
 AddReadyzCheck(name string, check healthz.Checker) error
 Start(ctx context.Context) error
 GetWebhookServer() *webhook.Server
 GetLogger() logr.Logger
 GetControllerOptions() v1alpha1.ControllerConfigurationSpec
}

In this post, we’ll mainly check the following: cluster, Add(Runnable), and Start(ctx context.Context) error.

How is a manager used?

As we’ve seen in the simple example in the first episode of this series, a Manager is initialized with NewManager

manager, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{})

Then, manager is passed to a builder using NewControllerManagedBy and a controller is registered internally.

err = ctrl.
  NewControllerManagedBy(manager). // Create the Controller
  For(&appsv1.ReplicaSet{}).       // ReplicaSet is the Application API
  Owns(&corev1.Pod{}).             // ReplicaSet owns Pods created by it
  Complete(&ReplicaSetReconciler{Client: manager.GetClient()})

Lastly, manager starts all the registered controllers:

manager.Start(ctrl.SetupSignalHandler())

Let’s see what happens inside the manager in each of those steps.

What’s happening internally?

NewManager

New function sets up another component cluster, which includes the resources shared among the registered controllers, e.g. a client to Kubernetes API server, cache, etc. We’ll study cluster in details in another post.

Then, the New function initializes controllerManager, which is the actual instance that works as a manager.

I attached the struct of controllerManager from manager/internal.go:


var _ Runnable = &controllerManager{}

type controllerManager struct {
 sync.Mutex
 started bool

 ...

 runnables            *runnables

 // cluster holds a variety of methods to interact with a cluster. Required.
 cluster cluster.Cluster

 ...
}

The whole struct looks pretty overwhelming so I just picked up a few important fields:

runnables: *runnables This field stores a set of the registered controllers (and more). We’ll take a look at the runnables later.
cluster: cluster provides shared methods to interact with a cluster. e.g. client, cache, etc.
leader election related fields
health, liveness, readiness
etc.

Next question would be what exactly runnables is. Let’s see runnables:

// runnables handles all the runnables for a manager by grouping them accordingly to their
// type (webhooks, caches etc.).
type runnables struct {
 Webhooks       *runnableGroup
 Caches         *runnableGroup
 LeaderElection *runnableGroup
 Others         *runnableGroup
}

runnables is a struct that holds four types of runnableGroup. The runnables struct is used to group different types of components that can be registered with the controller manager. Each runnableGroup holds a list of components of the same type, and provides methods to add, remove, and start the components. The different groups are used to ensure that components are started in the correct order, and to allow different types of components to be managed separately.

Webhooks: a group of webhook servers defined for controllers. let’s dive into more detail about webhooks later in this series
Caches: a group of runnable with GetCache() methods. cluster will be added to this group.
LeaderElection: a group of runnable with NeedLeaderElection() method. We’ll come back to look into details when studying leader election.
Others: When we register a controller, the controller will be added to this group.

Theses runnables are set by Add function:

func (r *runnables) Add(fn Runnable) error {
 switch runnable := fn.(type) {
 case hasCache:
  return r.Caches.Add(fn, func(ctx context.Context) bool {
   return runnable.GetCache().WaitForCacheSync(ctx)
  })
 case *webhook.Server:
  return r.Webhooks.Add(fn, nil)
 case LeaderElectionRunnable:
  if !runnable.NeedLeaderElection() {
   return r.Others.Add(fn, nil)
  }
  return r.LeaderElection.Add(fn, nil)
 default:
  return r.LeaderElection.Add(fn, nil)
 }
}

The Add function in runnables takes a Runnable and adds it to the appropriate group based on its type. The hasCache and LeaderElectionRunnable interfaces are used to determine whether a component should be added to the Caches or LeaderElection group, respectively.

You might have noticed controllerManager itself also implements another interface Runnable:

var _ Runnable = &controllerManager{}

Runnable is a very simple interface, which has Start function with context as the only argument and returns error.

// Runnable allows a component to be started.
// It's very important that Start blocks until
// it's done running.
type Runnable interface {
 // Start starts running the component.  The component will stop running
 // when the context is closed. Start blocks until the context is closed or
 // an error occurs.
 Start(context.Context) error
}

NewControllerManagedBy

NewControllerManagedBy is a function under builder package as you can see here:

 // NewControllerManagedBy returns a new controller builder that will be started by the provided Manager.
 NewControllerManagedBy = builder.ControllerManagedBy

So we can study more details in a future post for builder. Here let’s just check what it does.

Builder is the following struct:

// Builder builds a Controller.
type Builder struct {
 forInput         ForInput
 ownsInput        []OwnsInput
 watchesInput     []WatchesInput
 mgr              manager.Manager
 globalPredicates []predicate.Predicate
 ctrl             controller.Controller
 ctrlOptions      controller.Options
 name             string
}

ContrllerManagedBy just set the specified manager to mgr field:

// ControllerManagedBy returns a new controller builder that will be started by the provided Manager.
func ControllerManagedBy(m manager.Manager) *Builder {
 return &Builder{mgr: m}
}

And with Complete function, Build function is called, and internally doController and doWatch are called. In doContrller, the shared resources in the manager is used to initialize a controller. doWatch starts the controller.

NewControllerManagedBy is used for each controller to register a controller to the Manager. We’ll check in details about builder in the future.

Start()

Lastly, we start the manager. As we saw in the Manager interface, manager must have a function Start(context.Context) error

type Manager interface {
  ...
  Start(ctx context.Context) error
  ..
}

controllerManager, an implementation of the Manager interface, has Start(context.Context) error.

The Start() function mainly does the followings:

Add the cluster to runnables (cluster is also a runnable, which is added to runnables.Cache)
Start runnables.Webhooks, runnables.Caches, runnables.Others, and runnables.LeaderElection.


// (1) Add the cluster runnable.
if err := cm.add(cm.cluster); err != nil {
...

// (2) First start any webhook servers
if err := cm.runnables.Webhooks.Start(cm.internalCtx); err != nil {
...

// (3) Start and wait for caches.
if err := cm.runnables.Caches.Start(cm.internalCtx); err != nil {
...

// (4) Start the non-leaderelection Runnables after the cache has synced.
if err := cm.runnables.Others.Start(cm.internalCtx); err != nil {

// (5) Start the leader election and all required runnables.
if err := cm.startLeaderElection(ctx); err != nil {
...
if err := cm.startLeaderElectionRunnables(); err != nil {
...

Simply input, the Start() starts all the registered runnables in the manager. We might be unclear what exactly are started here but we’ll study those components one by one in the incoming posts.

Summary

Manager is to manage the lifecycle of a set of controllers. e.g. registration, start and stop.
controllerManager implements Manager interface and it has cluster and runnables.
cluster has the shared resources, such as client to Kubernetes API server, cache, schema, etc. cluster is initialized in Manager.New function.
runnables has four types Webhooks, Caches, LeaderElection, and Others.
To bind Manager with Controller, we need a builder.
NewControllerManagedBy is used to set the manager to a builder, with which new controller is registered to the manager. The controller is registered to runnables.Others.
Start() starts all the registered runnables in the controllerManager.

This is rough summary of the Manager component. As the manager also interact with several other components, so you might be unclear about many things, we’ll study them one by one and you’ll have better understanding of whole picture gradually.

So please stay tuned for incoming posts!