Part 8: "The Operator" — Custom Resources and Operators

The Container Odyssey — Season 2: Beyond the Cluster

The Platform Lead’s Dilemma

The promotion to “Platform Lead” at NovaCraft had been a whirlwind for Alex. Six months in, the initial excitement of wrangling Kubernetes deployments had been replaced by a new, more profound set of challenges. The days of simply getting containers to run were over. Now, with over ten engineering teams relying on the platform, the questions were bigger, the stakes higher.

“How do we run this at scale?” a backend engineer had asked in the last platform sync. “Securely?” a security engineer chimed in. “And can we even tell what it’s costing us?” the Head of Engineering had queried, peering over his glasses.

These were the questions that now kept Alex up at night. The Kubernetes cluster, once a pristine landscape of well-behaved stateless applications, was becoming a sprawling metropolis of complex, stateful services. The latest challenge, and the one currently causing Alex the most grief, was the new in-house-built, stateful, distributed database, codenamed “ChronosDB.”

ChronosDB was a beast. It required a complex dance of initialization, peer discovery, and stateful-set scaling. The team that built it had provided a 20-page runbook, a collection of bash scripts, and a Helm chart that looked more like a Rube Goldberg machine than a deployment descriptor. Every time a new ChronosDB instance was needed for a new environment, it was a manual, error-prone process that took hours.

“This isn’t sustainable,” Alex muttered, staring at the growing backlog of requests for new ChronosDB clusters. “We’re treating ChronosDB like a pet. We need to treat it like cattle. We need to automate this, make it a first-class citizen of our Kubernetes ecosystem. There has to be a better way.”

And that’s when the idea sparked. What if they could extend Kubernetes itself? What if they could teach Kubernetes what a ChronosDB cluster is, and how to manage it, just like it manages Deployments and Services? This line of thinking led Alex down a rabbit hole of discovery, a journey into the world of Custom Resources and the powerful Operator Pattern.

Teaching Kubernetes New Tricks: An Introduction to Custom Resources

At its core, Kubernetes is a master of resource management. It knows about Pods, Deployments, Services, and a host of other built-in resource types. You declare your desired state in a YAML file, and Kubernetes controllers work tirelessly to make that state a reality. But what happens when you have a resource that Kubernetes doesn’t know about, like our ChronosDB?

This is where Custom Resource Definitions (CRDs) come in. A CRD is a way to extend the Kubernetes API, to teach it about your own custom objects. Think of it like adding a new word to a dictionary. Once you’ve defined a new word, you can use it in sentences, and people who know the definition will understand you. Similarly, once you define a CRD, you can create Custom Resources (CRs) of that type, and Kubernetes will store and manage them for you.

What is a Custom Resource Definition (CRD)?

A CRD is itself a Kubernetes resource. You create a CRD by POSTing a YAML manifest to the Kubernetes API server. This manifest defines the group, version, and scope of your new resource, along with a schema that defines its structure.

Let’s imagine we want to create a CRD for our ChronosDB. The CRD definition might look something like this:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: chronosdbs.db.novacraft.com
spec:
  group: db.novacraft.com
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  minimum: 1
                version:
                  type: string
  scope: Namespaced
  names:
    plural: chronosdbs
    singular: chronosdb
    kind: ChronosDB
    shortNames:
    - cdb

Once this CRD is created, you can create ChronosDB resources just like you would create a Pod or a Deployment:

apiVersion: db.novacraft.com/v1alpha1
kind: ChronosDB
metadata:
  name: my-chronos-instance
spec:
  replicas: 3
  version: "1.2.3"

Now, you can use kubectl get chronosdbs to see your custom resources. But here’s the catch: creating a CRD and a CR doesn’t, by itself, do anything. Kubernetes now knows about ChronosDB and will store your my-chronos-instance object, but it has no idea how to make that object a reality. It doesn’t know how to create the necessary StatefulSet, Services, or ConfigMaps that make up a ChronosDB cluster. For that, we need a controller.

The Operator Pattern: Bringing Your CRDs to Life

This is where the Operator Pattern comes in. An Operator is a custom Kubernetes controller that watches for your Custom Resources and takes action to bring the current state of the cluster to the desired state defined in the CR. In essence, an Operator encodes the operational knowledge of a human operator into software.

Think about the 20-page runbook for ChronosDB. It contains a series of steps a human would take to deploy, manage, and scale a ChronosDB cluster. An Operator automates those steps. It’s a domain-specific controller that understands the intricacies of a particular application.

The Control Loop

At the heart of every controller, including Operators, is a control loop. The control loop continuously performs the following steps:

1. Observe: The Operator watches for changes to resources of its kind (e.g., ChronosDB resources).
2. Analyze: It compares the desired state (defined in the CR’s spec) with the actual state of the cluster.
3. Act: It takes action to reconcile the difference. This could involve creating, updating, or deleting other Kubernetes resources like StatefulSets, Services, ConfigMaps, and Secrets.

For our ChronosDB example, the Operator would see the my-chronos-instance CR, notice that there is no corresponding StatefulSet, and create one with 3 replicas. If a user later updates the CR to have 5 replicas, the Operator would see this change and scale up the StatefulSet.

Building Operators: Operator SDK and Kubebuilder

Writing an Operator from scratch can be complex. You need to interact with the Kubernetes API, manage caches, and handle all the intricacies of a distributed system. Fortunately, the community has created powerful tools to simplify this process.

The two most popular frameworks for building Operators are:

• Operator SDK: Part of the Operator Framework, the SDK provides high-level APIs, abstractions, and project scaffolding to make building operators easier. It supports Go, Ansible, and Helm-based operators.
• Kubebuilder: A project from the Kubernetes Special Interest Group (SIG) API Machinery, Kubebuilder provides a framework for building Kubernetes APIs using CRDs. It is focused on Go-based operators and is the foundation upon which the Operator SDK’s Go support is built.

Both tools help you bootstrap a new project, generate the boilerplate code for your CRD and controller, and provide a streamlined development workflow.

Operator Maturity Levels

The Operator Framework defines a maturity model (also called capability levels) to classify the sophistication of an Operator. This helps users understand what an Operator can do.

Discovering Operators: OperatorHub.io

Before you set out to build your own Operator, it’s worth checking if one already exists. OperatorHub.io is a public registry for finding and sharing Kubernetes Operators. Many popular open-source projects and vendors provide Operators for their applications, which can save you a significant amount of time and effort.

To Build or Not to Build?

Writing an Operator is a significant investment. It’s not something you should do for every application. So, when should you consider building one?

Build an Operator when:

• You are managing a complex, stateful application.
• The application requires domain-specific operational knowledge to manage.
• You need to automate complex lifecycle tasks like backups, restores, and upgrades.
• You want to provide a “database-as-a-service” or “platform-as-a-service” experience to your users.

Don’t build an Operator when:

• A simple Helm chart is sufficient to manage the application.
• The application is stateless and can be easily managed with a Deployment.
• An existing Operator already meets your needs.
• You don’t have the resources to maintain the Operator over the long term.

Hands-On: Building Your First Operator with Kubebuilder

Now, let’s get our hands dirty and build a simple Operator. We’ll create a WebApp operator that manages a simple web application. This operator will watch for WebApp resources and, for each one, create a Deployment and a Service.

Prerequisites

Before we begin, you’ll need a few tools installed on your macOS machine:

• Homebrew: The missing package manager for macOS. If you don’t have it, you can install it from https://brew.sh/.
• Docker Desktop: To run a local Kubernetes cluster. You can download it from https://www.docker.com/products/docker-desktop.
• Go: The programming language we’ll use to write our operator. You can install it with Homebrew: brew install go.
• minikube: For running a local Kubernetes cluster. Install it with Homebrew: brew install minikube.

Once you have these installed, start minikube:

minikube start

Installing Kubebuilder

Kubebuilder is the scaffolding for our operator. We can install it using the official installation script:

curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

Scaffolding the Project

Now, let’s create a new directory for our project and initialize it with Kubebuilder:

mkdir webapp-operator && cd webapp-operator

go mod init webapp-operator

kubebuilder init --domain novacraft.com --repo webapp-operator

This will create a new project with a bunch of files and directories. The most important ones are:

• go.mod: Defines the Go module for our project.
• main.go: The entry point for our operator.
• Dockerfile: To build a container image for our operator.
• config/: Contains YAML manifests for deploying our operator.

Creating the API

Next, let’s create the API for our WebApp resource:

kubebuilder create api --group web --version v1 --kind WebApp

This command will do two things:

1. Create a new file api/v1/webapp_types.go, which defines the schema for our WebApp resource.
2. Create a new file internal/controller/webapp_controller.go, which contains the reconciliation logic for our controller.

Let’s edit api/v1/webapp_types.go to define the spec for our WebApp resource. We’ll add a Replicas field and an Image field:

// WebAppSpec defines the desired state of WebApp
type WebAppSpec struct {
  // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
  // Important: Run "make" to regenerate code after modifying this file

  // +kubebuilder:validation:Minimum=1
  // Replicas is the number of desired replicas.
  // +optional
  Replicas *int32 `json:"replicas,omitempty"`

  // Image is the container image to run.
  Image string `json:"image"`
}

After modifying this file, run make to regenerate the code:

make

Implementing the Controller

Now for the fun part: implementing the controller logic. Open internal/controller/webapp_controller.go and find the Reconcile method. This is where we’ll add our logic to create a Deployment and a Service.

The generated Reconcile method is mostly empty. We’ll need to add code to:

1. Fetch the WebApp resource that triggered the reconciliation.
2. Create a Deployment with the desired number of replicas and container image.
3. Create a Service to expose the Deployment.

Here’s a simplified version of what the Reconcile method will look like. Replace the existing Reconcile method with the following code:

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  log := log.FromContext(ctx)

  var webapp webv1.WebApp
  if err := r.Get(ctx, req.NamespacedName, &webapp); err != nil {
    log.Error(err, "unable to fetch WebApp")
    return ctrl.Result{}, client.IgnoreNotFound(err)
  }

  // Create or update the Deployment
  deployment := &appsv1.Deployment{
    ObjectMeta: metav1.ObjectMeta{
      Name:      webapp.Name,
      Namespace: webapp.Namespace,
    },
  }

  if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, deployment, func() error {
    replicas := int32(1)
    if webapp.Spec.Replicas != nil {
      replicas = *webapp.Spec.Replicas
    }
    deployment.Spec.Replicas = &replicas
    deployment.Spec.Selector = &metav1.LabelSelector{
      MatchLabels: map[string]string{
        "app": webapp.Name,
      },
    }
    deployment.Spec.Template = corev1.PodTemplateSpec{
      ObjectMeta: metav1.ObjectMeta{
        Labels: map[string]string{
          "app": webapp.Name,
        },
      },
      Spec: corev1.PodSpec{
        Containers: []corev1.Container{
          {
            Name:  "webapp",
            Image: webapp.Spec.Image,
          },
        },
      },
    }
    return controllerutil.SetControllerReference(&webapp, deployment, r.Scheme)
  }); err != nil {
    log.Error(err, "unable to create or update Deployment")
    return ctrl.Result{}, err
  }

  // Create or update the Service
  service := &corev1.Service{
    ObjectMeta: metav1.ObjectMeta{
      Name:      webapp.Name,
      Namespace: webapp.Namespace,
    },
  }

  if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, service, func() error {
    service.Spec.Selector = map[string]string{
      "app": webapp.Name,
    }
    service.Spec.Ports = []corev1.ServicePort{
      {
        Port: 80,
      },
    }
    return controllerutil.SetControllerReference(&webapp, service, r.Scheme)
  }); err != nil {
    log.Error(err, "unable to create or update Service")
    return ctrl.Result{}, err
  }

  return ctrl.Result{}, nil
}

This code uses the controller-runtime library to create or update the Deployment and Service. The SetControllerReference call is important: it sets the WebApp resource as the owner of the Deployment and Service, so that when the WebApp is deleted, the Deployment and Service are garbage collected.

Running the Operator

Now, let’s run our operator. First, we need to install the CRD into our minikube cluster:

make install

Next, we can run the operator locally:

make run

This will start the operator on your local machine. It will connect to your minikube cluster and start watching for WebApp resources.

Deploying a WebApp

Now, let’s deploy a WebApp. Create a file named config/samples/webapp_v1_webapp.yaml with the following content:

apiVersion: web.novacraft.com/v1
kind: WebApp
metadata:
  name: my-webapp
spec:
  replicas: 2
  image: "nginxdemos/hello"

Now, apply this manifest:

kubectl apply -f config/samples/webapp_v1_webapp.yaml

You should see that a Deployment and a Service have been created:

kubectl get deployments
# NAME        READY   UP-TO-DATE   AVAILABLE   AGE
# my-webapp   2/2     2            2           10s

kubectl get services
# NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
# my-webapp   ClusterIP   10.108.144.34   <none>        80/TCP    10s

If you update the config/samples/webapp_v1_webapp.yaml file to change the number of replicas to 3 and re-apply it, you’ll see the operator scale up the Deployment.

Cleaning Up

To clean up, you can delete the WebApp resource:

kubectl delete -f config/samples/webapp_v1_webapp.yaml

Because we set the owner reference, the Deployment and Service will be automatically deleted as well.

To stop the operator, press Ctrl+C in the terminal where it’s running.

Debugging and Troubleshooting

Building and running operators can sometimes feel like black magic. Here are a few common issues you might encounter and how to solve them:

• CRD Not Found: If you get an error like the server could not find the requested resource, it usually means you haven’t installed the CRD into your cluster. Run make install to install it.
• Controller Not Starting: If your controller doesn’t start, check the logs for errors. Common issues include incorrect RBAC permissions or problems connecting to the Kubernetes API server.
• Reconciliation Loop Not Triggering: If your controller doesn’t seem to be doing anything when you create or update a CR, make sure the Reconcile function is being called. You can add log statements to the beginning of the function to verify this. Also, check the operator’s logs for any errors.
• Garbage Collection Not Working: If the resources created by your operator are not being deleted when you delete the CR, it’s likely you forgot to set the controller reference. Make sure you are calling controllerutil.SetControllerReference in your reconciliation logic.

Key Takeaways

This chapter covered a lot of ground. Here are the key takeaways:

• Custom Resource Definitions (CRDs) allow you to extend the Kubernetes API with your own custom resource types.
• The Operator Pattern is a way to automate the management of complex applications on Kubernetes by writing a custom controller.
• Operators use a control loop to observe the state of the cluster and take action to reconcile the desired state with the actual state.
• Kubebuilder and the Operator SDK are powerful tools that simplify the process of building operators.
• The Operator Maturity Model helps classify the capabilities of an operator.
• OperatorHub.io is a great place to find and share community-built operators.

The Journey Continues

Alex leaned back, a sense of accomplishment washing over them. The WebApp operator was simple, but it was a proof of concept. It was the key to taming ChronosDB and the myriad of other complex services that were popping up at NovaCraft. The path forward was clear: they would build a ChronosDB operator, encoding all the operational knowledge from that 20-page runbook into a self-managing, autonomous system.

The journey to platform maturity was long, but with the power of the Operator Pattern, Alex felt equipped to handle whatever came next. The Kubernetes cluster was no longer just a container orchestrator; it was becoming a true application platform, a platform they could extend and mold to their will.

But as Alex started sketching out the design for the ChronosDB operator, a new question emerged. How would they manage the lifecycle of the operator itself? How would they deploy it, upgrade it, and manage its dependencies? The answer, Alex suspected, lay in the next chapter of their container odyssey: the world of Operator Lifecycle Management (OLM).

This is Part 8 of Season 2 of The Container Odyssey.