Part 6: "The Gatekeeper" — Policy Engines with OPA/Gatekeeper and Kyverno

The Container Odyssey — Season 2: Beyond the Cluster

It was a typical Monday morning at NovaCraft. Alex, recently promoted to Platform Lead, was sipping a freshly brewed coffee while scrolling through the weekend deployment logs. The platform had been humming along smoothly, a testament to the solid foundation he and his team had built. The frantic days of just trying to keep the lights on were mostly behind them. Now, the challenges were more subtle, more about scaling, security, and governance for the 10+ engineering teams that relied on the Kubernetes platform.

A high-priority alert shattered the morning calm. “Critical Security Vulnerability Detected in Production,” the message blared from his screen. A knot of dread tightened in Alex’s stomach. He quickly drilled down into the details. A new microservice, part of the company’s flagship product, had been deployed over the weekend. The vulnerability? A container running as the root user.

While not an immediate breach, it was a gaping security hole. A compromised container running as root could potentially gain control of the entire node, and from there, pivot to other parts of the cluster. Alex’s team had documented best practices, held training sessions, and written wikis, but in the rush of a weekend release, a developer had missed a crucial security setting.

As Alex initiated the rollback and coordinated with the development team to fix the immediate issue, a more profound realization settled in. Relying on documentation and best-effort from developers wasn’t enough. The platform itself needed to enforce these rules. It needed a gatekeeper.

The Need for a Gatekeeper: Why Policies Matter

Alex’s Monday morning fire drill is a classic example of a reactive security posture. A problem is detected after it’s already in production, forcing a scramble to fix it. This approach doesn’t scale, and it certainly doesn’t lead to a good night’s sleep for the platform team. What’s needed is a proactive approach, a way to prevent these kinds of issues from ever reaching production in the first place.

This is where policy enforcement comes in. In the context of Kubernetes, a policy is a set of rules that governs what is and isn’t allowed in the cluster. Think of it as a security guard for your Kubernetes API server. Before any resource (like a Pod, Deployment, or Service) is created or updated, it’s first inspected by the guard. If the resource complies with all the rules, it’s allowed in. If not, it’s rejected with a clear explanation of what needs to be fixed.

This proactive approach yields several key benefits. First and foremost, it enhances security by enforcing best practices, such as preventing containers from running as root, requiring specific security contexts, or restricting access to host resources. Secondly, it ensures compliance by guaranteeing that all deployments meet regulatory or internal requirements. Thirdly, it promotes consistency, maintaining a well-governed environment across multiple teams and clusters. Finally, it can lead to cost optimization by enforcing policies that prevent the over-provisioning of resources, thus helping to control cloud costs.

The Bouncers of Kubernetes: Admission Controllers

So how does Kubernetes enforce these policies? The magic happens through admission controllers. An admission controller is a piece of code that intercepts requests to the Kubernetes API server before an object is persisted in etcd, but after the request is authenticated and authorized.

There are two main types of admission controllers:

• Validating Admission Webhooks: These webhooks can inspect a request and decide whether to admit it or reject it. They can’t modify the object.
• Mutating Admission Webhooks: These webhooks can modify the object before it’s persisted. For example, a mutating webhook could automatically add a sidecar container to every Pod.

Kubernetes has a number of built-in admission controllers, but the real power comes from the ability to create your own custom admission controllers using webhooks. This is where policy engines like OPA/Gatekeeper and Kyverno come into play. They provide a framework for writing and managing policies without having to write and maintain your own admission controller from scratch.

The Contenders: OPA/Gatekeeper vs. Kyverno

Now that we understand the why and how of policy enforcement, let’s look at the two most popular policy engines in the Kubernetes ecosystem: Open Policy Agent (OPA) with Gatekeeper and Kyverno.

The Veteran: OPA and Gatekeeper

Open Policy Agent (OPA) is a general-purpose policy engine that can be used in a wide variety of contexts, from microservices to CI/CD pipelines to, of course, Kubernetes. OPA is a CNCF graduated project, which speaks to its maturity and adoption.

When used with Kubernetes, OPA is typically deployed with Gatekeeper. Gatekeeper is a Kubernetes-native project that provides a CRD-based interface for OPA. This allows you to manage your policies as Kubernetes objects, just like you would with any other resource.

Architecture

The Gatekeeper architecture consists of a few key components:

• Gatekeeper Pod: This is the main component that runs the OPA engine and the Gatekeeper controller.
• ConstraintTemplates: These are CRDs that define the schema and the Rego logic for a policy. Rego is the special-purpose query language used by OPA to write policies.
• Constraints: These are CRDs that instantiate a ConstraintTemplate and specify the parameters for the policy.

Here’s a high-level overview of how it works:

1. You define a ConstraintTemplate that contains the Rego code for your policy.
2. You create a Constraint that references the ConstraintTemplate and specifies the parameters for the policy (e.g., which namespaces to apply the policy to).
3. When a request comes into the API server, the Gatekeeper admission webhook sends it to the OPA engine for evaluation.
4. The OPA engine evaluates the request against the policies defined in the Constraints.
5. If the request is valid, it’s admitted. If not, it’s rejected.

The Power and the Pain of Rego

Rego is an incredibly powerful and flexible language. It allows you to write complex policies that can reason about arbitrary JSON or YAML data. However, it also has a steep learning curve. For developers who are not familiar with logic programming, Rego can be a significant hurdle.

The Challenger: Kyverno

Kyverno is a policy engine that was designed specifically for Kubernetes. It’s also a CNCF graduated project and has been gaining popularity rapidly.

Kyverno’s main selling point is its simplicity. Policies are written in plain YAML, as Kubernetes resources. This means that you don’t need to learn a new language to write policies. If you know how to write a Kubernetes manifest, you know how to write a Kyverno policy.

Architecture

Kyverno’s architecture is much simpler than Gatekeeper’s. It consists of a single component: the Kyverno Pod. This Pod contains the admission controller and the policy engine.

Policies in Kyverno are defined using ClusterPolicy or Policy CRDs. These CRDs allow you to define rules for validating, mutating, and generating resources.

Here’s how it works:

1. You define a ClusterPolicy or Policy that contains the rules for your policy.
2. When a request comes into the API server, the Kyverno admission webhook evaluates it against the policies.
3. If the request is valid, it’s admitted. If not, it’s rejected.

Head-to-Head: Gatekeeper vs. Kyverno

So which one should you choose? The decision between OPA/Gatekeeper and Kyverno depends on your specific needs and team dynamics. OPA/Gatekeeper is the right choice when you require the flexibility to write complex policies using Rego, when you are already using OPA in other parts of your technology stack and want to maintain consistency, or when you have a dedicated platform or security team that can invest the time in learning and managing Rego policies. On the other hand, Kyverno is the better option when you want to get started with policy enforcement quickly and easily, when your policies are relatively straightforward and can be expressed in YAML, or when you want to empower your developers to write and manage their own policies.

For NovaCraft, Alex decides to start with Kyverno. The ease of use and the ability for development teams to self-service their policies are the deciding factors. He wants to build a culture of “paved roads” and guardrails, not a culture of gatekeeping and bottlenecks. With that decision made, it’s time to put theory into practice.

Hands-On: Taming the Cluster with Kyverno

Now it’s time to get our hands dirty. In this section, we’ll walk through installing Kyverno on a local minikube cluster and creating a couple of essential policies.

Prerequisites

• minikube
• kubectl

1. Start Your Engines: Firing up minikube

First, let’s start our local Kubernetes cluster:

minikube start

2. Installing Kyverno

Installing Kyverno is as simple as applying a single YAML file:

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.11.1/install.yaml

This will install Kyverno in its own namespace, kyverno, and set up all the necessary CRDs, Deployments, and Services.

You can verify the installation by checking the pods in the kyverno namespace:

kubectl get pods -n kyverno

You should see a pod named something like kyverno-7c7c7c7c7c-abcde in the Running state.

3. Policy Time: Enforcing Labels

Our first policy will be a simple one: we’ll require that all new Pods have a team label. This is a common policy that helps with cost allocation, ownership, and general organization.

Create a file named require-team-label.yaml with the following content:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-team-label
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-for-team-label
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "The label `team` is required."
      pattern:
        metadata:
          labels:
            team: "?*"

Now, apply the policy:

kubectl apply -f require-team-label.yaml

Let’s test it out. Create a file named good-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: good-pod
  labels:
    team: nova
spec:
  containers:
  - name: nginx
    image: nginx

And a file named bad-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
spec:
  containers:
  - name: nginx
    image: nginx

Now, try to create both pods:

kubectl apply -f good-pod.yaml
# pod/good-pod created

kubectl apply -f bad-pod.yaml
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
#
# resource Pod/default/bad-pod was blocked due to the following policies
#
# require-team-label:
#   check-for-team-label: 'validation error: The label `team` is required. Rule check-for-team-label failed at path /metadata/labels/team/'

Success! The good-pod was created, but the bad-pod was rejected because it was missing the team label.

4. Locking it Down: Blocking Privileged Containers

Now for a more security-focused policy. We’ll create a policy that blocks any container from running in privileged mode. This is a critical security best practice that helps to prevent container escapes.

Create a file named disallow-privileged-containers.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-for-privileged-containers
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Privileged containers are not allowed."
      pattern:
        spec:
          containers:
          - securityContext:
              privileged: false

Apply the policy:

kubectl apply -f disallow-privileged-containers.yaml

Now, let’s create a pod that tries to run a privileged container. Create a file named privileged-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
  labels:
    team: security
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      privileged: true

Try to create the pod:

kubectl apply -f privileged-pod.yaml
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
#
# resource Pod/default/privileged-pod was blocked due to the following policies
#
# disallow-privileged-containers:
#   check-for-privileged-containers: 'validation error: Privileged containers are not allowed. Rule check-for-privileged-containers failed at path /spec/containers/0/securityContext/privileged/'

As expected, Kyverno blocked the creation of the privileged pod. Our cluster is now a little bit safer.

5. Debugging and Troubleshooting

Even with a tool as simple as Kyverno, things can sometimes go wrong. Here are a few common issues and how to solve them:

• Policy not being enforced: If a policy doesn’t seem to be working, the first thing to check is the Kyverno logs. You can view the logs with kubectl logs -n kyverno -l app=kyverno. Look for any errors related to your policy.
• Understanding why a resource was blocked: When Kyverno blocks a resource, it provides a detailed error message explaining which policy and which rule caused the failure. Read this message carefully to understand what needs to be fixed.
• Testing policies: It’s always a good idea to test your policies in a non-production environment before rolling them out to the entire cluster. You can use the validationFailureAction: Audit setting to log policy violations without actually blocking them.

Key Takeaways

• Policy enforcement is essential for running Kubernetes at scale. It helps to improve security, compliance, and consistency.
• Admission controllers are the gatekeepers of the Kubernetes API server. They allow you to intercept and validate requests before they are persisted.
• OPA/Gatekeeper and Kyverno are the two most popular policy engines for Kubernetes.
• Kyverno is a great choice for teams that want to get started with policy enforcement quickly and easily. Its YAML-based policies are easy to write and understand.

Back at his desk, Alex watched the Kyverno policies do their work. A developer from another team pinged him on Slack. “Hey, I tried to deploy a new service, but it got blocked. Something about a missing team label?” Alex smiled. The gatekeeper was on duty. He quickly explained the new policy and pointed the developer to the documentation. It was a small victory, but it was a start. The platform was becoming more secure, more resilient, and easier to manage, one policy at a time.

Next time on The Container Odyssey: Join us for Part 7, where Alex dives into the world of service mesh with Istio and Linkerd, and discovers how to bring true observability and security to his microservices.

This is Part 6 of Season 2 of The Container Odyssey.

References

1. Kubernetes Policies
2. Kyverno
3. Open Policy Agent (OPA)
4. Gatekeeper