Part 7: "The Bill" — Cost Optimization and Resource Efficiency

Series: The Container Odyssey — Season 2: Beyond the Cluster

The Bill Always Comes Due

The past six months at NovaCraft had been a whirlwind for Alex. What started as a senior backend engineering role had morphed into a full-blown platform leadership position. The Kubernetes clusters, once a manageable handful of nodes serving a few core services, had exploded into a sprawling digital metropolis. Ten engineering teams now deployed their applications to the platform Alex and his small team had painstakingly built. The frantic energy of a fast-growing startup was both exhilarating and terrifying.

Alex leaned back in his chair, the morning sun casting long shadows across his desk. He’d just finished his daily ritual: checking the platform’s vitals. ArgoCD showed all applications were synced and healthy. Prometheus and Grafana reported no critical alerts. The cluster was, by all technical measures, stable. Yet, a sense of unease lingered. It had been quiet… too quiet.

As if on cue, a Slack notification popped up, pulling him from his thoughts. It was from Sarah, the CFO.

Sarah: “Morning Alex. Got a minute? I want to chat about our cloud bill.”

Alex’s stomach tightened. He knew this day was coming. He’d seen the billing alerts in his inbox, each one a little more alarming than the last. He’d mentally filed them under “deal with it later,” a debt of attention he could no longer afford to defer.

Alex: “Of course. I’m free now. Want to hop on a quick call?”

Sarah: “Perfect. Sending you a link.”

A moment later, Sarah’s face appeared on his screen. She was known for her directness, a trait Alex appreciated, though it could be intimidating.

“Morning, Alex,” she began, skipping the pleasantries. “I’ve been reviewing our cloud spend for the last quarter. It’s… significant. To be precise, it has tripled in the last three months. Can you help me understand why?”

Alex felt a bead of sweat form on his brow. He had the technical knowledge, the deep understanding of Pods, Deployments, and Services. He had mastered the art of getting applications running. But this? This was a different beast entirely. This wasn’t about uptime; it was about cost. He could talk all day about reconciliation loops and etcd, but he couldn’t, with any real confidence, explain the ‘why’ behind the ballooning bill.

“I’ve been noticing the increase as well,” Alex admitted, his voice steady despite his internal churn. “The platform’s adoption has been faster than we anticipated. More teams, more services, more traffic… it all adds up.”

“I get that,” Sarah said, her expression softening slightly. “Growth is good. But we need to ensure it’s sustainable. A 3x increase in three months isn’t a scalable trend. I need you to dig into this, Alex. I need to know where the money is going, if we’re being efficient, and what we can do to get these costs under control without stifling innovation.”

The call ended, but the weight of Sarah’s words remained. Alex stared at the Grafana dashboard again, but this time he wasn’t looking at CPU and memory charts in terms of performance. He was looking at them as dollar signs. Every CPU cycle, every gigabyte of memory, every persistent volume claim was a line item on an invoice that was now under intense scrutiny. The challenge had shifted. It was no longer just about keeping the lights on; it was about making sure they weren’t paying for a lighthouse when a lightbulb would do.

Alex was no longer just a platform lead. He was now a digital economist, and his first order of business was to conduct a full-scale audit of NovaCraft’s Kubernetes economy. The bill had come due, and it was time to pay up—not just with money, but with a new level of expertise. The era of “deploy and forget” was over. The age of cost optimization and resource efficiency had begun.

Deconstructing the Kubernetes Bill: A Deep Dive

To tackle the beast of a cloud bill, Alex first needed to understand its anatomy. A Kubernetes cluster, much like a bustling city, has its own economy. The currency is cloud credits, and the resources are the infrastructure that powers the applications. To optimize the economy, you need to understand the key drivers of cost.

The Three Pillars of Kubernetes Cost

Think of your Kubernetes cluster as a digital city. The cost of running this city can be broken down into three main categories:

1. Compute: This is the real estate of your city – the land on which you build your houses (Pods). In Kubernetes, compute resources are the virtual machines (VMs) or bare-metal servers that form the nodes of your cluster. The more nodes you have, and the bigger they are (in terms of CPU and memory), the more you pay. This is often the largest portion of the cloud bill.

2. Storage: This is the warehouse district of your city. Every time a Pod needs to store data persistently, it uses a PersistentVolume (PV), which is backed by a cloud provider’s storage service (like AWS EBS, GCP Persistent Disk, or Azure Disk Storage). The more data you store, and the faster the storage you choose, the higher the cost.

3. Networking: This is the road and highway system of your city. Data flowing in and out of your cluster (ingress and egress traffic), as well as traffic between services within the cluster, can incur costs. This is especially true for cross-zone or cross-region traffic, which is like paying a toll to move goods between different parts of your city.

The Art of Right-Sizing: The Goldilocks Principle

One of the biggest sources of waste in Kubernetes is overprovisioning. It’s like booking a 10-ton truck to move a single box of books. You’re paying for capacity you don’t need. This is where right-sizing comes in. It’s the process of analyzing the actual resource consumption of your applications and adjusting their resource requests and limits to match.

In Kubernetes, you can specify two types of resource constraints for a container:

• Requests: This is the minimum amount of resources that Kubernetes will reserve for a container. It’s a guarantee. If a container requests 1 CPU core, Kubernetes will ensure it always has access to at least that much.

• Limits: This is the maximum amount of resources that a container can use. If a container tries to exceed its limit, it will be throttled (for CPU) or terminated (for memory).

Finding the “just right” values for requests and limits is the essence of right-sizing. If you set your requests too low, your application might not get the resources it needs to start up or run properly. If you set them too high, you’re wasting resources and driving up costs. The same goes for limits. If they’re too low, your application might be unfairly throttled or killed. If they’re too high, a memory leak in your application could bring down the entire node.

The Cluster Autoscaler: Your Automated Real Estate Developer

The Cluster Autoscaler is a key component in the cost optimization toolkit. It automatically adjusts the size of your cluster by adding or removing nodes based on the resource demands of your Pods. Think of it as an automated real estate developer for your digital city. When new residents (Pods) arrive and there’s no more space, the developer builds a new apartment building (node). When residents leave and a building becomes empty, the developer demolishes it to save on maintenance costs.

The Cluster Autoscaler works by monitoring the cluster for Pods that are in a “Pending” state because there are no nodes with enough resources to run them. When it sees such Pods, it adds a new node to the cluster. Conversely, when it finds a node that is underutilized for a certain period of time, and all the Pods running on it can be rescheduled to other nodes, it removes the node from the cluster.

Spot/Preemptible Instances: The Budget-Friendly Suburbs

Cloud providers offer a special type of VM instance called spot instances (on AWS and GCP) or preemptible VMs (on GCP) that are available at a significantly lower price than regular on-demand instances. The catch is that the cloud provider can reclaim these instances at any time with very little notice. It’s like living in a budget-friendly suburb with the constant risk of eviction.

These instances are perfect for running stateless, fault-tolerant workloads that can handle interruptions, such as batch processing jobs, CI/CD pipelines, or even some web servers. By running these types of workloads on spot instances, you can dramatically reduce your compute costs. However, you need to be careful not to run critical, stateful applications on them, as a sudden termination could lead to data loss or service disruption.

Tools of the Trade: Gaining Cost Visibility

You can’t optimize what you can’t measure. Fortunately, there are several open-source tools that can help you gain visibility into your Kubernetes costs:

VPA vs. HPA: A Tale of Two Autoscalers

In Season 1, we talked about the Horizontal Pod Autoscaler (HPA), which automatically scales the number of Pods in a deployment based on CPU or memory usage. But what if you don’t want to scale out? What if you want to scale up? This is where the Vertical Pod Autoscaler (VPA) comes in.

The VPA automatically adjusts the resource requests and limits of your containers to match their actual usage. It’s like having a personal trainer for your Pods, constantly adjusting their workout routine to ensure they’re in peak condition. The VPA can be particularly useful for stateful applications or applications that are difficult to scale horizontally.

It’s important to note that you cannot use the VPA and the HPA together on the same set of Pods for CPU or memory. You have to choose one or the other. The HPA is generally a better choice for stateless applications that can be easily replicated, while the VPA is a better choice for stateful applications or applications with vertical scaling needs.

Resource Quotas and LimitRanges: Enforcing the Rules of the City

In a multi-tenant cluster where multiple teams are sharing the same resources, it’s important to have some rules in place to ensure fair play. This is where ResourceQuotas and LimitRanges come in.

• ResourceQuotas allow you to set a limit on the total amount of resources that can be consumed by a namespace. For example, you can say that the “analytics” namespace can use a maximum of 100 CPU cores and 500Gi of memory.

• LimitRanges allow you to set default resource requests and limits for containers in a namespace. This is useful for ensuring that every container has some resource constraints, even if the developer forgets to specify them.

Under the Hood: The Mechanics of Cost Optimization

Now that we have a conceptual understanding of the key cost optimization strategies, let’s dive deeper into the technical details. As a platform lead, Alex needs to understand not just the what, but also the how.

The Cluster Autoscaler in Action

The Cluster Autoscaler is a standalone program that you run in your cluster. It connects to your cloud provider’s API to add or remove nodes. Here’s a step-by-step look at how it works:

1. Monitoring for Unscheduled Pods: The autoscaler periodically scans the cluster for Pods that are in the Pending state. A Pod can be in this state for various reasons, but the autoscaler is specifically looking for Pods that are unschedulable due to insufficient resources.

2. Evaluating Node Groups: When the autoscaler finds an unschedulable Pod, it looks at the available node groups (in a cloud environment, this is typically a group of VMs with the same instance type and configuration). It then simulates the addition of a new node from each node group and checks if the unschedulable Pod could be scheduled on the new node.

3. Choosing the Best Node Group: If there are multiple node groups that could accommodate the unschedulable Pod, the autoscaler uses an “expander” strategy to choose which one to scale up. The default strategy is random, which randomly picks one of the available node groups. Other strategies include most-pods, which picks the node group that would be able to schedule the most waiting pods, and least-waste, which picks the node group that would have the least idle CPU after the scale-up.

4. Scaling Up: Once a node group is chosen, the autoscaler calls the cloud provider’s API to request a new node. The cloud provider then provisions a new VM and adds it to the cluster.

5. Scaling Down: The autoscaler also monitors the utilization of existing nodes. If a node is underutilized for a certain period of time (the default is 10 minutes), and all the Pods running on it can be safely rescheduled to other nodes, the autoscaler will evict the Pods from the node and then terminate the node.

Important: The Cluster Autoscaler respects PodDisruptionBudgets (PDBs), which means it won’t evict a Pod if it would violate the PDB. This is a crucial safety mechanism that prevents the autoscaler from causing service disruptions.

The Vertical Pod Autoscaler (VPA) Architecture

The VPA is a more complex beast than the Cluster Autoscaler. It consists of three main components:

1. VPA Recommender: This is the brains of the operation. It monitors the resource usage of Pods over time and uses this data to generate recommendations for resource requests. The Recommender stores its recommendations in a Custom Resource Definition (CRD) called VerticalPodAutoscaler.

2. VPA Updater: This component is responsible for applying the recommendations from the Recommender. It checks if any running Pods have outdated resource requests and, if so, evicts them so that they can be recreated with the updated requests. The Updater only evicts a Pod if the VPA’s updateMode is set to Auto or Recreate.

3. VPA Admission Controller: This is a webhook that intercepts new Pods before they are created. It checks if there is a matching VPA configuration for the Pod and, if so, overwrites the resource requests of the containers with the recommended values from the VPA. This is how the VPA ensures that new Pods are created with the correct resource requests.

Here’s a diagram illustrating the interaction between these components:

          +--------------------+
          |   VPA Recommender  |
          +--------------------+
                  |
                  | 1. Monitors Pod usage
                  | 2. Generates recommendations
                  v
          +--------------------+
          | VPA Custom Resource|
          +--------------------+
                  ^
                  | 3. Updater evicts Pods
                  |    with outdated requests
                  |
          +--------------------+
          |     VPA Updater    |
          +--------------------+
                  |
                  | 4. Admission Controller
                  |    overwrites requests
                  |    on new Pods
                  v
          +--------------------+
          | New Pod Creation   |
          +--------------------+

ResourceQuotas and LimitRanges: The Gatekeepers

ResourceQuotas and LimitRanges are enforced by the Kubernetes API server at admission time. When you create a new resource (like a Pod or a PersistentVolumeClaim), the API server checks if the creation of that resource would violate any of the quotas for the namespace. If it would, the API server rejects the request.

Here’s an example of a ResourceQuota that limits the total amount of CPU and memory that can be used by a namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-quota
  namespace: my-namespace
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi

And here’s an example of a LimitRange that sets default resource requests and limits for containers in a namespace:

apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: my-namespace
spec:
  limits:
  - default:
      memory: 512Mi
      cpu: 250m
    defaultRequest:
      memory: 256Mi
      cpu: 100m
    type: Container

By using these two resources together, you can create a robust system for managing resource usage in a multi-tenant cluster. You can set a global quota for each namespace to prevent any one team from consuming too many resources, and you can use LimitRanges to ensure that every container has a reasonable set of resource constraints.

Hands-On: Taming the Cloud Bill with Goldilocks, VPA, and ResourceQuotas

Theory is great, but there’s no substitute for hands-on experience. In this section, we’ll walk through a practical example of how to use Goldilocks, the VPA, and ResourceQuotas to optimize a sample application running on minikube.

Prerequisites

Before we begin, make sure you have the following tools installed on your macOS machine:

• minikube
• kubectl
• Helm

Step 1: Start Minikube

First, let’s start a minikube cluster:

minikube start --cpus=4 --memory=8192

This will create a single-node Kubernetes cluster with 4 CPU cores and 8GB of memory.

Step 2: Deploy a Sample Application

Next, let’s deploy a sample application to our cluster. We’ll use a simple Nginx deployment with no resource requests or limits specified.

Create a file called nginx-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

Now, apply the deployment to your cluster:

kubectl apply -f nginx-deployment.yaml

Step 3: Deploy Goldilocks

Now it’s time to bring in the experts. We’ll use Helm to deploy Goldilocks to our cluster.

First, add the Fairwinds Ops Helm repository:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm repo update

Next, install Goldilocks in a new namespace called goldilocks:

helm install goldilocks fairwinds-stable/goldilocks --namespace goldilocks --create-namespace

This will deploy the Goldilocks controller and dashboard to your cluster.

Step 4: Annotate Your Deployment

Goldilocks works by looking for deployments that have a specific annotation. Let’s add this annotation to our Nginx deployment:

kubectl annotate deployment nginx-deployment goldilocks.fairwinds.com/enabled=true

This tells Goldilocks to start monitoring our Nginx deployment and generate resource recommendations.

Step 5: Analyze the Recommendations

After a few minutes, Goldilocks will have collected enough data to generate some recommendations. To view the recommendations, you can forward the Goldilocks dashboard to your local machine:

kubectl -n goldilocks port-forward svc/goldilocks-dashboard 8080:80

Now, open your web browser and navigate to http://localhost:8080. You should see a dashboard that looks something like this:

Click on the nginx-deployment to see the resource recommendations. Goldilocks will provide recommendations for both requests and limits, as well as a “Quality of Service” (QoS) class for your Pods.

Step 6: Set Up the Vertical Pod Autoscaler (VPA)

Now that we have some baseline recommendations from Goldilocks, let’s set up the VPA to automatically adjust the resource requests of our containers.

First, you need to install the VPA components in your cluster. The installation process can be a bit involved, so we’ll use the official installation script from the Kubernetes autoscaler repository:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

This will deploy the VPA Recommender, Updater, and Admission Controller to your cluster.

Next, let’s create a VPA resource for our Nginx deployment. Create a file called nginx-vpa.yaml with the following content:

apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       nginx-deployment
  updatePolicy:
    updateMode: "Auto"

Now, apply the VPA to your cluster:

kubectl apply -f nginx-vpa.yaml

This tells the VPA to monitor our Nginx deployment and automatically update the resource requests of the containers based on their actual usage.

Step 7: Configure ResourceQuotas

Finally, let’s create a ResourceQuota to limit the total amount of resources that can be consumed by our default namespace.

Create a file called quota.yaml with the following content:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi

Now, apply the quota to your cluster:

kubectl apply -f quota.yaml

This will create a ResourceQuota that limits the default namespace to a maximum of 1 CPU core and 1Gi of memory for requests, and 2 CPU cores and 2Gi of memory for limits. If you try to create a Pod that would exceed this quota, the request will be rejected.

Debugging and Troubleshooting

Even with the best tools, things can still go wrong. Here are some common issues you might encounter when working with these cost optimization tools and how to solve them:

• Cluster Autoscaler Not Scaling Up: If the Cluster Autoscaler isn’t adding new nodes when you expect it to, check the following:

  • Are there any Pods in the Pending state? If not, the autoscaler has no reason to scale up.
  • Check the Cluster Autoscaler logs for any errors. You can view the logs by running kubectl -n kube-system logs -f deployment/cluster-autoscaler.
  • Make sure your node groups have enough capacity to scale up. If you’ve hit the maximum size of your node group, the autoscaler won’t be able to add any more nodes.

• Cluster Autoscaler Not Scaling Down: If the Cluster Autoscaler isn’t removing nodes when you expect it to, check the following:

  • Is the node underutilized? The autoscaler will only remove a node if its utilization is below a certain threshold (the default is 50%).
  • Are there any Pods on the node that can’t be rescheduled? The autoscaler won’t remove a node if it’s running a Pod with a restrictive PodDisruptionBudget (PDB) or a Pod that uses local storage.

• VPA Not Updating Pods: If the VPA isn’t updating the resource requests of your Pods, check the following:

  • Is the updateMode of your VPA set to Auto or Recreate? If it’s set to Off, the VPA will only provide recommendations and won’t actually update your Pods.
  • Check the VPA Updater logs for any errors. You can view the logs by running kubectl -n kube-system logs -f deployment/vpa-updater.

• Goldilocks Not Showing Recommendations: If you’re not seeing any recommendations in the Goldilocks dashboard, check the following:

  • Is your deployment annotated with goldilocks.fairwinds.com/enabled=true? Goldilocks will only monitor deployments that have this annotation.
  • Check the Goldilocks controller logs for any errors. You can view the logs by running kubectl -n goldilocks logs -f deployment/goldilocks-controller.

Key Takeaways

After a week of intense research and experimentation, Alex had a much clearer picture of NovaCraft’s Kubernetes economy. Here are the key takeaways from his journey:

• Cost optimization is a continuous journey, not a destination. It requires constant monitoring, analysis, and adjustment.
• The three pillars of Kubernetes cost are compute, storage, and networking. To control your costs, you need to have a handle on all three.
• Right-sizing is the name of the game. Overprovisioning is the single biggest source of waste in Kubernetes. Use tools like Goldilocks to find the “just right” values for your resource requests and limits.
• Automate, automate, automate. The Cluster Autoscaler and VPA are your best friends when it comes to automating resource management. Let them do the heavy lifting so you can focus on more important things.
• Visibility is key. You can’t optimize what you can’t measure. Use tools like Kubecost to get a detailed breakdown of your cluster costs.
• In a multi-tenant world, rules are your friend. Use ResourceQuotas and LimitRanges to ensure fair play and prevent any one team from hogging all the resources.

The New Frontier

As Alex presented his findings to Sarah, he felt a sense of pride and accomplishment. He had not only answered her question about the rising cloud bill, but he had also laid the foundation for a more cost-effective and efficient platform. He had a plan, he had the tools, and he had the knowledge to execute it.

Sarah was impressed. “This is exactly what I was looking for, Alex,” she said, a genuine smile on her face. “This is a solid plan. Let’s touch base in a month and see how these changes are impacting the bill.”

As the call ended, Alex looked at his Grafana dashboard again. The charts and graphs no longer filled him with a sense of dread. Instead, he saw them as a challenge, a puzzle to be solved. He had tamed the beast of the cloud bill, and in doing so, he had unlocked a new level of mastery over the Kubernetes platform.

But as one challenge was conquered, another one loomed on the horizon. A message from the security team popped up on his screen: “Alex, we need to talk about cluster security. We’ve detected some suspicious activity in one of the namespaces.”

Alex sighed, but it was a sigh of anticipation, not exhaustion. The journey of a platform lead was never-ending. There was always a new frontier to explore, a new challenge to overcome. And Alex was ready for it.

Next up in Season 2: Part 8: “The Guardians” — Deep Dive into Kubernetes Security.

This is Part 7 of Season 2 of The Container Odyssey.