Part 10: "The Horizon" — Platform Engineering, Developer Experience, and What's Next

Six months. It felt like a lifetime ago that Alex was wrestling with YAML manifests, trying to get a single microservice to reliably talk to another. Now, standing in front of a panoramic whiteboard, the challenges weren’t about a single pod, but the entire universe of them. Alex, now the newly minted “Platform Lead” at NovaCraft, was no longer just a passenger on the container ship; Alex was helping design the fleet.

The promotion had been a whirlwind. One moment, Alex was the go-to person for “anything Kubernetes”; the next, Alex was responsible for the productivity and happiness of over ten engineering teams. The questions had changed. It wasn’t “How do I deploy my app?” anymore. It was “Why is my build taking 20 minutes?”, “How can I get a staging environment without waiting two days?”, “Is our cluster secure?”, and the one that kept Alex up at night: “Are we spending a fortune on cloud bills without even knowing it?”

This was the world of Platform Engineering. It was no longer about individual deployments but about building a paved road for all developers at NovaCraft. A road that was smooth, fast, secure, and didn’t have unexpected tolls. The goal was to create a seamless Developer Experience (DevEx), making the complex machinery of Kubernetes disappear behind a curtain of elegant, self-service tooling. Alex took a deep breath, picked up a marker, and wrote on the board: “Internal Developer Platform.” The journey beyond the cluster had truly begun.

The Rise of the Internal Developer Platform (IDP)

As NovaCraft scaled, the raw power of kubectl that once felt liberating started to feel like a bottleneck. Every new service, every environment, every database connection required a bespoke set of commands and YAML files. The cognitive load on developers was immense. This is the exact problem that an Internal Developer Platform (IDP) is designed to solve.

An IDP is a layer of tooling and automation that sits on top of your underlying infrastructure (like Kubernetes) to provide a cohesive, self-service experience for developers. It codifies and automates the entire lifecycle of a service, from creation to deployment to monitoring.

Think of it like this: Kubernetes is the engine, the transmission, and the chassis of a car. It’s incredibly powerful, but you wouldn’t expect every driver to know how to assemble it from scratch. An IDP is the dashboard, the steering wheel, and the pedals—the user-friendly interface that lets you drive the car without needing to be a mechanic. It provides a “golden path” for developers, abstracting away the complexity without sacrificing the power.

Two of the most prominent players in the open-source IDP space are Backstage and Port.

For NovaCraft, Alex saw the appeal of Backstage. Its plugin architecture meant they could start small, with a simple service catalog, and gradually add more capabilities like CI/CD visibility, security scanning, and cost management, all in one place.

From kubectl to Self-Service Portals

The shift to an IDP is a fundamental change in developer experience. Instead of a command-line-driven workflow, developers interact with a web-based portal. This has profound implications:

• Reduced Cognitive Load: Developers no longer need to be Kubernetes experts to do their jobs. They can provision a new microservice, a database, or a CI/CD pipeline with a few clicks in a web UI.
• Increased Standardization: The IDP enforces best practices for security, observability, and deployment. Every new service created through the portal is automatically configured with the right labels, annotations, and monitoring dashboards.
• Faster Onboarding: New engineers can become productive in days, not weeks. They don’t need to spend time learning the intricacies of the underlying infrastructure.

This is the future of software development at scale. It’s not about taking power away from developers, but about enabling them to move faster and focus on what they do best: writing code.

The Unseen Revolution: eBPF, Cilium, and the Future of the Kernel

As Alex dug deeper into building a robust platform, a new technology kept surfacing in discussions about networking, security, and observability: eBPF. It stands for extended Berkeley Packet Filter, but its name belies its revolutionary potential. Initially designed for filtering network packets, eBPF has evolved into a general-purpose, event-driven programming framework inside the Linux kernel.

Imagine you could safely inject a small, sandboxed program directly into the kernel to run when specific events happen—like a system call, a network packet arriving, or a function being called. You could observe and manipulate the very fabric of the operating system without changing the kernel source code or loading clunky kernel modules. That’s eBPF.

For Kubernetes, this is a game-changer. It moves logic that was traditionally handled by sidecars or complex userspace daemons directly into the kernel, leading to massive performance and security gains.

Cilium: Networking and Security on eBPF Steroids

One of the most mature and widely adopted projects built on eBPF is Cilium. It completely reimagines Kubernetes networking.

Cilium is an open-source project that provides networking, observability, and security for container workloads using eBPF. It replaces traditional iptables-based networking with a much more efficient and powerful eBPF-based datapath.

Here’s why this matters for a platform like NovaCraft’s:

For Alex, this meant that instead of just seeing Pod A (10.0.1.23) talking to Pod B (10.0.2.45), they could see service: frontend making a GET /api/users call to service: user-api. This level of visibility, provided by projects like Tetragon (also from the creators of Cilium), is invaluable for troubleshooting and security enforcement.

The Next Wave: WebAssembly (Wasm) on Kubernetes

While containers revolutionized how we package and run applications, another technology is emerging that could redefine the future of cloud-native computing: WebAssembly (Wasm).

Originally created to run code in web browsers at near-native speed, Wasm has a set of properties that make it incredibly compelling for server-side applications, especially within a Kubernetes environment:

• Sandboxed by Default: Every Wasm module runs in a lightweight, secure sandbox. It has no access to the host system unless capabilities are explicitly granted. This provides a much stronger security boundary than traditional containers.
• Lightning-Fast Start-up: Wasm modules can start in microseconds, compared to the seconds it can take to start a container. This is ideal for bursty, event-driven workloads and serverless functions.
• Polyglot and Portable: You can compile code from languages like Rust, Go, C++, and Python into a single, portable .wasm binary that runs on any compliant Wasm runtime, on any architecture.

So, where does this fit into the Kubernetes world? Instead of running a container image that contains an entire operating system filesystem, you can run a Wasm module directly. This is made possible by projects that integrate Wasm runtimes with the Kubelet, using technologies like crun or runwasi.

The promise of Wasm on Kubernetes is to run workloads that are smaller, faster, and more secure than containers, making it a perfect fit for edge computing, serverless, and untrusted plugin systems.

For Alex’s platform team, this is a glimpse into the future. Imagine allowing other teams at NovaCraft to extend the platform with their own plugins. Running those plugins as Wasm modules would be far more secure than running them as containers with broad permissions. Projects like WasmEdge and WasmCloud are pioneering this space, building frameworks for creating distributed, stateful applications with Wasm and Kubernetes.

Powering Intelligence: AI/ML Workloads on Kubernetes

NovaCraft’s business was increasingly driven by data. The data science team was constantly experimenting with new models, and they needed a way to train and serve these models at scale. Naturally, they turned to Alex’s platform team, and Kubernetes was the answer.

Running AI and Machine Learning (ML) workloads on Kubernetes presents a unique set of challenges:

• Hardware Acceleration: Many ML training jobs require specialized hardware like GPUs (Graphics Processing Units). How do you make these expensive resources available to containerized workloads?
• Complex Workflows: A typical ML workflow involves multiple stages: data ingestion, preprocessing, model training, validation, and serving. This isn’t a simple stateless application; it’s a complex, multi-step pipeline.
• Resource Management: Training jobs can be long-running and resource-intensive. How do you schedule them efficiently without starving other applications on the cluster?

GPU Scheduling and the NVIDIA Device Plugin

To address the hardware challenge, Kubernetes has a Device Plugin framework. This allows hardware vendors to create plugins that expose custom resources to the cluster. The most common one is the NVIDIA Device Plugin, which discovers the GPUs on a node and makes them available for pods to request.

A developer can then request a GPU in their pod spec just like they would CPU or memory:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      image: "nvidia/cuda:11.6.2-base-ubuntu20.04"
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU

This simple declaration is the key that unlocks hardware acceleration for containerized workloads.

Orchestrating the Pipeline with KubeFlow

To manage the entire ML lifecycle, a higher-level framework is needed. This is where KubeFlow comes in.

KubeFlow is an open-source ML toolkit for Kubernetes. It aims to make deploying ML workflows on Kubernetes simple, portable, and scalable. It provides a curated set of tools and frameworks for each stage of the ML lifecycle.

KubeFlow isn’t a single monolithic application; it’s a collection of powerful components, including:

• KubeFlow Pipelines: For building and managing complex, multi-step ML workflows.
• TFJob, PyTorchJob: Custom controllers that make it easy to run distributed training jobs with TensorFlow and PyTorch.
• Katib: For hyperparameter tuning and neural architecture search.
• KServe (formerly KFServing): For serving models with advanced features like canary deployments and autoscaling.

For Alex and the NovaCraft team, KubeFlow provided the perfect abstraction layer. The data scientists could define their pipelines in Python, and KubeFlow would handle the heavy lifting of creating the necessary Kubernetes resources, scheduling the jobs, and managing the entire workflow. This allowed the data science team to focus on building models, not on wrestling with infrastructure.

Navigating the Infinite Galaxy: The CNCF Landscape

As Alex explored the vast ecosystem of cloud-native tools, one organization stood at the center of it all: the Cloud Native Computing Foundation (CNCF). The CNCF is the vendor-neutral home for many of the open-source projects that define the cloud-native world, including Kubernetes itself, Prometheus, Envoy, and many more.

The CNCF provides a curated path for projects as they mature, from Sandbox (for early-stage, experimental projects) to Incubating (for projects with growing adoption and a healthy community) to Graduated (for projects that are considered stable, mature, and battle-tested).

For a platform leader like Alex, the CNCF landscape is both a treasure trove and a potential minefield. The sheer number of projects can be overwhelming. The famous CNCF Landscape diagram is a testament to the explosive growth of the ecosystem.

Navigating the CNCF landscape requires a strategic approach. It’s not about adopting the newest, shiniest tool. It’s about identifying the specific problems your platform needs to solve and then finding the right mature, well-supported project to solve it.

Here’s a mental model Alex developed for evaluating CNCF projects:

1. Problem First, Tool Second: Don’t start with a tool. Start with a clear definition of the problem. Are you trying to improve observability? Secure your software supply chain? Automate deployments?
2. Start with Graduated Projects: For critical platform components, favor Graduated projects. They have a proven track record, a large community, and long-term support.
3. Look for Integration: The power of the CNCF ecosystem lies in how the projects work together. Does the tool you’re considering integrate well with the other components of your platform (like Prometheus for metrics or Fluentd for logging)?
4. Assess Community Health: Look at the project’s GitHub repository. Is it active? Are issues being addressed? Is there a vibrant community of users and contributors?
5. Consider the Cost of Ownership: Open source is free to use, but it’s not free to operate. Factor in the cost of learning, maintaining, and supporting the tool.

By applying this framework, Alex could confidently navigate the CNCF landscape, choosing the right tools to build a platform that was both powerful and sustainable.

Hands-On: Your First Internal Developer Platform

Talk is cheap. Let’s build something. In this hands-on section, we’ll deploy a basic Backstage instance to our local minikube cluster. This will serve as the foundation for NovaCraft’s new Internal Developer Platform.

Prerequisites:

• A running minikube cluster (minikube start)
• kubectl installed and configured to talk to your minikube cluster.

Step 1: Create a Namespace

First, let’s create a dedicated namespace for our Backstage installation to keep things organized.

kubectl create namespace backstage

Step 2: Deploy a PostgreSQL Database

Backstage needs a database. We’ll deploy a simple PostgreSQL instance within our cluster.

1. Create the Database Secret

Create a file named postgres-secrets.yaml with the following content. This will hold our database credentials. Remember, in a real-world scenario, you would use a more secure way to manage secrets!

apiVersion: v1
kind: Secret
metadata:
  name: postgres-secrets
  namespace: backstage
type: Opaque
data:
  POSTGRES_USER: YmFja3N0YWdl
  POSTGRES_PASSWORD: aHVudGVyMg==

The values are Base64 encoded. YmFja3N0YWdl is backstage and aHVudGVyMg== is hunter2.

Apply the secret:

kubectl apply -f postgres-secrets.yaml

2. Create the Persistent Volume

Next, we need storage for our database. Create a file named postgres-storage.yaml.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-storage
  namespace: backstage
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-storage-claim
  namespace: backstage
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Apply the storage:

kubectl apply -f postgres-storage.yaml

3. Create the PostgreSQL Deployment

Now, create the PostgreSQL deployment itself. Create a file named postgres-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:13-alpine
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          envFrom:
            - secretRef:
                name: postgres-secrets
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgresdb
      volumes:
        - name: postgresdb
          persistentVolumeClaim:
            claimName: postgres-storage-claim

Apply the deployment:

kubectl apply -f postgres-deployment.yaml

4. Create the PostgreSQL Service

Finally, expose the PostgreSQL deployment with a service so Backstage can connect to it. Create a file named postgres-service.yaml.

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: backstage
spec:
  selector:
    app: postgres
  ports:
    - port: 5432

Apply the service:

kubectl apply -f postgres-service.yaml

You now have a running PostgreSQL database in your cluster!

Step 3: Deploy Backstage

Now for the main event. We need to create a Backstage application, containerize it, and deploy it.

1. Create a Backstage App

On your local machine, create a new Backstage app. This requires Node.js.

npx @backstage/create-app@latest
# Follow the prompts to create the app
cd your-app-name

2. Configure the App for Kubernetes

You’ll need to edit the app-config.yaml file to connect to the PostgreSQL database we just created. Update the database section:

database:
  client: pg
  connection:
    host: postgres.backstage
    port: 5432
    user: ${POSTGRES_USER}
    password: ${POSTGRES_PASSWORD}

3. Build the Backstage Docker Image

From your app’s directory, build the Docker image. This command packages the backend and frontend into a single container.

yarn install
yarn tsc
yarn build:backend
# This creates a Dockerfile in packages/backend/dist
cd packages/backend
yarn build-image

4. Create the Backstage Deployment

Now, create a backstage-deployment.yaml file. Make sure to replace your-backstage-image with the name of the image you just built.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backstage
  namespace: backstage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backstage
  template:
    metadata:
      labels:
        app: backstage
    spec:
      containers:
        - name: backstage
          image: your-backstage-image:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 7007
          envFrom:
            - secretRef:
                name: postgres-secrets

Apply the deployment:

kubectl apply -f backstage-deployment.yaml

5. Create the Backstage Service

Finally, expose the Backstage deployment. Create a backstage-service.yaml file.

apiVersion: v1
kind: Service
metadata:
  name: backstage
  namespace: backstage
spec:
  type: NodePort
  selector:
    app: backstage
  ports:
    - port: 80
      targetPort: 7007
      nodePort: 30007

Apply the service:

kubectl apply -f backstage-service.yaml

Step 4: Access Your Backstage Portal

Your Backstage instance is now running! To access it, you’ll need to get your minikube IP and the NodePort.

minikube service backstage --url -n backstage

Open the URL in your browser. You should see the Backstage UI. Congratulations, you’ve deployed your first Internal Developer Platform!

From Monolith Engineer to Platform Leader

Alex stared at the Backstage UI, a simple catalog of NovaCraft’s services. It was a small start, but it was a foundation. The journey from wrestling with a single monolith to orchestrating a platform for an entire company had been a profound one. It was a shift in perspective, from code to systems, from features to capabilities.

The world of Kubernetes was vast and ever-expanding. eBPF was changing the rules of networking, Wasm was promising a new era of lightweight, secure workloads, and the CNCF landscape was a constant source of innovation. It was impossible to know everything, but that wasn’t the goal. The goal was to build a platform that could evolve, a platform that empowered developers, a platform that could carry NovaCraft into the future.

Alex smiled. The horizon was vast, but for the first time, it felt within reach.

Key Takeaways

• Platform Engineering is about building a paved road for developers, abstracting away the complexity of the underlying infrastructure.
• Internal Developer Platforms (IDPs) like Backstage provide a self-service portal for developers, increasing standardization and reducing cognitive load.
• eBPF is a revolutionary technology that is reshaping Kubernetes networking, security, and observability, with projects like Cilium leading the way.
• WebAssembly (Wasm) offers a new paradigm for running code on Kubernetes that is faster, smaller, and more secure than traditional containers.
• KubeFlow provides a comprehensive toolkit for running complex AI/ML workloads on Kubernetes, from GPU scheduling to pipeline orchestration.
• The CNCF landscape is a rich ecosystem of tools, but it requires a strategic approach to navigate effectively.

This is Part 10 of Season 2 of The Container Odyssey.

Debugging and Troubleshooting

When you are building your platform, things will inevitably go wrong. Here are some common issues you might encounter when working with the technologies in this chapter:

• Backstage Deployment:
  • Database Connection Issues: If your Backstage pod is crash-looping, the most likely culprit is a database connection problem. Use kubectl logs -f <backstage-pod-name> -n backstage to check the logs. Ensure your app-config.yaml is pointing to the correct service name (postgres.backstage) and that the POSTGRES_USER and POSTGRES_PASSWORD environment variables are correctly being passed from the secret.
  • ImagePullBackOff: If you see this error, it means Kubernetes can’t pull your Backstage Docker image. Make sure you have loaded the image into your minikube’s Docker daemon using minikube image load your-backstage-image:latest.
• Cilium/eBPF:
  • Connectivity Issues: If pods can’t communicate after installing Cilium, use the cilium status and cilium connectivity test commands to diagnose the health of the Cilium deployment and run a connectivity test suite.
• General Minikube Tips:
  • Resource Issues: Minikube can be resource-intensive. If your cluster is slow or pods are getting evicted, you may need to allocate more CPU and memory to the minikube VM with minikube start --cpus 4 --memory 8192.
  • Resetting: Sometimes, the easiest way to fix a broken minikube cluster is to start over. minikube delete will destroy the cluster, allowing you to start fresh with minikube start.