The Container Odyssey: Beyond the Cluster: Season 2
Season 2 of The Container Odyssey: a 10-part series for engineers who know core Kubernetes and need the broader ecosystem—multi-cluster management, service mesh, observability, secrets, policy, cost, operators, DR, and platform engineering.
The Container Odyssey — Season 2: Beyond the Cluster
A Senior Engineer’s Journey into the Kubernetes Ecosystem
About This Series
This is the second season of “The Container Odyssey,” a 10-part Kubernetes tutorial series written for experienced software engineers who have mastered core Kubernetes concepts and are now ready to tackle the broader ecosystem. It continues the story of Alex, who has been promoted to Platform Lead at NovaCraft and now faces the challenges of running Kubernetes at scale for 10+ engineering teams.
Season 1 covered the foundations: Pods, Deployments, Services, ConfigMaps, Secrets, Storage, RBAC, Helm, and CI/CD. Season 2 goes beyond the cluster itself, into the tools, patterns, and practices that make Kubernetes production-grade at scale.
Prerequisites
Before starting this series, you should have completed Season 1 or have equivalent knowledge of the following core Kubernetes concepts.
| Concept | Season 1 Coverage |
|---|---|
| Pods, Deployments, ReplicaSets | Parts 3-4 |
| Services, Ingress, Networking | Part 5 |
| ConfigMaps, Secrets | Part 6 |
| PersistentVolumes, StatefulSets | Part 7 |
| Health Probes, Resource Limits, HPA | Part 8 |
| Jobs, CronJobs, DaemonSets | Part 9 |
| RBAC, Network Policies, Security | Part 10 |
| Helm, Kustomize | Part 11 |
| CI/CD, GitOps with ArgoCD | Part 12 |
Additional tools you will install during this series:
| Tool | Chapter | Purpose |
|---|---|---|
| Rancher | Part 1 | Multi-cluster management platform |
| Istio + Kiali | Part 2 | Service mesh and visualization |
| Prometheus + Grafana | Part 3 | Monitoring and dashboarding |
| Loki + Promtail | Part 4 | Centralized logging |
| HashiCorp Vault | Part 5 | Secret management |
| Kyverno | Part 6 | Policy enforcement |
| Goldilocks + VPA | Part 7 | Cost optimization and right-sizing |
| Kubebuilder | Part 8 | Building Kubernetes Operators |
| Velero | Part 9 | Backup, restore, and disaster recovery |
| Backstage | Part 10 | Internal Developer Platform |
Series Table of Contents
Platform Management (Part 1)
Part 1: “The Platform Lead” — Rancher and Multi-Cluster Management Alex is promoted to Platform Lead and must manage dev, staging, and production clusters. We explore Rancher’s architecture, compare it to Lens/k9s/Portainer, and walk through installing Rancher, importing a minikube cluster, and deploying an app through the UI.
Service Communication (Part 2)
Part 2: “The Mesh” — Service Mesh with Istio NovaCraft’s 50+ microservices are experiencing cascading failures. We dive into the service mesh concept, the sidecar proxy pattern with Envoy, Istio’s architecture, traffic management (routing, retries, circuit breaking, fault injection), mutual TLS for zero-trust networking, and observability with Kiali.
Observability (Parts 3-4)
Part 3: “Eyes Everywhere” — Monitoring with Prometheus and Grafana A phantom latency issue that logs cannot explain. We cover the observability trinity (metrics, logs, traces), Prometheus’s pull-based architecture and PromQL, the Prometheus Operator with ServiceMonitors, Grafana dashboarding, and alerting pipelines to Slack/PagerDuty.
Part 4: “The Log Trail” — Centralized Logging with Loki and the EFK Stack A production bug scattered across 50 Pods. We compare the EFK stack (Elasticsearch, Fluentd, Kibana) with Grafana Loki + Promtail, cover log aggregation patterns, structured logging best practices, LogQL queries, and log-metric correlation.
Security (Parts 5-6)
Part 5: “The Vault” — Secret Management with HashiCorp Vault and Sealed Secrets A secret is committed to Git in plain text. We explore why K8s Secrets are not enough, HashiCorp Vault’s architecture (seal/unseal, auth methods, secret engines, dynamic secrets), the Vault Agent Injector, Sealed Secrets for GitOps, the External Secrets Operator, and when to use each.
Part 6: “The Gatekeeper” — Policy Engines with OPA/Gatekeeper and Kyverno A container running as root reaches production. We cover admission controllers, OPA with Gatekeeper and the Rego language, Kyverno’s Kubernetes-native YAML policies, common policy patterns, and a hands-on walkthrough of enforcing labels and blocking privileged containers.
Cost and Efficiency (Part 7)
Part 7: “The Bill” — Cost Optimization and Resource Efficiency The CFO asks why the cloud bill tripled. We break down Kubernetes cost drivers (compute, storage, networking), right-sizing with resource analysis, Cluster Autoscaler, spot instances, Kubecost and Goldilocks, VPA vs HPA, and ResourceQuotas for multi-tenant clusters.
Extending Kubernetes (Part 8)
Part 8: “The Operator” — Custom Resources and Operators NovaCraft needs to manage a complex stateful database as a first-class K8s citizen. We cover Custom Resource Definitions (CRDs), the Operator pattern, the reconciliation loop, Operator maturity levels, and build a simple operator using Kubebuilder.
Multi-Cluster and DR (Part 9)
Part 9: “The Federation” — Multi-Cluster Strategies and Disaster Recovery NovaCraft expands to multiple regions. We explore active-active, active-passive, and hub-spoke architectures, KubeFed concepts, multi-cluster service discovery, disaster recovery with Velero, cross-cluster networking, and GitOps for multi-cluster with ArgoCD ApplicationSets.
The Future (Part 10)
Part 10: “The Horizon” — Platform Engineering, Developer Experience, and What’s Next Alex reflects on the journey and looks ahead. We cover Internal Developer Platforms with Backstage, eBPF and Cilium for next-gen networking, WebAssembly on Kubernetes, AI/ML workloads with KubeFlow, and navigating the CNCF landscape.
How to Use This Series
Each chapter follows the same structure established in Season 1, designed to maximize learning for senior engineers.
Story Opening presents a real platform engineering challenge Alex faces, grounding the technical content in a relatable scenario. Conceptual Deep-Dive explains the “why” with analogies, architecture descriptions, and mental models. Technical Explanation covers the “how” with component interactions, data flows, and design decisions. Step-by-Step Hands-On provides practical, runnable examples on macOS with minikube. Debugging/Troubleshooting Tips covers common pitfalls specific to each topic. Key Takeaways summarizes the essentials. Story Closing + Teaser wraps up Alex’s progress and sets up the next chapter.
This is the index for Season 2 of “The Container Odyssey: A Senior Engineer’s Journey into the Kubernetes Ecosystem.”