Part 9: "The Federation" — Multi-Cluster Strategies and Disaster Recovery

This is Part 9 of Season 2 of The Container Odyssey.

The Story Opening: A Single Point of Failure

Alex, now six months into the Platform Lead role at NovaCraft, stared at the monitoring dashboard. A sea of green metrics stared back, a testament to the stability of the Kubernetes platform they had painstakingly built. The 10+ engineering teams were shipping features faster than ever, and the platform was humming along. But a nagging feeling of dread was growing in the pit of Alex’s stomach. NovaCraft was growing, and with that growth came new challenges. The single, monolithic Kubernetes cluster, once a symbol of their success, was now starting to feel like a single point of failure.

The wake-up call came during a routine maintenance window. A seemingly innocuous network configuration change in the cloud provider’s environment caused a cascading failure, taking down the entire cluster for over an hour. The impact was immediate and severe. All of NovaCraft’s services were down, and the engineering teams were left scrambling. The post-mortem was a sobering experience. The incident highlighted the fragility of their single-cluster architecture and the urgent need for a more resilient and scalable solution.

Alex knew that the time had come to venture beyond the confines of a single cluster and explore the world of multi-cluster Kubernetes. The challenge was no longer about deploying applications, but about running them at scale, with high availability, and across multiple regions. The next chapter in NovaCraft’s container odyssey was about to begin.

Conceptual Deep-Dive: Beyond the Single Cluster

As NovaCraft’s story illustrates, a single Kubernetes cluster, no matter how well-managed, can become a bottleneck and a single point of failure. As organizations grow and their applications become more critical, the need for a multi-cluster strategy becomes increasingly apparent. Let’s explore the core concepts and architectures that underpin a multi-cluster world.

Why Go Multi-Cluster?

Adopting a multi-cluster architecture offers several compelling advantages. It provides high availability and disaster recovery by distributing applications across multiple clusters in different regions, allowing you to tolerate the failure of an entire cluster or region. A multi-cluster setup also enhances scalability and performance, as a single cluster has its limits. As the number of applications and users grows, you may hit scaling limits in terms of the number of nodes, pods, or services. A multi-cluster architecture allows you to scale horizontally by adding more clusters. Furthermore, it enables geographic distribution of your applications, reducing latency for a global user base. You can also achieve better isolation and security by using separate clusters for different environments or applications with varying security needs. Finally, a multi-cloud, multi-cluster strategy can help you avoid being locked into a single cloud provider.

Multi-Cluster Architectures

There are several common architectural patterns for managing multiple Kubernetes clusters. Let’s explore three of the most popular ones:

Kubernetes Federation (KubeFed)

Kubernetes Federation (KubeFed) is an open-source project that provides a way to manage multiple Kubernetes clusters from a single control plane. It allows you to create “federated” resources that are automatically propagated to all the clusters in the federation. For example, you can create a federated deployment, and KubeFed will create a corresponding deployment in each of the federated clusters.

Note: The original Kubernetes Federation v1 is no longer actively developed. The current version, KubeFed v2, is a complete rewrite and is the recommended way to implement federation. However, it’s important to note that the kubernetes-retired/kubefed repository has been archived, and the project is no longer under active development. While still a valuable concept to understand, other tools have emerged as more popular choices for multi-cluster management.

Multi-Cluster Service Discovery and Traffic Routing

In a multi-cluster environment, service discovery and traffic routing are more complex than in a single cluster. How does a service in one cluster discover and communicate with a service in another cluster? How do you route external traffic to the right cluster?

Several solutions exist to tackle the complexities of multi-cluster service discovery and traffic routing. The Kubernetes ecosystem is evolving to include multi-cluster services APIs, which will provide a standard way to expose services across clusters. A service mesh, such as Istio or Linkerd, can offer a unified control plane for managing traffic and security across multiple clusters. Cloud providers also offer global load balancers that can distribute traffic across clusters in different regions. Additionally, tools like ExternalDNS can be used to automatically create DNS records for your services, making them discoverable from outside the cluster.

Disaster Recovery with Velero

Velero is an open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. It’s an essential tool for disaster recovery in a multi-cluster environment. With Velero, you can take a snapshot of your entire cluster, including all the resources and persistent volumes, and store it in a safe location like an S3 bucket. If a disaster strikes, you can use the backup to restore your cluster to its previous state.

Cross-Cluster Networking

Enabling communication between pods in different clusters is another critical aspect of a multi-cluster architecture. You can establish secure connections between the networks of your clusters using VPNs or tunnels. Cloud providers also offer various networking solutions that can be used to connect clusters in different regions. Additionally, open-source projects like Submariner provide a dedicated solution for connecting the networks of multiple Kubernetes clusters.

GitOps for Multi-Cluster with ArgoCD ApplicationSets

GitOps is a modern approach to continuous delivery that uses Git as the single source of truth for declarative infrastructure and applications. ArgoCD is a popular GitOps tool for Kubernetes. ApplicationSets are a feature of ArgoCD that allows you to automate the creation of applications across multiple clusters. With ApplicationSets, you can define a template for your application and then use a generator to create applications for each of your clusters. This makes it easy to manage and deploy applications to a large number of clusters in a consistent and automated way.

Technical Explanation: Under the Hood

Now that we have a high-level understanding of the concepts, let’s dive deeper into the technical details of how these multi-cluster components work.

Velero Architecture

Velero’s architecture is composed of two primary components: the Velero server and the Velero CLI. The Velero server runs as a deployment within your Kubernetes cluster and is responsible for executing backup and restore operations. The Velero CLI is a command-line tool that you use to interact with the Velero server.

When you initiate a backup, the Velero server communicates with the Kubernetes API server to query the resources you want to back up. It then creates a tarball of the resources and stores it in a backend storage location, such as an S3 bucket. For persistent volumes, Velero uses the cloud provider’s snapshot APIs to create a snapshot of the volume.

When you restore a backup, the Velero server retrieves the tarball from the backend storage and creates the resources in your cluster. For persistent volumes, it creates a new volume from the snapshot and attaches it to the appropriate pod.

ArgoCD ApplicationSet Controller

The ApplicationSet controller is a separate component that you install in your cluster alongside ArgoCD. It introduces a new custom resource called ApplicationSet. The ApplicationSet resource defines a template for an ArgoCD application and a generator that creates applications based on that template.

The ApplicationSet controller offers several types of generators to create applications. The List Generator creates applications from a static list of clusters. The Cluster Generator automatically discovers clusters that are registered with ArgoCD and creates applications for them. The Git Generator discovers clusters from a Git repository.

When the ApplicationSet controller detects a new ApplicationSet resource, it uses the generator to create a list of parameters. It then combines the parameters with the application template to generate a set of ArgoCD Application resources. These Application resources are then managed by ArgoCD just like any other application.

Step-by-Step Hands-On: Multi-Cluster Mayhem

Now, let’s get our hands dirty. In this tutorial, we’ll simulate a multi-cluster environment on our local machine using minikube. We’ll set up two clusters, cluster-1 and cluster-2. We’ll then use Velero to back up an application from cluster-1 and restore it to cluster-2. Finally, we’ll use ArgoCD ApplicationSets to deploy an application to both clusters simultaneously.

Prerequisites

• minikube
• kubectl
• helm
• argocd cli

Setting Up the Environment

First, let’s create our two minikube clusters:

minikube start -p cluster-1
minikube start -p cluster-2

Now, let’s switch to cluster-1:

kubeconfig use-context cluster-1

Backing Up and Restoring with Velero

Installing Velero

We’ll install Velero using the official Helm chart. For this tutorial, we’ll use the built-in MinIO object store that comes with the Velero installation.

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm install velero vmware-tanzu/velero --namespace velero --create-namespace --set-string snapshots.enabled=false --set-string deployRestic=false --set-string deployNodeAgent=false --set-string configuration.provider=aws --set-string configuration.backupStorageLocation.name=default --set-string configuration.backupStorageLocation.bucket=velero --set-string configuration.backupStorageLocation.config.region=minio --set-string configuration.backupStorageLocation.config.s3ForcePathStyle=true --set-string configuration.backupStorageLocation.config.s3Url=http://minio.velero.svc:9000 --set-string image.repository=velero/velero --set-string image.pullPolicy=IfNotPresent --set-string initContainers[0].name=velero-plugin-for-aws --set-string initContainers[0].image=velero/velero-plugin-for-aws:v1.2.1 --set-string initContainers[0].volumeMounts[0].mountPath=/target --set-string initContainers[0].volumeMounts[0].name=plugins

Deploying a Sample Application

Now, let’s deploy a sample application to cluster-1 that we can back up.

kubectl create namespace nginx-example
kubectl -n nginx-example create deployment nginx --image=nginx

Creating a Backup

Now, let’s create a backup of the nginx-example namespace.

velero backup create nginx-backup --include-namespaces nginx-example

You can check the status of the backup with the following command:

velero backup describe nginx-backup

Restoring to a New Cluster

Now, let’s switch to cluster-2 and restore our application.

kubectl config use-context cluster-2

First, we need to install Velero on cluster-2 with the same configuration as cluster-1.

helm install velero vmware-tanzu/velero --namespace velero --create-namespace --set-string snapshots.enabled=false --set-string deployRestic=false --set-string deployNodeAgent=false --set-string configuration.provider=aws --set-string configuration.backupStorageLocation.name=default --set-string configuration.backupStorageLocation.bucket=velero --set-string configuration.backupStorageLocation.config.region=minio --set-string configuration.backupStorageLocation.config.s3ForcePathStyle=true --set-string configuration.backupStorageLocation.config.s3Url=http://minio.velero.svc:9000 --set-string image.repository=velero/velero --set-string image.pullPolicy=IfNotPresent --set-string initContainers[0].name=velero-plugin-for-aws --set-string initContainers[0].image=velero/velero-plugin-for-aws:v1.2.1 --set-string initContainers[0].volumeMounts[0].mountPath=/target --set-string initContainers[0].volumeMounts[0].name=plugins

Now, we can restore the backup.

velero restore create --from-backup nginx-backup

You can check the status of the restore with the following command:

velero restore describe nginx-backup-20260228-....

Once the restore is complete, you should see the nginx-example namespace and the nginx deployment in cluster-2.

kubectl get deployments -n nginx-example

Deploying with ArgoCD ApplicationSets

Now, let’s use ArgoCD ApplicationSets to deploy an application to both of our clusters.

Installing ArgoCD and the ApplicationSet Controller

First, let’s install ArgoCD and the ApplicationSet controller on cluster-1.

kubectl config use-context cluster-1
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/applicationset/v0.4.1/manifests/install.yaml

Adding Clusters to ArgoCD

Now, we need to add our two clusters to ArgoCD. First, let’s get the credentials for cluster-2.

KUBECONFIG=~/.kube/config kubectl config view --flatten --minify -o jsonpath='{.clusters[?(@.name=="cluster-2")].cluster.server}'

Now, let’s add cluster-2 to ArgoCD.

argocd cluster add cluster-2 --name cluster-2 --server <server-url-from-previous-command>

Creating an ApplicationSet

Now, let’s create an ApplicationSet to deploy a sample application to both of our clusters. Create a file called guestbook-appset.yaml with the following content:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: cluster-1
        url: https://kubernetes.default.svc
      - cluster: cluster-2
        url: <server-url-for-cluster-2>
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: HEAD
        path: guestbook
      destination:
        server: '{{url}}'
        namespace: guestbook

Now, apply the ApplicationSet:

kubectl apply -f guestbook-appset.yaml

ArgoCD will now create two applications, one for each cluster. You can see the applications in the ArgoCD UI. To access the UI, run the following command:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Then, open your browser to https://localhost:8080. The username is admin and the password can be retrieved with the following command:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Once you log in, you should see two guestbook applications, one for each cluster. This demonstrates how you can use ApplicationSets to easily manage and deploy applications to multiple clusters.

Debugging/Troubleshooting Tips

Here are some common issues you might encounter when working with multi-cluster Kubernetes and how to solve them:

• Velero backup/restore fails:
  • Check the Velero server logs for errors: kubectl logs -n velero deployment/velero
  • Ensure that the storage provider credentials are correct.
  • Make sure that the Velero server has the necessary permissions to access the Kubernetes API and the storage backend.
• ArgoCD ApplicationSet not generating applications:
  • Check the ApplicationSet controller logs for errors: kubectl logs -n argocd deployment/argocd-applicationset-controller
  • Verify that the ApplicationSet YAML is correct and that the generator is configured properly.
  • Make sure that the clusters are correctly registered with ArgoCD.
• Cross-cluster connectivity issues:
  • Ensure that the firewall rules and security groups are configured to allow traffic between the clusters.
  • Use tools like ping and traceroute to diagnose network connectivity issues.
  • If you are using a service mesh, check the service mesh configuration to ensure that it is correctly configured for multi-cluster communication.

Key Takeaways

• A multi-cluster Kubernetes strategy is essential for high availability, scalability, and disaster recovery.
• There are several multi-cluster architectures to choose from, including active-active, active-passive, and hub-spoke.
• Velero is a powerful tool for backing up and restoring Kubernetes clusters.
• ArgoCD ApplicationSets provide an easy way to manage and deploy applications to multiple clusters.
• Multi-cluster Kubernetes introduces new challenges, such as service discovery, traffic routing, and cross-cluster networking.

Story Closing + Teaser

Alex watched as the guestbook application was successfully deployed to both cluster-1 and cluster-2. The ArgoCD dashboard glowed with the satisfying green of a successful sync. They had taken their first steps into a larger world, a world of federated clusters and resilient systems. The single point of failure that had once been a source of anxiety was now a relic of the past. NovaCraft was no longer just a single, isolated island; it was the beginning of an archipelago, a network of interconnected clusters ready to weather any storm.

But as Alex leaned back, a new thought began to form. They had conquered the challenge of deploying applications to multiple clusters, but what about the data? How could they ensure that the data was consistent across all the clusters? How could they build a truly stateful, multi-cluster application? The next chapter in the container odyssey would take them even deeper, into the world of distributed data and stateful workloads. The journey was far from over.

References

[1] Kubernetes Multi-Cluster [2] Kubernetes Federation (KubeFed) [3] Velero [4] ArgoCD ApplicationSet