Six months. It had been six months since Alex, now Platform Lead at NovaCraft, had tamed the beast that was Kubernetes. The frantic days of wrestling with YAML files, debugging container crashes, and deciphering cryptic error messages had given way to a sense of calm. The core platform was stable. Deployments were smooth, automated by a slick ArgoCD pipeline. The ten engineering teams that relied on Alex’s platform were shipping features faster than ever. Life was good.

Or so Alex thought.

The calm was shattered during the weekly platform engineering sync. “We had a cascading failure last night,” Sarah, the lead for the payments team, said, her voice strained. “The checkout service started throwing 503s, which caused the order service to time out, and that, in turn, brought down the shipping service. It took us two hours to figure out what was going on. We had no visibility into the service-to-service communication.”

Alex’s mind raced. The number of microservices at NovaCraft had exploded from a handful to over fifty in the last year. The once-simple architecture now resembled a tangled web of dependencies. A failure in one service could trigger a domino effect, bringing the entire system to its knees. The existing monitoring tools, focused on individual services, were blind to the intricate dance of communication between them.

“We need to do better,” Alex said, a sense of determination in their voice. “We need to understand how our services are communicating. We need to be able to control the flow of traffic, enforce security policies, and get deep visibility into the entire system. We need to build a resilient and observable platform, not just a deployment pipeline.”

And so, Alex embarked on a new quest, a journey into the world of service mesh. The goal: to bring order to the chaos of microservices communication and build a platform that was not just scalable, but also resilient, secure, and observable. This is the story of that journey, a journey into “The Mesh.”

What is a Service Mesh and Why Do You Need One?

As NovaCraft’s microservice architecture grew, the complexity of inter-service communication became a significant challenge. This is a common problem in distributed systems. When you have a handful of services, managing their interactions is relatively straightforward. But as the number of services grows, the number of potential connections grows exponentially. This is where a service mesh comes in.

A service mesh is a dedicated infrastructure layer that you add to your application. It allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your own code. The term “service mesh” is often used to describe the network of microservices that make up such applications and the interactions between them.

Think of it like this: in a city, you have buildings (your services) and roads (the network). As the city grows, you can’t just let everyone build their own roads. You need a dedicated infrastructure for managing traffic, ensuring safety, and providing visibility. A service mesh is like the city’s road network, complete with traffic lights, road signs, and a central traffic control system.

The Sidecar Proxy Pattern (Envoy)

The magic of a service mesh lies in the sidecar proxy pattern. Instead of having your application code handle the complexities of network communication, you delegate that responsibility to a dedicated proxy that runs alongside your application in the same Kubernetes Pod. This proxy is called a sidecar.

Envoy is a high-performance proxy designed for cloud-native applications. It was originally built at Lyft and is now a graduated project of the Cloud Native Computing Foundation (CNCF). Istio uses an extended version of Envoy as its sidecar proxy.

Here’s how it works: all incoming and outgoing traffic to and from your application container now flows through the Envoy proxy. This means Envoy can:

  • Control traffic: Route traffic to different versions of a service, perform load balancing, and handle retries and timeouts.
  • Secure communication: Automatically encrypt and decrypt traffic using mutual TLS (mTLS), and enforce access control policies.
  • Provide observability: Collect detailed metrics, logs, and traces for all traffic, giving you deep visibility into your service interactions.

The beauty of this pattern is that your application code remains blissfully unaware of the service mesh. Your developers can focus on writing business logic, while the service mesh handles the complexities of the network.

Istio Architecture: istiod, Data Plane, Control Plane

Now that we understand the concept of a service mesh and the sidecar proxy pattern, let’s dive into the architecture of Istio. Istio’s architecture is logically split into two planes:

  • Data Plane: The data plane is composed of the set of Envoy proxies that are deployed as sidecars to your services. These proxies mediate and control all network communication between microservices. They also collect and report telemetry on all mesh traffic.
  • Control Plane: The control plane is the brain of the service mesh. It is responsible for configuring the proxies to route traffic, enforce policies, and collect telemetry. In Istio, the control plane is managed by a single binary called istiod.

The Control Plane: istiod

Prior to Istio 1.5, the control plane was composed of multiple components (Pilot, Citadel, Galley, and Mixer). To simplify the architecture, these were consolidated into a single binary, istiod. istiod is responsible for:

  • Service Discovery: istiod gets information about services and endpoints from the underlying platform (in our case, Kubernetes) and builds an abstract model of the services in the mesh. This model is then used to configure the Envoy proxies.
  • Configuration: istiod translates high-level routing rules, retry policies, and security policies that you create into Envoy-specific configurations and distributes them to the sidecar proxies.
  • Certificate Management: istiod acts as a Certificate Authority (CA) and generates certificates to allow for secure mTLS communication between services in the data plane.

The Data Plane: Envoy Proxies

The data plane, as we’ve discussed, is composed of the Envoy proxies running as sidecars. These proxies are the workhorses of the service mesh. They are the ones that actually handle the traffic, enforce the policies, and collect the telemetry. The Envoy proxies are configured by istiod, but they operate independently. This means that if the control plane (istiod) goes down, the data plane will continue to function with the last known configuration. This makes the service mesh highly resilient.

Here is a simplified text-based diagram of the architecture:

+--------------------------------------------------------------------+
|                                                                    |
|  +-----------------+       +------------------+       +------------------+   |
|  |   Service A     |       |    Service B     |       |    Service C     |   |
|  | +-------------+ |       | +--------------+ |       | +--------------+ |   |
|  | | Application | | ----> | | Application  | | ----> | | Application  | |   |
|  | +-------------+ |       | +--------------+ |       | +--------------+ |   |
|  | | Envoy Proxy | | <---> | | Envoy Proxy  | | <---> | | Envoy Proxy  | |   |
|  | +-------------+ |       | +--------------+ |       | +--------------+ |   |
|  +-----------------+       +------------------+       +------------------+   |
|                                 Data Plane                               |
+--------------------------------------------------------------------+
                                      ^
                                      |
                                      v
+--------------------------------------------------------------------+
|                              Control Plane                           |
|                          +-------------------+                         |
|                          |       istiod      |                         |
|                          +-------------------+                         |
|                                                                    |
+--------------------------------------------------------------------+

Traffic Management: Routing, Retries, Circuit Breaking, Fault Injection

One of the most powerful features of a service mesh is the ability to control the flow of traffic between services. Istio provides a rich set of traffic management features that allow you to do things like:

  • Traffic Routing: Dynamically control how traffic is routed to different versions of a service. This is incredibly useful for canary releases, A/B testing, and blue-green deployments.
  • Retries: Automatically retry failed requests. This can help to improve the resilience of your application by making it more tolerant of transient failures.
  • Circuit Breaking: Automatically detect and isolate failing instances of a service. This prevents a single failing instance from bringing down the entire service.
  • Fault Injection: Intentionally inject faults into your application to test its resilience. This is a key practice of chaos engineering.

Let’s explore each of these in more detail.

Traffic Routing

Istio’s traffic routing capabilities are configured using two custom resources:

  • VirtualService: A VirtualService defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for traffic of a specific protocol. If the traffic is matched, then it is sent to a named destination service (or subset/version of it) defined in a DestinationRule.
  • DestinationRule: A DestinationRule defines policies that apply to traffic intended for a service after routing has occurred. These rules specify configuration for load balancing, connection pool size from the sidecar, and outlier detection settings to detect and evict unhealthy hosts from the load balancing pool.

For example, you could use a VirtualService to send 90% of the traffic to version v1 of a service and 10% to version v2. This is a common pattern for canary releases, where you gradually roll out a new version of a service to a small subset of users before rolling it out to everyone.

Retries and Timeouts

In a distributed system, transient failures are inevitable. A service might be temporarily unavailable due to a network glitch or a brief spike in load. Istio can automatically retry failed requests, which can help to mask these transient failures and improve the overall resilience of your application. You can configure the number of retries and the timeout for each retry.

Circuit Breaking

Circuit breaking is a pattern for building resilient microservices. The basic idea is that when a service is failing, you “open the circuit” to that service to prevent cascading failures. This means that instead of continuing to send requests to the failing service, you immediately fail the requests. This gives the failing service time to recover and prevents a single failing service from bringing down the entire system.

Istio’s circuit breaking is configured in the DestinationRule. You can configure thresholds for things like the number of consecutive errors, and when the threshold is reached, Istio will open the circuit to that service.

Fault Injection

Fault injection is a testing technique that involves intentionally injecting errors into your application to see how it behaves. This is a key practice of chaos engineering. With Istio, you can inject two types of faults:

  • Delays: Delays are timing failures. They mimic increased network latency or an overloaded upstream service.
  • Aborts: Aborts are crash failures. They mimic failures in upstream services. Aborts usually manifest in the form of HTTP error codes or TCP connection failures.

By injecting faults into your application, you can identify and fix weaknesses in your system before they cause problems in production.

Mutual TLS (mTLS) — Zero-Trust Networking

In a traditional network security model, you have a perimeter firewall that protects your network from the outside world. Once inside the perimeter, however, traffic is often allowed to flow freely. This is known as the “castle-and-moat” model of security. The problem with this model is that if an attacker breaches the perimeter, they have free rein to move around your network.

A zero-trust network is a security model that assumes that the network is always hostile. In a zero-trust network, no traffic is trusted by default, regardless of where it originates. Every request is authenticated and authorized before it is allowed to proceed.

Mutual TLS (mTLS) is a key technology for building zero-trust networks. In traditional TLS, the client verifies the identity of the server. In mTLS, both the client and the server verify each other’s identity. This provides a strong guarantee that both parties are who they say they are.

Istio can automatically encrypt and decrypt all traffic between services in the mesh using mTLS. This is done transparently by the Envoy sidecar proxies. Your application code doesn’t have to do anything to enable mTLS. This is a huge win for security, as it allows you to build a zero-trust network without having to modify your application code.

Here’s how it works:

  1. Certificate Issuance: The istiod control plane acts as a Certificate Authority (CA) and issues certificates to each service in the mesh.
  2. Certificate Distribution: The certificates are distributed to the Envoy sidecar proxies.
  3. mTLS Handshake: When two services communicate, their sidecar proxies perform an mTLS handshake. They exchange certificates and verify each other’s identity.
  4. Secure Communication: Once the mTLS handshake is complete, the sidecar proxies establish a secure, encrypted connection. All traffic between the services is then sent over this encrypted connection.

By enabling mTLS, you can ensure that all communication between your services is secure and that only authorized services can communicate with each other. This is a critical step in building a secure and resilient microservices architecture.

Observability: Distributed Tracing, Metrics, and Service Graphs

Observability is the ability to understand the internal state of a system by examining its external outputs. In a microservices architecture, observability is critical for understanding how your services are interacting and for debugging problems when they occur. Istio provides three key observability features:

  • Distributed Tracing: Distributed tracing allows you to trace the path of a request as it flows through your system. This is incredibly useful for debugging latency issues and for understanding the dependencies between your services.
  • Metrics: Istio generates detailed metrics for all traffic in the mesh. These metrics can be used to monitor the health of your services and to create dashboards and alerts.
  • Service Graphs: Istio can generate a service graph that shows the dependencies between your services. This is a great way to visualize the architecture of your system and to understand how your services are interacting.

Distributed Tracing

Distributed tracing is a technique for monitoring and troubleshooting microservices-based distributed systems. A distributed trace is a collection of events that are causally related. Each event represents a unit of work in the system, such as an HTTP request or a database query. By collecting and analyzing these events, you can get a detailed picture of how a request is being processed by your system.

Istio can automatically generate trace spans for all traffic in the mesh. All you have to do is propagate the trace context from one service to the next. This is typically done by forwarding a set of HTTP headers. Istio supports a number of tracing backends, including Jaeger, Zipkin, and Datadog.

Metrics

Istio generates a wealth of metrics for all traffic in the mesh. These metrics are exposed in the Prometheus format and can be scraped by a Prometheus server. Some of the key metrics that Istio generates include:

  • Request Volume: The number of requests per second.
  • Request Duration: The latency of requests.
  • Request Size: The size of requests and responses.
  • Error Rate: The percentage of requests that result in an error.

These metrics can be used to create dashboards and alerts that allow you to monitor the health of your services and to quickly identify and diagnose problems.

Service Graphs

Istio can generate a service graph that shows the dependencies between your services. The service graph is generated from the telemetry data that is collected by the Envoy proxies. The service graph can be visualized using a tool like Kiali. The service graph is a great way to visualize the architecture of your system and to understand how your services are interacting. It can also be used to identify and diagnose problems, such as services that are receiving a high number of errors.

Hands-on: Install Istio on minikube, deploy a sample app with sidecars, configure traffic splitting, enable mTLS, view Kiali dashboard

Now that we have a good understanding of what a service mesh is and how Istio works, it’s time to get our hands dirty. In this section, we will walk through the process of installing Istio on a local minikube cluster, deploying a sample application, and exploring some of Istio’s key features.

Prerequisites

Before we begin, make sure you have the following tools installed on your macOS machine:

  • minikube: A tool that lets you run Kubernetes locally.
  • kubectl: The Kubernetes command-line tool.

Installing Istio

First, let’s install Istio. We will use the istioctl command-line tool to install Istio on our minikube cluster.

  1. Download Istio:

    Terminal window
    curl -L https://istio.io/downloadIstio | sh -

  2. Add istioctl to your path:

    Terminal window
    cd istio-*
    export PATH=$PWD/bin:$PATH

  3. Install Istio on minikube:

    We will use the demo profile to install Istio. This profile is designed for testing and includes the core Istio components, as well as some additional tools like Kiali and Jaeger.

    Terminal window
    istioctl install --set profile=demo -y

  4. Verify the installation:

    Check the status of the Istio pods to make sure they are all running:

    Terminal window
    kubectl get pods -n istio-system

Deploying the Bookinfo Sample Application

Now that we have Istio installed, let’s deploy a sample application. We will use the bookinfo application that is provided with Istio. The bookinfo application is a simple application that is composed of four microservices:

  • productpage: The productpage microservice is the frontend of the application. It calls the details and reviews microservices to get the book details and reviews.
  • details: The details microservice contains the book information.
  • reviews: The reviews microservice contains the book reviews. There are three versions of the reviews microservice:
    • v1: The v1 version of the reviews microservice does not call the ratings microservice.
    • v2: The v2 version of the reviews microservice calls the ratings microservice and displays the ratings as 1 to 5 black stars.
    • v3: The v3 version of the reviews microservice calls the ratings microservice and displays the ratings as 1 to 5 red stars.
  • ratings: The ratings microservice contains the book ratings.

  1. Label the default namespace for Istio injection:

    Istio needs to know which namespaces to monitor. We can tell Istio to automatically inject the Envoy sidecar proxy into all pods that are deployed to the default namespace by labeling the namespace:

    Terminal window
    kubectl label namespace default istio-injection=enabled

  2. Deploy the bookinfo application:

    Terminal window
    kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

  3. Verify the deployment:

    Check the status of the bookinfo pods to make sure they are all running:

    Terminal window
    kubectl get pods

    You should see that each pod has two containers: the application container and the Envoy sidecar proxy.

  4. Deploy the bookinfo gateway:

    To allow traffic from outside the cluster to reach the bookinfo application, we need to create an Istio Gateway and a VirtualService.

    Terminal window
    kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

  5. Determine the ingress IP and port:

    Terminal window
    export INGRESS_HOST=$(minikube ip)
    export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
    export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

  6. Access the bookinfo application:

    You can now access the bookinfo application in your browser at http://<GATEWAY_URL>/productpage.

    If you refresh the page multiple times, you will see that the reviews change. This is because the traffic is being routed to all three versions of the reviews service in a round-robin fashion.

Configuring Traffic Splitting

Now let’s configure Istio to send 90% of the traffic to v1 of the reviews service and 10% to v2. This is a common pattern for canary releases.

  1. Create a VirtualService:

    Create a file called reviews-virtualservice.yaml with the following content:

    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: reviews
    spec:
      hosts:
      - reviews
      http:
      - route:
        - destination:
            host: reviews
            subset: v1
          weight: 90
        - destination:
            host: reviews
            subset: v2
          weight: 10

  2. Create DestinationRules:

    Create a file called reviews-destinationrule.yaml with the following content:

    apiVersion: networking.istio.io/v1alpha3
    kind: DestinationRule
    metadata:
      name: reviews
    spec:
      host: reviews
      subsets:
      - name: v1
        labels:
          version: v1
      - name: v2
        labels:
          version: v2
      - name: v3
        labels:
          version: v3

  3. Apply the VirtualService and DestinationRule:

    Terminal window
    kubectl apply -f reviews-virtualservice.yaml
    kubectl apply -f reviews-destinationrule.yaml

  4. Verify the traffic splitting:

    If you refresh the productpage in your browser multiple times, you should now see that approximately 90% of the time the reviews are displayed without any stars, and 10% of the time they are displayed with black stars.

Enabling mTLS

Now let’s enable mTLS for the entire mesh. This will ensure that all communication between services is encrypted.

  1. Create a PeerAuthentication policy:

    Create a file called mtls.yaml with the following content:

    apiVersion: "security.istio.io/v1beta1"
    kind: "PeerAuthentication"
    metadata:
      name: "default"
      namespace: "istio-system"
    spec:
      mtls:
        mode: STRICT

  2. Apply the PeerAuthentication policy:

    Terminal window
    kubectl apply -f mtls.yaml

That’s it! You have now enabled mTLS for the entire mesh. All communication between services is now encrypted.

Viewing the Kiali Dashboard

Kiali is a dashboard for Istio that allows you to visualize your service mesh and to view the traffic flowing through it.

  1. Install Kiali:

    The demo profile that we used to install Istio includes Kiali. You can install it with the following command:

    Terminal window
    kubectl apply -f samples/addons/kiali.yaml

  2. Open the Kiali dashboard:

    Terminal window
    istioctl dashboard kiali

    This will open the Kiali dashboard in your browser. In the Kiali dashboard, you can view the service graph, see the traffic flowing through the mesh, and inspect the configuration of your services.

Debugging/Troubleshooting Tips

Even with a powerful tool like Istio, things can go wrong. Here are some common issues you might encounter and how to solve them:

  • Sidecar Injection Not Working: If your pods are starting without the Envoy sidecar, double-check that you have labeled the namespace correctly (kubectl label namespace <your-namespace> istio-injection=enabled). You can also check the logs of the istiod pod for any errors related to sidecar injection.

  • 503 Errors: If you are seeing 503 errors, it could be due to a number of reasons. One common cause is that the traffic is being routed to a service that does not exist or is unhealthy. Use istioctl proxy-config to inspect the configuration of the Envoy proxies and to see how traffic is being routed. You can also use the Kiali dashboard to visualize the traffic flow and to identify any services that are returning errors.

  • mTLS Not Working: If you are having trouble with mTLS, make sure that you have created a PeerAuthentication policy and that it is configured correctly. You can also use the istioctl authn tls-check command to check the mTLS status of your services.

  • Performance Issues: If you are experiencing performance issues, it could be due to the overhead of the Envoy sidecar proxies. You can use the istioctl proxy-status command to see the resource usage of the Envoy proxies. You can also use the Kiali dashboard to identify any services that are experiencing high latency.

Key Takeaways

  • A service mesh is a dedicated infrastructure layer that you add to your application to transparently add capabilities like observability, traffic management, and security.
  • Istio uses the sidecar proxy pattern, where a proxy (Envoy) runs alongside your application in the same Pod.
  • Istio’s architecture is split into a data plane (the Envoy proxies) and a control plane (istiod).
  • Istio provides powerful traffic management features, including routing, retries, circuit breaking, and fault injection.
  • Mutual TLS (mTLS) can be used to build a zero-trust network by encrypting all traffic between services.
  • Istio provides deep observability into your service mesh through distributed tracing, metrics, and service graphs.

Story Closing + Teaser

Alex leaned back, a sense of accomplishment washing over them. The Kiali dashboard glowed on the screen, a beautiful, intricate web of services, all communicating securely and efficiently. The chaos was gone, replaced by order. The platform was no longer a black box; it was a living, breathing system that could be understood, controlled, and secured.

The journey into “The Mesh” had been challenging, but the rewards were immense. NovaCraft’s platform was now more resilient, secure, and observable than ever before. But Alex knew that the journey was far from over. The world of cloud-native was vast and ever-changing. New challenges and new technologies were always on the horizon.

As Alex looked at the service graph, a new thought began to form. “We have control over the network,” Alex mused, “but what about the policies that govern our services? How can we enforce fine-grained authorization policies? How can we ensure that only the right services can talk to each other?” The next chapter in The Container Odyssey was beginning to write itself, a chapter on policy enforcement with Open Policy Agent (OPA).

This is Part 2 of Season 2 of The Container Odyssey.