Part 5: "Going Global" — Sentinel, Clustering, and High Availability

Story Opening

Maya, Staff Engineer at GlobalMart, had successfully transformed the monolithic application. Redis was caching products, managing shopping carts, and powering a real-time leaderboard. The architecture was fast, durable, and memory-efficient.

But it was still fragile.

The entire e-commerce platform relied on a single Redis instance. While Maya had configured a passive replica (a backup node) in another availability zone, the failover process was entirely manual. If the primary Redis server died at 3 AM, the application would start throwing Connection refused errors. The on-call engineer would have to wake up, SSH into the replica, run the REPLICAOF NO ONE command to promote it to primary, and then update the application’s configuration to point to the new IP address. That meant at least 15 minutes of downtime.

Furthermore, as GlobalMart expanded internationally, a single Redis server, even a very large one, was reaching its limits. The network interface card (NIC) on the server was saturating under the sheer volume of read requests. They couldn’t just scale vertically (buy a bigger server) anymore; they needed to scale horizontally.

Maya needed a system that could automatically detect failures, promote replicas, and route traffic seamlessly. More importantly, she needed a way to distribute the dataset across multiple machines to handle the massive read and write load. She needed to understand Redis Sentinel and Redis Cluster.

Conceptual Deep-Dive

When moving from a single instance to a distributed system, Redis offers two distinct architectural patterns: Sentinel (for High Availability) and Cluster (for High Availability and Horizontal Scalability).

1. Master-Replica Replication

The foundation of both patterns is asynchronous replication.

In a Master-Replica setup, the Master node handles all writes. The Replica nodes connect to the Master and receive a continuous stream of the write commands, applying them locally to maintain an exact copy of the dataset.

Because replication is asynchronous, a write command returns OK to the client before the Master has confirmed the Replica has received it. This ensures the Master remains blazingly fast, but it introduces a window for data loss. If the Master crashes immediately after acknowledging a write but before replicating it, that data is lost. (Redis does offer the WAIT command for synchronous replication, but it severely impacts performance).

Replicas are incredibly useful for scaling reads. You can configure your application to send all GET requests to the Replicas, leaving the Master dedicated entirely to SET and INCR operations.

2. Redis Sentinel: Automatic Failover

Replication provides redundancy, but it doesn’t provide automatic failover. If the Master dies, the application must be reconfigured to point to a Replica.

Redis Sentinel is a separate, distributed system designed specifically to monitor Redis instances, detect failures, and perform automatic failovers.

Sentinels are lightweight processes that run alongside your Redis servers. They constantly ping the Master and Replicas. If a Sentinel cannot reach the Master, it flags it as Subjectively Down (SDOWN).

However, a single Sentinel cannot trigger a failover. What if that specific Sentinel is experiencing a network partition, but the Master is actually fine?

To prevent split-brain scenarios, Sentinels use a quorum system. They communicate with each other. Only if a configured majority (the quorum) of Sentinels agree that the Master is unreachable does it become Objectively Down (ODOWN).

Once ODOWN is reached, the Sentinels hold an election to choose a leader among themselves. That leader then selects the best Replica (based on replication offset, priority, and run ID), sends the REPLICAOF NO ONE command to promote it to Master, and reconfigures the other Replicas to follow the new Master.

Crucially, the client application (Spring Boot) no longer connects directly to the Redis Master. Instead, it connects to the Sentinels, asks “Who is the current Master?”, and then routes its traffic accordingly. When a failover happens, the Sentinels notify the client, and the client automatically reconnects to the new Master.

3. Redis Cluster: Horizontal Scalability

Sentinel provides High Availability, but all writes still go to a single Master, and the entire dataset must fit in the RAM of every single node. If GlobalMart needed to cache 500GB of data, every node in a Sentinel setup would need 500GB of RAM.

Redis Cluster solves this by sharding (partitioning) the data across multiple Masters.

Instead of using consistent hashing (like Cassandra or DynamoDB), Redis Cluster uses a concept called Hash Slots.

There are exactly 16,384 hash slots in a Redis Cluster. When you set a key (e.g., SET user:123 "Maya"), Redis calculates the CRC16 hash of the key and takes the modulo 16384:

HASH_SLOT = CRC16("user:123") % 16384

Every Master node in the cluster is responsible for a subset of these 16,384 slots.

For example, in a 3-Master cluster:

• Master A handles slots 0 to 5500.
• Master B handles slots 5501 to 11000.
• Master C handles slots 11001 to 16383.

If the hash of user:123 is 6000, that key must be stored on Master B.

If a client connects to Master A and tries to read user:123, Master A will not proxy the request. Instead, it returns a MOVED 6000 <IP_OF_MASTER_B> error. The client (e.g., Spring Data Redis) intercepts this error, caches the routing table, and transparently redirects the request to Master B.

Redis Cluster does not use Sentinels. The Master nodes themselves form a decentralized quorum. They constantly gossip with each other via a cluster bus port. If Master B fails, Masters A and C will detect it, hold an election, and promote Replica B to become the new Master B, taking over slots 5501-11000.

Technical Explanation

Maya had to choose between Sentinel and Cluster for GlobalMart.

• Sentinel: Easier to set up, supports multi-key operations (like MGET or transactions) easily because all data is on one node. Perfect if the dataset fits in one machine’s RAM and write throughput is manageable.
• Cluster: Required for massive datasets (>100GB) or massive write throughput. However, multi-key operations are restricted. You cannot run an MGET across keys that reside in different hash slots (unless you use Hash Tags, e.g., user:{123}:profile and user:{123}:cart, which forces them into the same slot).

Because GlobalMart’s product catalog and shopping carts were growing exponentially, Maya chose Redis Cluster.

Step-by-Step Hands-On

Let’s look at how Maya configured the Spring Boot application to connect to the new Redis Cluster.

Step 1: Cluster Configuration in application.yml

Spring Boot makes connecting to a Redis Cluster incredibly simple. Instead of providing a single host and port, Maya provided a list of the initial cluster nodes.

spring:
  data:
    redis:
      cluster:
        # Provide at least a few nodes. The client will discover the rest.
        nodes:
          - 10.0.1.10:6379
          - 10.0.1.11:6379
          - 10.0.1.12:6379
        # Maximum number of redirects to follow (e.g., when a MOVED error occurs)
        max-redirects: 3
      # Connection timeout
      timeout: 2000ms

Step 2: Advanced Cluster Topology with Lettuce

Spring Data Redis uses the Lettuce client by default. Lettuce is a fully non-blocking, reactive Redis client that is highly optimized for Cluster topologies.

Maya needed to ensure that Lettuce would automatically refresh its internal routing table (the map of which hash slots belong to which IPs). If Master B failed and Replica B took over, the IPs would change. If Lettuce didn’t refresh its topology, it would keep sending requests to the dead Master B.

She configured the LettuceConnectionFactory to enable periodic topology refreshes.

import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;

import java.time.Duration;
import java.util.List;

@Configuration
public class RedisClusterConfig {

    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {

        // 1. Define the cluster nodes (usually injected from properties)
        RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration(
            List.of("10.0.1.10:6379", "10.0.1.11:6379", "10.0.1.12:6379")
        );

        // 2. Configure Topology Refresh Options
        ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enablePeriodicRefresh(Duration.ofSeconds(30)) // Refresh every 30 seconds
                .enableAllAdaptiveRefreshTriggers() // Refresh immediately if a MOVED/ASK error occurs
                .build();

        // 3. Apply options to the Lettuce Client
        ClusterClientOptions clientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(topologyRefreshOptions)
                .build();

        LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
                .clientOptions(clientOptions)
                .build();

        // 4. Build the factory
        return new LettuceConnectionFactory(clusterConfig, clientConfig);
    }
}

With this configuration, the Spring Boot application was now fully Cluster-aware. The @Cacheable annotations and RedisTemplate operations (from Parts 1-4) required zero code changes. The Lettuce client transparently handled the hashing, routing, and failover redirects.

Debugging/Troubleshooting Tips

• The Cross-Slot Error: If you try to execute a transaction (MULTI/EXEC) or a multi-key command (MGET key1 key2) where key1 and key2 hash to different slots, Redis Cluster will throw a CROSSSLOT error. Solution: Use Hash Tags. If you name your keys {user123}:profile and {user123}:cart, Redis will only hash the substring inside the curly braces (user123). Both keys will guaranteeably land in the same hash slot on the same node, allowing multi-key operations.
• The Gossip Overhead: In a very large cluster (e.g., 100+ nodes), the decentralized gossip protocol can consume significant network bandwidth and CPU as every node constantly pings every other node. Solution: Redis Cluster is generally recommended for clusters up to ~1000 nodes. Beyond that, consider multiple smaller clusters.
• Scaling Up/Down: Adding a new Master node to a live cluster requires resharding. You must manually (or via scripts) tell the existing Masters to migrate a portion of their hash slots (and the underlying data) to the new node. During migration, keys might be in transit, resulting in ASK redirects instead of MOVED redirects. Lettuce handles this transparently, but it adds latency during the migration window.

Key Takeaways

• High Availability: Requires automatic failover. Sentinel provides this for single-master setups; Cluster provides it natively for sharded setups.
• Hash Slots: Redis Cluster shards data deterministically across 16,384 slots using CRC16 hashing, avoiding the complexities of consistent hashing rings.
• Client-Side Routing: In Cluster mode, the client (Lettuce) is responsible for routing the request to the correct node based on the hash slot.
• Topology Refresh: Always configure your client to periodically refresh the cluster topology to handle failovers and resharding events gracefully.

Story Closing + Teaser

The migration to Redis Cluster was a massive undertaking, but it paid off immediately. GlobalMart’s caching layer was now distributed across six servers (3 Masters, 3 Replicas) in three different availability zones. Read throughput quadrupled, and the system could survive the loss of any single node without a hiccup.

Maya had solved the caching, the memory limits, and the scalability. But the product team had a new requirement, one that didn’t fit the mold of a simple key-value cache or a sorted set.

They wanted to build a real-time, event-driven order processing pipeline. When a user clicked “Checkout,” that event needed to be broadcasted reliably to the Inventory service, the Billing service, and the Shipping service. They needed a message broker.

Maya knew about Kafka and RabbitMQ, but she wondered: could the Redis Cluster she just built handle this too? She needed to dive into the world of Pub/Sub, and more importantly, the durable, Kafka-like data structure introduced in Redis 5.0: Redis Streams. That event-driven challenge awaited in the next chapter.

References

[1] Redis Cluster Specification. https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/ [2] Redis Sentinel Documentation. https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/ [3] Lettuce Reference Guide: Redis Cluster. https://lettuce.io/core/release/reference/index.html#redis-cluster