Part 3: "The Power Outage" — RDB, AOF, and the Persistence Dilemma

Story Opening

The “Trending Now” leaderboard, powered by Redis Sorted Sets, was a massive success for GlobalMart. Maya, the Staff Engineer who architected it, was enjoying a rare moment of peace when her pager exploded. A catastrophic power failure had struck the primary data center.

The PostgreSQL cluster automatically failed over to a synchronous replica in another availability zone, preserving the core transactional data (orders, inventory, user accounts). However, the Redis cluster, which was running with default configurations, simply vanished from the network. When the power was restored and the Redis instances rebooted, they came back up empty.

The cached product catalog was gone, which meant the database was immediately hammered by the Cache Stampede Maya had feared in Part 1. Worse, the real-time leaderboard was wiped clean. But the most critical loss was the shopping cart data. GlobalMart stored active, un-purchased shopping carts exclusively in Redis to avoid writing to PostgreSQL on every “Add to Cart” click. Thousands of users suddenly found their carts empty.

Maya realized a fundamental truth about in-memory databases: RAM is volatile. When the power dies, the electrons scatter, and the data is gone. Redis is incredibly fast, but without persistence, it is incredibly fragile. She needed to configure Redis to survive a reboot without losing its mind. She needed to understand the persistence dilemma.

Conceptual Deep-Dive

Redis offers two primary mechanisms for persisting data to disk: RDB (Redis Database) snapshots and AOF (Append Only File) logging. Understanding how they work, their trade-offs, and how they interact with the underlying operating system is crucial for any senior engineer managing Redis in production.

1. RDB (Redis Database) Snapshots

RDB is the default persistence mechanism. It creates point-in-time snapshots of the entire dataset at specified intervals (e.g., “save every 60 seconds if at least 10,000 keys changed”).

The resulting .rdb file is a highly compact, binary representation of the data. Because it’s just a raw dump of memory, restarting Redis from an RDB file is incredibly fast.

However, the major drawback is data loss. If Redis crashes, you lose all data written since the last snapshot. If your policy is to save every 5 minutes, you could lose up to 5 minutes of critical data (like shopping carts).

The Magic of fork() and Copy-on-Write

The most fascinating aspect of RDB is how Redis takes a snapshot of a massive, actively changing dataset without blocking the single thread that serves client requests. If Redis paused to write 50GB of RAM to a slow disk, the application would freeze for minutes.

Redis solves this using the Unix fork() system call.

When it’s time to take a snapshot, the main Redis process calls fork(). The operating system instantly creates a child process that is an exact duplicate of the parent process. Crucially, the OS does not actually copy the 50GB of RAM. That would be slow and double the memory usage.

Instead, modern operating systems use a technique called Copy-on-Write (CoW). Both the parent (serving clients) and the child (writing to disk) point to the exact same physical memory pages. They share the RAM.

The child process begins iterating over the shared memory, serializing the data, and writing it to the .rdb file on disk. Because the child is only reading, it doesn’t modify the shared memory.

But what happens if a client sends a SET command to the parent process while the child is still writing? If the parent modified the shared memory page, the child’s snapshot would become corrupted.

This is where Copy-on-Write kicks in. The OS marks the shared memory pages as read-only. When the parent process attempts to modify a page (e.g., Memory Page B), the OS intercepts the write (a page fault). The OS transparently copies that specific page, allows the parent to modify the copy, and leaves the original page intact for the child process.

This brilliant architecture allows Redis to take consistent, point-in-time snapshots of massive datasets with near-zero performance impact on the main thread.

2. AOF (Append Only File)

If losing 5 minutes of data is unacceptable (as it was for GlobalMart’s shopping carts), you need AOF.

AOF works completely differently. Instead of dumping the memory state, it logs every single write operation received by the server (e.g., SET user:1 "Maya", INCR views). These commands are appended to an .aof file on disk.

When Redis restarts, it simply replays the entire log of commands from the .aof file to reconstruct the dataset in memory.

AOF provides much stronger durability guarantees. You can configure Redis to fsync (flush the OS buffer to physical disk) the AOF file:

• Always: Every single write is synced to disk before returning success to the client. Safest, but drastically reduces write performance (you are now bottlenecked by disk I/O).
• Everysec: (The default and recommended setting) The write is buffered in the OS, and a background thread syncs it to disk once per second. You lose at most 1 second of data if a crash occurs, offering an excellent balance of speed and safety.
• No: Let the OS decide when to flush. Fast, but risky.

The AOF Rewrite Problem

The obvious problem with AOF is that the file grows infinitely. If you increment a counter 1 million times, the AOF file will contain 1 million INCR commands. Replaying that on startup would be agonizingly slow.

To solve this, Redis performs an AOF Rewrite in the background (again, using fork() and Copy-on-Write). It creates a new, minimal AOF file containing only the shortest sequence of commands needed to recreate the current state in memory. Those 1 million INCR commands are rewritten as a single SET counter 1000000 command.

Technical Explanation

Maya evaluated the trade-offs for GlobalMart:

She decided on a Mixed Persistence Strategy (available in Redis 4.0+).

In this mode, Redis uses both. When an AOF rewrite occurs, it doesn’t write the minimal commands as text. Instead, it writes an RDB binary snapshot to the beginning of the AOF file, and then appends any new commands as standard AOF text logs.

This provides the best of both worlds: the fast restart speed of RDB (loading the bulk of the data quickly) combined with the 1-second durability guarantee of AOF.

Step-by-Step Hands-On

Maya updated the redis.conf file to implement the mixed persistence strategy.

Step 1: Configuring RDB and AOF in redis.conf

# --- RDB Configuration ---
# Save the DB to disk.
# Format: save <seconds> <changes>
# Save after 900 sec (15 min) if at least 1 key changed
save 900 1
# Save after 300 sec (5 min) if at least 10 keys changed
save 300 10
# Save after 60 sec if at least 10000 keys changed
save 60 10000

# The filename where to dump the DB
dbfilename dump.rdb

# --- AOF Configuration ---
# Enable AOF
appendonly yes

# The name of the append only file
appendfilename "appendonly.aof"

# The fsync policy: everysec is the sweet spot
appendfsync everysec

# --- Mixed Persistence (Redis 4.0+) ---
# When rewriting the AOF file, use RDB preamble for faster restarts
aof-use-rdb-preamble yes

# Auto-rewrite AOF when it grows by 100% compared to the last rewrite,
# and the minimum size is 64MB.
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Step 2: Spring Boot Considerations

From the application’s perspective (Spring Boot), persistence is entirely transparent. The RedisTemplate and @Cacheable annotations don’t care whether Redis is writing to disk or not.

However, Maya had to ensure that the shopping cart data was explicitly serialized in a way that survived restarts cleanly. She configured a custom RedisTemplate specifically for the carts, using JSON serialization for the values.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

@Configuration
public class CartRedisConfig {

    @Bean(name = "cartRedisTemplate")
    public RedisTemplate<String, ShoppingCart> cartRedisTemplate(RedisConnectionFactory connectionFactory) {
        RedisTemplate<String, ShoppingCart> template = new RedisTemplate<>();
        template.setConnectionFactory(connectionFactory);

        // Keys are Strings (e.g., "cart:user123")
        template.setKeySerializer(new StringRedisSerializer());

        // Values are complex Objects serialized to JSON
        template.setValueSerializer(new GenericJackson2JsonRedisSerializer());

        return template;
    }
}

Debugging/Troubleshooting Tips

• The CoW Memory Spike: While Copy-on-Write saves memory, it is not free. If your Redis instance is highly active (many writes) during an RDB save or AOF rewrite, the OS will have to copy many memory pages. If Redis is already using 80% of the physical RAM, the CoW process could consume the remaining 20%, triggering the Linux OOM (Out Of Memory) killer, which will terminate the Redis process. Solution: Never run Redis at 100% memory capacity. Always leave 30-40% of RAM free to accommodate CoW spikes during persistence operations.
• Disk I/O Blocking: Even with background threads, heavy disk I/O from RDB saves can impact the overall system performance. Solution: In a clustered environment (covered in Part 5), disable persistence on the Master nodes entirely to ensure maximum write performance, and enable AOF/RDB only on the Replica nodes, which handle the disk I/O away from the critical path.
• The BGSAVE Command: You can manually trigger an RDB snapshot using the BGSAVE command. Never use the synchronous SAVE command in production, as it blocks the main thread until the entire file is written to disk.

Key Takeaways

• Volatility: In-memory databases lose data on restart unless configured otherwise.
• RDB (Snapshots): Compact binary dumps, fast restarts, but higher potential for data loss. Uses fork() and Copy-on-Write to avoid blocking.
• AOF (Append Only File): Logs every write command, slower restarts, but provides strong durability (up to 1-second guarantees with everysec).
• Mixed Persistence: The modern standard. Combines RDB snapshots for fast loading with AOF logs for recent durability.
• Memory Headroom: Always leave significant RAM free to handle Copy-on-Write spikes during background saves.

Story Closing + Teaser

With the mixed persistence strategy deployed, Maya simulated another “power outage” by brutally killing the Redis processes. When she restarted them, the AOF logs replayed instantly, loading the RDB preamble and the subsequent commands. The shopping carts were intact. The leaderboard was preserved. The crisis was truly over.

However, a new, more insidious problem was creeping up on the horizon. As GlobalMart’s catalog expanded and users added more items to their carts, the Redis memory usage began to climb steadily. It crossed 60%, then 70%, then 85%. Maya knew that if it hit 100%, the OS would start swapping pages to disk, completely destroying Redis’s performance, or worse, the OOM killer would terminate the process.

She couldn’t just keep buying larger and larger RAM instances. She needed to teach Redis how to forget. She needed to configure eviction policies and understand the fascinating mathematics behind the Approximated LRU algorithm. That memory management challenge awaited in the next chapter.

References

[1] Redis Persistence Documentation. https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/ [2] Understanding Copy-on-Write in Linux. https://en.wikipedia.org/wiki/Copy-on-write