Part 8: "The Intelligent Agent" — Production RAG Architectures
Maya builds an AI shopping assistant. We move from naive retrieval to a production-grade Retrieval-Augmented Generation (RAG) system. We cover chunking strategies, hybrid search (combining vectors with metadata filters), and using Redis as conversational memory for LLMs.
Story Opening
The “Find Similar Products” feature, powered by Redis Vector Search (Part 7), was a massive success for GlobalMart. The semantic understanding of user queries drastically improved conversion rates. But the VP of Product had a more ambitious vision: an AI shopping assistant, a chatbot that could guide users through complex purchasing decisions.
“I want a user to be able to say, ‘I need a tent for a winter camping trip in the Rockies, but I only have $200, and it must be in stock right now.’ And the bot should give them a personalized recommendation, citing our actual inventory.”
Maya, the Staff Engineer, knew that simply sending that prompt to a Large Language Model (LLM) like GPT-4 would fail disastrously. The LLM has no idea what GlobalMart’s current inventory is, what the prices are today, or which specific tents are in stock. If asked, the LLM would confidently hallucinate a product that doesn’t exist, or recommend a tent that GlobalMart stopped selling three years ago.
The LLM needed to be grounded in reality. It needed access to GlobalMart’s proprietary, real-time data. Maya needed to build a Retrieval-Augmented Generation (RAG) architecture.
She realized that her existing Redis cluster was the perfect foundation. She already had the product catalog cached (Part 1), she had the vector embeddings for semantic search (Part 7), and she had the real-time inventory data. She just needed to tie it all together into a cohesive, production-grade AI agent.
Conceptual Deep-Dive
RAG is the industry-standard architecture for grounding LLMs in enterprise data. It works by intercepting the user’s question, searching a database for relevant information, and then passing both the question and the retrieved information to the LLM to generate an answer.
However, moving from a simple “tutorial RAG” to a production-grade system requires solving several complex architectural challenges.
1. The Naive RAG Problem
In a naive RAG setup, you embed the user’s question, perform a vector search (KNN) against your document database, retrieve the top 3 results, stuff them into a prompt, and ask the LLM to answer.
This fails in production for several reasons:
- The “Lost in the Middle” Phenomenon: If you retrieve a massive 10,000-word document and stuff it into the context window, the LLM often ignores the information in the middle, focusing only on the beginning and end.
- Irrelevant Context: If the vector search returns documents that are only tangentially related to the user’s specific constraint (e.g., retrieving a great winter tent that costs 200), the LLM might still recommend it, ignoring the user’s budget.
- Stale Data: Vector embeddings are static. If a product goes out of stock, its vector embedding doesn’t change. A pure vector search will happily retrieve an out-of-stock item as the “most semantically relevant” result.
2. Chunking Strategies
To solve the “Lost in the Middle” problem, you must break large documents (like a 50-page product manual for a complex electronic device) into smaller, semantically meaningful pieces called chunks.
Instead of embedding the entire manual as one vector, you embed each chunk as its own vector.
- Fixed-Size Chunking: Splitting by character count (e.g., every 500 characters). This is easy but terrible, as it often slices sentences or paragraphs in half, destroying the semantic meaning.
- Sentence/Paragraph Chunking: Splitting by natural language boundaries (periods, newlines). This preserves meaning much better.
- Semantic Chunking: Using an NLP model to group sentences that discuss the same topic into a single chunk, regardless of length. This is the gold standard for RAG.
When a user asks a question, the vector search retrieves only the specific chunks (paragraphs) that contain the answer, providing the LLM with highly concentrated, relevant context without overwhelming its context window.
3. Hybrid Search: Vectors + Metadata
To solve the “Stale Data” and “Irrelevant Context” problems (like the $200 budget or the “in stock” requirement), you cannot rely on vector search alone. Vectors understand meaning (“winter tent”), but they are terrible at exact filtering (“price < 200” AND “stock > 0”).
This is where Hybrid Search comes in.
Redis Stack (RediSearch) excels at this. It allows you to combine a high-dimensional vector similarity search (KNN) with traditional, exact metadata filters (numeric ranges, tags, full-text) in a single, atomic query.
You index the product data as a Hash containing the vector, the price (as a NUMERIC field), and the stock status (as a TAG field).
When the user asks for a winter tent under $200 that is in stock, the query looks like this:
@price:[-inf 200] @stock:{true} => [KNN 5 @vector $query_vec]
Redis first filters the dataset down to only products under $200 that are in stock (using fast inverted indexes or numeric tries). Then, it performs the vector search (HNSW) only on that filtered subset to find the most semantically relevant “winter tents.”
This guarantees the LLM receives context that perfectly matches the user’s hard constraints.
4. Conversational Memory
A shopping assistant isn’t a single Q&A session; it’s a conversation.
- User: “I need a winter tent under $200.”
- Bot: “Here is the ‘Arctic Dome’ for $150.”
- User: “Does it come in blue?”
If you send “Does it come in blue?” to the vector database, it will fail. “It” is a pronoun referring to the “Arctic Dome,” but the database doesn’t know that.
The RAG system must maintain Conversational Memory. It needs to remember the history of the chat. Redis is the perfect place to store this ephemeral session state.
Before executing the vector search, the system retrieves the chat history from Redis, sends it to the LLM, and asks the LLM to rewrite the user’s query into a standalone search query.
- History: User asked for tent, Bot suggested Arctic Dome.
- User Query: “Does it come in blue?”
- LLM Rewritten Query: “Does the Arctic Dome winter tent come in blue?”
This rewritten query is then embedded and sent to the Redis vector search, ensuring accurate retrieval.
Technical Explanation
Maya architected the complete RAG pipeline for the GlobalMart AI Assistant:
- User Input: User sends a message via the chat UI.
- Memory Retrieval: The Spring Boot backend fetches the user’s chat history (the last 10 messages) from a Redis List (
LRANGE chat:session:123 0 9). - Query Rewrite: The backend calls the LLM (e.g., GPT-4o-mini) with the history and the new user message, asking it to generate a standalone search query and extract any hard filters (price, category).
- Vector Embedding: The standalone query is embedded into a vector (e.g., 1536 dimensions).
- Hybrid Search: The backend executes a RediSearch query combining the metadata filters (price < 200) with the vector KNN search against the
product_chunks_idx. - Prompt Assembly: The retrieved product chunks (the context) are combined with the user’s original question and a system prompt (“You are a helpful GlobalMart assistant…”).
- Generation: The assembled prompt is sent to the LLM to generate the final response.
- Memory Update: The user’s question and the LLM’s response are appended to the Redis chat history List (
RPUSH chat:session:123 ...), and the TTL is refreshed (EXPIRE chat:session:123 3600).
Step-by-Step Hands-On
Let’s look at how Maya implemented the core Hybrid Search and Memory components using Spring Boot and the modern redis.clients.jedis library.
Step 1: Defining the Hybrid Index
Maya expanded the index definition from Part 7 to include numeric and tag fields for filtering.
import redis.clients.jedis.JedisPooled;import redis.clients.jedis.search.IndexDefinition;import redis.clients.jedis.search.IndexOptions;import redis.clients.jedis.search.Schema;
public class RagIndexService {
private final JedisPooled jedis; private static final String INDEX_NAME = "product_chunks_idx";
public RagIndexService(JedisPooled jedis) { this.jedis = jedis; }
public void createHybridIndex() { // Define the schema with Metadata + Vector Schema schema = new Schema() .addTextField("chunk_text", 1.0) .addNumericField("price") // For range queries (e.g., < 200) .addTagField("in_stock") // For exact match (e.g., "true") .addVectorField("chunk_vector", Schema.VectorField.VectorAlgo.HNSW, Map.of( "TYPE", "FLOAT32", "DIM", 1536, "DISTANCE_METRIC", "COSINE" ));
IndexDefinition def = new IndexDefinition(IndexDefinition.Type.HASH) .setPrefixes(new String[]{"chunk:"});
jedis.ftCreate(INDEX_NAME, IndexOptions.defaultOptions().setDefinition(def), schema); }}Step 2: Executing the Hybrid Search
When the LLM extracted the user’s constraints (e.g., max price $200), Maya constructed a hybrid query.
import redis.clients.jedis.JedisPooled;import redis.clients.jedis.search.Document;import redis.clients.jedis.search.Query;import redis.clients.jedis.search.SearchResult;
import java.util.List;import java.util.stream.Collectors;
public class HybridSearchService {
private final JedisPooled jedis; private final EmbeddingClient embeddingClient;
public HybridSearchService(JedisPooled jedis, EmbeddingClient embeddingClient) { this.jedis = jedis; this.embeddingClient = embeddingClient; }
public List<String> retrieveContext(String rewrittenQuery, double maxPrice) { // 1. Embed the query float[] queryVector = embeddingClient.embedText(rewrittenQuery); byte[] queryVectorBytes = floatArrayToByteArray(queryVector);
// 2. Construct the Hybrid Query // Syntax: (@price:[-inf MAX_PRICE] @in_stock:{true}) => [KNN 5 @chunk_vector $vec AS score] String filterStr = String.format("(@price:[-inf %f] @in_stock:{true})", maxPrice); String knnStr = "=>[KNN 5 @chunk_vector $vec AS score]"; String fullQueryString = filterStr + knnStr;
Query query = new Query(fullQueryString) .addParam("vec", queryVectorBytes) .returnFields("chunk_text", "price") .dialect(2); // Required for vector queries
// 3. Execute search SearchResult result = jedis.ftSearch("product_chunks_idx", query);
// 4. Return the text chunks to be injected into the LLM prompt return result.getDocuments().stream() .map(doc -> doc.getString("chunk_text")) .collect(Collectors.toList()); }}Step 3: Managing Conversational Memory
Maya used a simple Redis List to store the JSON representation of the chat history, ensuring it expired after an hour of inactivity.
import org.springframework.data.redis.core.StringRedisTemplate;import org.springframework.stereotype.Service;
import java.time.Duration;import java.util.List;
@Servicepublic class ChatMemoryService {
private final StringRedisTemplate redisTemplate; private static final Duration SESSION_TIMEOUT = Duration.ofHours(1);
public ChatMemoryService(StringRedisTemplate redisTemplate) { this.redisTemplate = redisTemplate; }
public void appendMessage(String sessionId, String role, String content) { String key = "chat:session:" + sessionId; String jsonMessage = String.format("{\"role\":\"%s\", \"content\":\"%s\"}", role, content);
// Append to the end of the list redisTemplate.opsForList().rightPush(key, jsonMessage);
// Reset the TTL so active sessions don't expire redisTemplate.expire(key, SESSION_TIMEOUT); }
public List<String> getHistory(String sessionId, int maxMessages) { String key = "chat:session:" + sessionId;
// Retrieve the last N messages (e.g., from index -10 to -1) long start = Math.max(0, redisTemplate.opsForList().size(key) - maxMessages); return redisTemplate.opsForList().range(key, start, -1); }}Debugging/Troubleshooting Tips
- The Pre-Filter vs. Post-Filter Dilemma: In hybrid search, if your metadata filter is extremely restrictive (e.g., finding a product that only has 3 items in stock out of 10 million), the HNSW graph traversal might struggle to find enough neighbors that satisfy the filter, resulting in fewer than results. Redis Stack handles this gracefully by dynamically switching between pre-filtering (filtering first, then calculating distances) and post-filtering (traversing the graph and discarding non-matching nodes) based on heuristics, but you must benchmark your specific queries.
- Context Window Overflow: Even with chunking, if you retrieve 10 chunks and the user has a long chat history, you might exceed the LLM’s token limit (e.g., 8K or 128K tokens). Solution: Implement a token counter (like
tiktoken) in your Spring Boot app before assembling the prompt. If you exceed the limit, summarize the older chat history or drop the least relevant chunks. - Data Synchronization: If the price of a product changes in PostgreSQL, you must immediately update the
@pricefield in the Redis Hash. If you don’t, the Hybrid Search will filter based on stale prices, and the RAG system will recommend items the user can’t afford. (Use the Redis Streams event pipeline from Part 6 to trigger these updates).
Key Takeaways
- RAG Architecture: Grounds LLMs in proprietary data by retrieving relevant context before generation, eliminating hallucinations.
- Chunking: Splitting large documents into smaller, semantically meaningful pieces is crucial to avoid the “Lost in the Middle” problem and provide concentrated context.
- Hybrid Search: Combining vector similarity (meaning) with exact metadata filters (price, stock) is the only way to build a production-grade search system.
- Conversational Memory: LLMs are stateless. Redis Lists are the perfect data structure for maintaining ephemeral, fast-access chat history to enable multi-turn conversations.
Story Closing
The AI Shopping Assistant launched to rave reviews. Users could chat naturally, specifying complex constraints and vague desires, and the bot responded instantly with highly accurate, in-stock recommendations. The hallucination rate dropped to zero because every answer was grounded in the precise chunks retrieved by the Redis Hybrid Search.
Maya leaned back in her chair and looked at the architecture diagram on her whiteboard. She had started with a struggling monolith and a melting relational database. Over the course of her journey, she had introduced in-memory caching (Part 1), utilized advanced data structures for real-time leaderboards (Part 2), mastered persistence and memory management (Parts 3 & 4), scaled the system globally with Redis Cluster (Part 5), built an event-driven backbone with Streams (Part 6), and finally, transformed the entire platform into an AI-powered semantic engine (Parts 7 & 8).
Redis was no longer just a cache at GlobalMart. It was the central nervous system of the entire enterprise. The journey from simple strings to high-dimensional vector graphs was complete.
References
[1] Redis Vector Search and Hybrid Queries. https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/vectors/ [2] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Lewis et al., 2020. [3] Lost in the Middle: How Language Models Use Long Contexts. Liu et al., 2023.