Part 7: "Semantic Discovery" — Vector Embeddings and HNSW Graphs

Story Opening

Maya, the Staff Engineer at GlobalMart, sat in a meeting with the VP of Product. The agenda was simple: “Fix Search.”

GlobalMart’s search functionality was powered by traditional SQL LIKE queries and a basic full-text search engine. It was fundamentally a keyword-matching system. If a user searched for “running shoes,” the system looked for products containing the words “running” and “shoes” in their title or description.

The VP pulled up a slide showing the problem. “A user searched for ‘footwear for a rainy marathon’. Our system returned zero results because no product explicitly has the words ‘footwear’, ‘rainy’, and ‘marathon’ in the description. Instead, it showed them umbrellas and running shorts. The products they actually needed—waterproof trail runners—were completely missed.”

Keyword search fails when the user’s vocabulary doesn’t exactly match the catalog’s vocabulary. It lacks semantic understanding. It doesn’t know that “footwear” means “shoes,” or that a “marathon” implies “running,” or that “rainy” implies “waterproof.”

Maya knew the solution. The industry was moving towards Vector Search, powered by Large Language Models (LLMs). She didn’t need to rip out her entire Redis infrastructure and deploy a specialized, unproven vector database. She just needed to upgrade to Redis Stack, which included the powerful RediSearch module, capable of storing and querying high-dimensional vectors natively in memory.

Conceptual Deep-Dive

To understand how Redis performs semantic search, we must first understand what a vector embedding is, and how we measure the “distance” between concepts.

1. The Geometry of Meaning: Vector Embeddings

An LLM (like OpenAI’s text-embedding-3-small or an open-source model like all-MiniLM-L6-v2) is a mathematical engine that translates human language into a coordinate in a high-dimensional space.

Imagine a simple 2-dimensional space where the X-axis represents “Sportiness” and the Y-axis represents “Waterproofness.”

• A standard leather dress shoe might be at coordinate [-0.8, -0.5] (not sporty, not waterproof).
• A lightweight mesh running shoe might be at [0.9, -0.8] (very sporty, not waterproof).
• A heavy rubber rain boot might be at [-0.5, 0.9] (not sporty, very waterproof).
• A waterproof trail running shoe (the exact item the user wanted) might be at [0.8, 0.8] (sporty AND waterproof).

When the user queries “footwear for a rainy marathon,” the LLM converts that sentence into a coordinate (a vector). Let’s say the query vector lands at [0.7, 0.9].

Semantic search is no longer about matching keywords. It is simply a geometry problem: Find the product points that are physically closest to the query point in this space. The closest point is the “Trail Runner,” even though the words don’t match.

Real embeddings don’t use 2 dimensions; they use hundreds or thousands (e.g., 1536 dimensions for OpenAI). This allows the model to capture incredibly subtle nuances of meaning, grammar, and context.

2. Measuring Distance

How does Redis calculate which vector is “closest”? It uses distance metrics. The two most common are:

1. Cosine Similarity (Cosine Distance): This measures the angle between two vectors, ignoring their magnitude (length). If two vectors point in the exact same direction, their cosine similarity is 1 (distance is 0). If they are orthogonal (90 degrees), similarity is 0. If they point in opposite directions, similarity is -1. This is the industry standard for text embeddings because it focuses on the “direction of meaning” rather than the length of the text.
2. Euclidean Distance (L2): This measures the straight-line physical distance between two points in space (using the Pythagorean theorem extended to $N$ dimensions).

3. The Search Problem: K-Nearest Neighbors (KNN)

If GlobalMart has 10 million products, finding the 10 closest products to a user’s query vector requires calculating the Cosine Distance between the query vector and every single one of the 10 million product vectors, and then sorting the results.

This is an $O(N)$ operation. In a 1536-dimensional space, calculating 10 million distances takes significant CPU time. It is too slow for a real-time search API. This exact, brute-force approach is called Flat Search (or exact KNN).

4. The Solution: HNSW (Hierarchical Navigable Small World)

To achieve sub-millisecond search times across millions of vectors, Redis uses an Approximate Nearest Neighbor (ANN) algorithm called HNSW.

HNSW is a graph-based algorithm. It builds a multi-layered graph of vectors, remarkably similar in concept to the Skip Lists we saw in Part 2 (Sorted Sets).

1. The Base Layer (Layer 0): Contains all the vectors. Each vector is connected to its closest neighbors via edges (like a web).
2. Upper Layers: Contain exponentially fewer vectors. They act as “expressways” across the semantic space.

When a search begins, Redis enters the graph at the highest layer (Layer 2). It looks at the nodes (A and C) and calculates the distance to the query vector (Q). If C is closer, it moves to C, then drops down to Layer 1.

At Layer 1, it checks C’s neighbors (A and D). If D is closer to Q, it moves to D, then drops down to Layer 0.

At Layer 0, it explores D’s neighbors (C and G) to find the absolute closest local minimum.

By navigating this hierarchical graph, HNSW avoids scanning all 10 million vectors. It only calculates distances for a tiny fraction of the nodes along the path. This provides logarithmic $O(\log N)$ search time. The trade-off is “Approximate”: it might occasionally miss the absolute closest vector if the graph topology is imperfect, but the speedup is astronomical (milliseconds instead of seconds).

Technical Explanation

Maya’s architecture for the new “Find Similar Products” feature involved two distinct phases:

1. Indexing (Write Path):

   • When a new product is added to PostgreSQL, an event is fired (via Redis Streams, Part 6).
   • A worker service consumes the event, takes the product description, and calls an Embedding API (like OpenAI or a local HuggingFace model) to generate a 1536-dimensional float array.
   • The worker stores the product metadata (JSON) and the raw vector byte array in a Redis Hash.
   • Redis Stack automatically detects the new Hash, extracts the vector, and inserts it into the HNSW graph index in the background.

2. Searching (Read Path):

   • The user types “rainy marathon footwear”.
   • The Spring Boot API calls the Embedding API to convert the query into a vector.
   • The API executes a RediSearch FT.SEARCH command against the HNSW index, passing the query vector.
   • Redis rapidly traverses the graph, finds the top 10 closest product IDs, retrieves their JSON metadata, and returns them to the user in milliseconds.

Step-by-Step Hands-On

Let’s see how Maya implemented this using Spring Boot and the modern redis.clients.jedis library (which provides excellent support for RediSearch modules).

Step 1: Creating the Vector Index

Before inserting data, Maya had to define the schema of the index. She specified that the description_vector field should be indexed using the HNSW algorithm and the COSINE distance metric.

import redis.clients.jedis.JedisPooled;
import redis.clients.jedis.search.IndexDefinition;
import redis.clients.jedis.search.IndexOptions;
import redis.clients.jedis.search.Schema;

public class VectorIndexService {

    private final JedisPooled jedis;
    private static final String INDEX_NAME = "product_idx";

    public VectorIndexService(JedisPooled jedis) {
        this.jedis = jedis;
    }

    public void createIndex() {
        try {
            // Check if index already exists to avoid errors on startup
            jedis.ftInfo(INDEX_NAME);
            System.out.println("Index already exists.");
            return;
        } catch (Exception e) {
            // Index does not exist, proceed with creation
        }

        // Define the schema: We index the title (as text) and the vector
        Schema schema = new Schema()
                .addTextField("title", 1.0)
                .addVectorField("description_vector", Schema.VectorField.VectorAlgo.HNSW,
                        Map.of(
                                "TYPE", "FLOAT32",
                                "DIM", 1536,          // OpenAI embedding dimension
                                "DISTANCE_METRIC", "COSINE"
                        ));

        // Tell RediSearch to automatically index any Hash whose key starts with "product:"
        IndexDefinition def = new IndexDefinition(IndexDefinition.Type.HASH)
                .setPrefixes(new String[]{"product:"});

        jedis.ftCreate(INDEX_NAME, IndexOptions.defaultOptions().setDefinition(def), schema);
        System.out.println("Vector index created successfully.");
    }
}

Step 2: Ingesting Data (The Write Path)

When a product was created, the worker service embedded the description and stored it in Redis.

import redis.clients.jedis.JedisPooled;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.HashMap;
import java.util.Map;

public class ProductIngestionService {

    private final JedisPooled jedis;
    private final EmbeddingClient embeddingClient; // Custom interface to OpenAI/HuggingFace

    public ProductIngestionService(JedisPooled jedis, EmbeddingClient embeddingClient) {
        this.jedis = jedis;
        this.embeddingClient = embeddingClient;
    }

    public void ingestProduct(String productId, String title, String description) {
        // 1. Generate the 1536-dimensional float array
        float[] vector = embeddingClient.embedText(description);

        // 2. Convert float[] to a byte array (required by Redis for binary safety)
        byte[] vectorBytes = floatArrayToByteArray(vector);

        // 3. Store in a Redis Hash
        Map<String, byte[]> hashData = new HashMap<>();
        hashData.put("title".getBytes(), title.getBytes());
        hashData.put("description".getBytes(), description.getBytes());
        hashData.put("description_vector".getBytes(), vectorBytes);

        String key = "product:" + productId;

        // HSET automatically triggers the RediSearch index update
        jedis.hset(key.getBytes(), hashData);
    }

    private byte[] floatArrayToByteArray(float[] input) {
        ByteBuffer buffer = ByteBuffer.allocate(input.length * 4);
        buffer.order(ByteOrder.LITTLE_ENDIAN); // Redis expects Little Endian
        for (float f : input) {
            buffer.putFloat(f);
        }
        return buffer.array();
    }
}

Step 3: Executing the Vector Search (The Read Path)

When a user searched, the API embedded their query and executed the FT.SEARCH command.

import redis.clients.jedis.JedisPooled;
import redis.clients.jedis.search.Document;
import redis.clients.jedis.search.Query;
import redis.clients.jedis.search.SearchResult;

import java.util.List;
import java.util.stream.Collectors;

public class SemanticSearchService {

    private final JedisPooled jedis;
    private final EmbeddingClient embeddingClient;

    public SemanticSearchService(JedisPooled jedis, EmbeddingClient embeddingClient) {
        this.jedis = jedis;
        this.embeddingClient = embeddingClient;
    }

    public List<String> searchProducts(String userQuery, int limit) {
        // 1. Embed the user's query into a vector
        float[] queryVector = embeddingClient.embedText(userQuery);
        byte[] queryVectorBytes = floatArrayToByteArray(queryVector);

        // 2. Construct the RediSearch query
        // The syntax means: "Find the K nearest neighbors to the vector parameter $vec"
        String queryString = "*=>[KNN " + limit + " @description_vector $vec AS vector_score]";

        Query query = new Query(queryString)
                .addParam("vec", queryVectorBytes)
                .returnFields("title", "vector_score") // Only return these fields
                .setSortBy("vector_score", true)       // Sort by closest distance
                .dialect(2);                           // Required for vector queries

        // 3. Execute the search against the HNSW index
        SearchResult result = jedis.ftSearch("product_idx", query);

        // 4. Map the results
        return result.getDocuments().stream()
                .map(doc -> String.format("Product: %s (Score: %s)",
                        doc.getString("title"),
                        doc.getString("vector_score")))
                .collect(Collectors.toList());
    }
}

Debugging/Troubleshooting Tips

• Memory Consumption: Vectors are huge. A 1536-dimensional float array takes ~6KB. 10 million products = 60GB of RAM just for the vectors, plus the overhead of the HNSW graph (which can double the memory usage). Solution: You must monitor memory closely (Part 4). Consider using smaller embedding models (e.g., 384 dimensions) or utilizing quantization techniques (storing floats as 8-bit integers) if absolute precision is less critical than memory cost.
• The Little Endian Trap: Redis expects binary vector data in Little Endian byte order. If your Java ByteBuffer uses Big Endian (the default in Java), the floats will be interpreted completely incorrectly by Redis, resulting in garbage search results and cosine distances that make no sense. Always explicitly set buffer.order(ByteOrder.LITTLE_ENDIAN).
• Graph Build Latency: When you insert a new vector, Redis must find its place in the HNSW graph. This takes CPU time. If you do a massive bulk import of 1 million products, the Redis CPU will spike, potentially impacting read latency. Solution: Throttle bulk imports, or perform them during off-peak hours.

Key Takeaways

• Semantic Search: LLM embeddings translate meaning into geometry. Search becomes finding the closest points in a high-dimensional space.
• Distance Metrics: Cosine Similarity measures the angle between vectors (meaning), while L2 measures physical distance.
• HNSW: A hierarchical graph algorithm that allows Redis to perform Approximate Nearest Neighbor (ANN) searches in logarithmic time, bypassing the $O(N)$ brute-force scan.
• Redis Stack: By using RediSearch, you can transform your existing caching layer into a high-performance, in-memory vector database without adding new infrastructure complexity.

Story Closing + Teaser

The new semantic search was a revelation. Users searching for “rainy marathon footwear” were instantly presented with the exact waterproof trail runners they needed. Conversion rates on obscure, long-tail search queries skyrocketed because the system finally understood intent, not just keywords.

But the VP of Product wasn’t finished. “This is great for the search bar. But what about a fully conversational AI shopping assistant? A chatbot that can answer complex questions, remember the user’s previous preferences, and recommend products based on our internal catalog policies, without hallucinating?”

Maya knew that simply hooking up an LLM to a chat interface was a recipe for disaster. The LLM would hallucinate product features or recommend items that were out of stock. She needed to ground the LLM’s responses in GlobalMart’s actual, real-time data.

She needed to build a Retrieval-Augmented Generation (RAG) architecture. She needed to combine the vector search she just built with metadata filtering (hybrid search) and use Redis to store the conversational memory of the chat sessions. The final, ultimate test of her Redis architecture awaited in the final chapter.

References

[1] Redis Vector Search Documentation. https://redis.io/docs/latest/develop/interact/search-and-query/advanced-concepts/vectors/ [2] Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (HNSW). Malkov, Yashunin, 2016.