Building the RAG Pipeline: Vector DBs, Embeddings, and Memory

Ella announced Companion in February — an AI that actually remembers, built on retrieval-augmented generation over all your conversations. I've been quiet about the engineering side because the architecture was settling. It's settled now. This post is the infrastructure under the announcement.

Want the vision? Read Ella's post. Want to know which vector database we chose and why chunking matters more than embedding model selection? Keep reading.

The Problem Statement

Companion stores every conversation and retrieves the right fragments at the right moment — not by keyword match, but by semantic relevance. "The deployment issue" should surface the conversation about the Kubernetes problem from three weeks ago, even if nobody used that word back then.

This is RAG: embed everything, store the embeddings in a vector database, query with the current context, and feed results into the model's prompt as retrieved memory. Simple in concept, fiddly in practice.

Choosing the Vector Database

I evaluated three options seriously:

pgvector (PostgreSQL extension). Stores vectors in the same database as the rest of the application. Operational simplicity — one database, joins between vectors and metadata are trivial, backups are unified.

Qdrant. Purpose-built vector database, written in Rust. Fast, clean API, good metadata filtering. No relations to speak of.

Milvus. The heavy option — distributed, designed for very large scale, more moving parts. Impressive benchmarks, real operational complexity.

The decision came down to scale and operational overhead. We're not at the scale where Milvus earns its complexity. Between pgvector and Qdrant, I went with pgvector for a specific reason: Companion's memory isn't just vectors. It's vectors plus structured metadata — timestamps, participants, conversation IDs, entity tags, summary text. Having all of that in one relational database, queryable with standard SQL alongside the vector similarity search, is worth more than the raw throughput Qdrant would give us.

-- "Find conversations semantically similar to this one,
--  involving this user, from the last 30 days"
SELECT id, summary, created_at
FROM memories
WHERE user_id = $1
  AND created_at > NOW() - INTERVAL '30 days'
ORDER BY embedding <=> $2
LIMIT 10;

The <=> operator is pgvector's cosine distance. That query — semantic search filtered by structured criteria — is the heart of Companion's retrieval. Doing it in one database is the whole game.

If we outgrow pgvector, migrating to Qdrant is a defined project, not a rewrite. The interfaces are abstracted. But I don't think we'll need to soon.

Embedding Model Selection

I tested several:

• OpenAI text-embedding-3-small — fast, cheap, good baseline.
• OpenAI text-embedding-3-large — better quality, higher dimensions, more storage.
• bge-large-en-v1.5 (BAAI, open source) — runs locally, competitive quality.
• e5-large-v2 — solid, open source, also local.

For privacy and cost reasons — and because we have a GPU sitting there — we went with bge-large-en-v1.5 on the local vLLM box. Quality was within shouting distance of OpenAI's large model on our eval set, costs nothing per token, and conversation data never leaves the network.

The general lesson: don't over-index on embedding model benchmarks. The difference between a good model and a slightly better one is smaller than the difference made by chunking and retrieval tuning.

Chunking Strategy

This is where most RAG systems go wrong, and where I spent the most time.

The naive approach — fixed-size chunks (say, 500 tokens) — is bad. Conversations aren't documents. They're turn-based, context-dependent, full of references to earlier turns. A 500-token window that splits mid-thought produces garbage embeddings.

What works for Companion:

• Turn-aware chunking. Chunk boundaries respect conversation turns. A chunk is one or more complete turns, never half a turn.
• Contextual overlap. Adjacent chunks share a small amount of context — the last turn of the previous chunk is included as preamble in the next. This preserves referential continuity.
• Summary augmentation. Each chunk is stored alongside a generated summary (produced by the subconscious process Ella described). The summary is what gets embedded; the raw text is stored as payload. This means retrieval matches on meaning, not on specific wording.

The code, simplified:

def chunk_conversation(turns: list[Turn], max_tokens: int = 512) -> list[Chunk]:
    chunks = []
    current = []
    current_len = 0
    for turn in turns:
        turn_len = count_tokens(turn.text)
        if current_len + turn_len > max_tokens and current:
            chunks.append(assemble_chunk(current))
            # Carry last turn as overlap context
            current = [current[-1]]
            current_len = count_tokens(current[-1].text)
        current.append(turn)
        current_len += turn_len
    if current:
        chunks.append(assemble_chunk(current))
    return chunks

Retrieval Tuning

Retrieval is where you tune the experience. Raw cosine similarity returns mathematically close vectors, but "closest in vector space" isn't always "most relevant." A few adjustments:

• Recency weighting. Recent memories get a small boost — people care about this week more than eight months ago, all else equal.
• Diversity enforcement. If the top 10 results are all from the same conversation, that's a replay, not retrieval. I cap results per source and prefer a spread.
• Re-ranking. A lightweight cross-encoder re-ranks the top candidates. More accurate than pure similarity, fast enough for the request path.

The final retrieval query blends these signals into a score, and the top N chunks become the "remembered context" injected into Companion's prompt.

What's Next

The pipeline works. Companion remembers, retrieves, and surfaces relevant context across conversations. The architecture Ella described — the subconscious process, the vectorized memory — all runs on this foundation.

Next challenges: scale (the database grows forever), quality (retrieval relevance is good but not great), and the subconscious process Ella will detail. My job is making sure the infrastructure keeps up with the ambition.

It's almost 8 PM. The embeddings are indexed. I'm going home.