HyDE (Hypothetical Document Embeddings)

There is a structural mismatch at the heart of dense retrieval that most RAG tutorials gloss over. During indexing you embed document chunks — passages that look like answers, explanations, definitions, policy paragraphs. At query time you embed a question. Cosine similarity then compares those two representations as if they lived in the same semantic neighborhood.

Often they do well enough. Often they do not — especially when questions are short, underspecified, or phrased very differently from the corpus style. HyDE exists to fix that mismatch inside RAG Me Up, without changing your index.

Document vectors and question vectors are not the same thing

An embedding model maps text to a point in a high-dimensional space. Points that are "about the same thing" end up nearby. That works beautifully when you compare like with like:

• Two policy paragraphs about parental leave → close together.
• Two user questions about parental leave → close together.

It works less well when you compare unlike with unlike:

• Policy paragraph: "Partners employed for at least 26 weeks are entitled to 1 week of partner leave at 100% pay, extendable…"
• User question: "partner leave?"

Both are "about partner leave", but their surface form, length, information density, and token distribution differ. The embedding of a question is shaped by interrogative structure, missing arguments, and keyword-ish phrasing. The embedding of a chunk is shaped by assertive prose. Retrieval then asks: which answer-like vectors are nearest to this question-like vector?

That is the apples-to-oranges problem. Dense retrieval assumes the two distributions overlap enough for nearest-neighbor search to work. For many corpora they do. For stylistically distant corpora (legal, medical, internal jargon, long-form policies) the overlap shrinks and good chunks get buried.

Sparse BM25 partly compensates when the question shares keywords with the chunk — which is one reason RAG Me Up always does hybrid search. But keyword overlap does not fix the dense half of the score, and it fails when users paraphrase without reusing the document's terms.

What HyDE does about it

HyDE (Hypothetical Document Embeddings) changes what you embed at query time, not what you stored at index time.

Instead of embedding the question, you ask an LLM to generate a short document that would answer the question, and you embed that. Retrieval then compares document-like text to document-like text:

1. User asks: "What's our parental leave policy for partners?"
2. LLM drafts a plausible HR-style paragraph that would contain the answer.
3. That draft is embedded with the same SentenceTransformer used at indexing.
4. Hybrid search runs against the draft (dense + BM25 on the draft's wording).
5. Real chunks from Postgres that resemble the draft are returned.

Crucially, the hypothetical document is not shown to the user as the answer. It is only a better query representation — a bridge from question-space into document-space.

flowchart LR
  Q[User question] --> H[LLM: draft hypothetical doc]
  H --> E[Embed draft]
  E --> R[Hybrid retrieval]
  R --> C[Real chunks from Postgres]
  C --> A[Answer LLM]

How RAG Me Up applies HyDE

HyDE is optional (use_hyde) and runs only when new documents will be fetched — after summarization and the fetch check, before retrieval:

if os.getenv("use_hyde") == "True":
    (response, _) = self.llm.generate_response(
        None,
        os.getenv("hyde_query").format(question=prompt),
        []
    )
    prompt = response

prompt_embedding = self.embeddings.encode(prompt)
documents = self.handle_documents(prompt, prompt_embedding, datasets)

After this block, the variable still named prompt no longer holds the user's question — it holds the hypothetical document. That string is what gets:

• embedded for dense search,
• passed as the text query for BM25,
• used as the query string for optional reranking.

Interactions with other steps

HyDE disables two other optional enhancements when it is on:

• The rewrite loop is skipped. You already transformed the query once; judging "can these docs answer the hypothetical document?" and rewriting again is the wrong abstraction.
• Re2 is also skipped. Re-reading a hypothetical document is not the same pedagogical trick as re-reading the user's question.

Those guards live in RAGHelper.handle_user_interaction: rewrite and Re2 only run when use_hyde is not True.

Configuration

Variable	Role
use_hyde	Enable/disable HyDE
hyde_query	Prompt that must contain {question}

A good HyDE prompt asks for a short document in the style of your corpus, with a hard length cap (the template defaults to at most 1000 characters). Longer drafts waste tokens and drift away from the question. If your corpus is Dutch HR policy, ask for a Dutch policy-style paragraph — not a generic English Wikipedia blurb.

When HyDE helps — and when it hurts

HyDE tends to help when:

• Questions are short or keyword-like but documents are prose-heavy.
• There is stylistic distance between how users ask and how documents are written.
• Dense retrieval alone underperforms on your eval set even though BM25 sometimes saves you.

HyDE tends to hurt when:

• The LLM hallucinates a hypothetical document in the wrong domain, steering both dense and sparse retrieval toward confidently irrelevant chunks.
• Latency budgets are tight (HyDE adds a full LLM call before any database hit).
• Your corpus is already question–answer formatted (FAQs), so question-to-chunk similarity was fine to begin with.

Treat HyDE as a hypothesis to validate with a small labeled question set — not as a default you leave on because the vector-space story is elegant. The story is correct; the win is empirical.