Skip to main content

Document fetch check

Still in pre-retrieval — still no database hit required if the answer is "no." Multi-turn RAG is not the same problem as single-shot Q&A. In a single shot you always retrieve. In a conversation, the user's next message might be:

  • A brand-new question about a different topic ("What does policy X say about remote work?").
  • A follow-up that still needs the same documents ("Can you quote the paragraph that says that?").
  • A clarification that should reuse context already in history ("Make that shorter" / "Translate it to Dutch").

If you retrieve on every turn, you pay latency and you risk swapping out a good document set for a worse one just because the follow-up phrasing embeds differently. If you never retrieve after the first turn, the system cannot change topic. The document fetch check sits between those extremes.

The decision problem​

RAG Me Up treats "should I fetch?" as an LLM classification over the current question given the conversation history. The prompt (rag_fetch_new_question) is required to force a yes/no style answer. Conceptually:

Given what we already talked about and which documents we already have in context, does this new user message require a fresh retrieval?

Only when history is non-empty is this check performed. An empty history always fetches.

Implementation​

fetch_new_documents = True
if len(history) > 0:
    # ... optional summarization ...
    (response, _) = self.llm.generate_response(
        None,
        os.getenv("rag_fetch_new_question").format(question=prompt),
        history
    )
    if response.lower().strip().startswith("no"):
        fetch_new_documents = False

The server then branches:

  • fetch_new_documents == True — continue into HyDE (optional), hybrid retrieval, reranking, rewrite loop, and eventually answer with a fresh document set injected into the system prompt.
  • fetch_new_documents == False — skip retrieval entirely and answer using the existing history (which still contains the earlier system prompt / documents the client kept). On the HTTP boundary, the previously provided docs from the client are returned again so the UI does not blank out citations.

That last point matters: because the Python server is stateless, "documents already retrieved" live in the client. The fetch check decides whether the server should ignore them and get new ones, or trust them.

Prompt design tips​

The default rag_fetch_new_question asks the model to answer with yes or no plus a short motivation. The code only inspects whether the reply starts with no (case-insensitive). Everything else is treated as "yes, fetch". That is deliberately conservative: ambiguous answers fail open into retrieval rather than skipping it.

When you customize this prompt:

  • Be explicit that the only valid leading tokens are yes or no.
  • Give the model a clear notion of what counts as a follow-up (clarification, reformulation, ask-for-quote) versus a new information need.
  • Remember the history is passed as chat messages, so the model already sees prior turns; you do not need to paste history into the question template.

Failure modes to watch for​

  • Over-fetching — the model says "yes" to almost everything. Latency goes up; answers may thrash between document sets. Tighten the prompt with concrete follow-up examples.
  • Under-fetching — the model says "no" when the user changed topic. Users then get confident answers grounded in the wrong documents. Prefer failing open, and log the yes/no decision in development (logging_level=DEBUG).
  • Language mismatch — if users chat in a language different from your prompt, yes/no detection can get noisier. Keep the forced answer format language-agnostic (yes/no) even if the motivation is multilingual.