Skip to main content

Prompt injection — how documents enter the LLM

After optional query-time machinery has run — summarization, fetch check, HyDE, retrieval, rerank, rewrite, Re2 — RAG Me Up still has to do what users think of as "the AI": build a prompt and call the LLM. This page is about how evidence is injected, with concrete examples of the resulting message thread.

The three prompt roles​

RAG Me Up separates concerns that beginners often jam into one mega-string:

  1. rag_instruction — system prompt: role, grounding rules, and {context} (formatted chunks).
  2. rag_question_initial — how the first user question is presented ({question}).
  3. rag_question_followup — how subsequent turns are presented ({question}).

Formatting documents into context​

Chunks are not dumped raw. format_documents wraps each one in an explicit block with filename and metadata:

[Document] *Filename* `data/hr/partner-leave.json`
*Content*: Partners employed for at least 26 weeks are entitled to ...
*Metadata* source: data/hr/partner-leave.json, dataset: hr, distance: 0.82 [/Document]

[Document] *Filename* `data/hr/expense-policy.json`
*Content*: Expense claims above €500 require director approval ...
*Metadata* source: data/hr/expense-policy.json, dataset: hr, distance: 0.71 [/Document]

Those markers make citation instructions enforceable: the filename sits next to the content in a predictable schema.

Worked example — first turn​

Suppose the user asks "Who approves expense claims over €500?", Re2 is on, and two chunks were retrieved.

System message (rag_instruction with {context} filled):

Instruction: You are a digital librarian that can answer generic questions on
relevant content quickly and succinctly. Here are a few documents from the
library that you can use to answer the user's question...

[Document] *Filename* `data/hr/expense-policy.json`
*Content*: Expense claims above €500 require director approval...
...
[/Document]

User message (rag_question_initial with Re2-enhanced {question}):

The initial question you have to answer:

Who approves expense claims over €500?
Read the question again: 
Who approves expense claims over €500?

LLMHelper.generate_response then builds the chat thread:

[system]  <rag_instruction + documents>
[user]    <rag_question_initial>

and returns (answer, thread).

Worked example — follow-up that needs new documents​

User continues: "And what about partner leave?" The fetch check says yes → retrieve again → rebuild system prompt with the new context, strip the old system message from history:

[system]  <rag_instruction + NEW documents about partner leave>
[user]    <earlier user turns without old system>
[assistant] <earlier answers>
[user]    <rag_question_followup for partner leave>

This prevents the model from seeing two conflicting document sets in one thread.

Worked example — follow-up that reuses documents​

User: "Quote the sentence that says that." Fetch check says no → system_prompt=None, full prior history (including the earlier system prompt with documents) is reused:

[system]  <still the previous documents>
...history...
[user]    The follow-up question you have to answer:

Quote the sentence that says that.

What belongs in rag_instruction​

  • Role and domain ("digital librarian for internal HR policies").
  • Grounding rules ("answer only from the documents"; "say when you cannot find it").
  • Citation rules ("always mention the filename you used").
  • Style (length, match the user's language).

Place {context} where documents should be treated as reference material — typically after the role description.

Temperature​

temperature from .env is applied inside LLMHelper. For factual RAG, start low (0–0.3). Streaming (/chat_stream) uses the same prompt construction; only token transport changes.