History summarization

Pre-retrieval starts with the conversation you already have — not with the database. Large Language Models have context windows measured in tens or hundreds of thousands of tokens. That sounds generous until you remember what a RAG prompt actually contains: a system instruction, retrieved document chunks, the current question, and the full prior conversation. Every follow-up turn makes that prompt larger. Even when the hard limit is not hit yet, long histories dilute attention — the model has more room to lose the thread of what matters.

History summarization is the optional first gate in the pre-retrieval phase. Instead of always shipping the full transcript to later LLM calls (fetch check, answer generation), you compress older turns once the conversation exceeds a token budget.

The idea

Summarization in a chat setting is not the same task as summarizing a news article. You need a short rewrite that:

• Preserves facts, decisions, and constraints the user already established.
• Keeps the language of the conversation (users notice when a Dutch chat suddenly gets an English summary injected).
• Remains usable as history — i.e. something you can put back into the message list and continue from.

RAG Me Up does this only when there is already history and when summarization is enabled in the .env file.

How RAG Me Up decides to summarize

Inside handle_user_interaction (and the streaming twin), the pipeline first checks whether summarization is on. If it is, the history is flattened into a single string and tokenized with tiktoken using the encoder named in summarization_encoder:

if os.getenv("use_summarization") == "True":
    history_string = "\n\n".join(
        [f"{message['role']}: {message['content']}" for message in history]
    )
    history_size = len(self.tiktoken_encoder.encode(history_string))
    if history_size > int(os.getenv("summarization_threshold")):
        (response, _) = self.llm.generate_response(
            None,
            os.getenv("summarization_query").format(history=history_string),
            []
        )
        history = history[:1] + [{"role": "assistant", "content": response}]

A few design choices are worth calling out:

• The threshold is measured in tokens, not characters. Character counts are a poor proxy across languages and tokenizers.
• When a summary is produced, RAG Me Up keeps the first history message and replaces the rest with a single assistant message containing the summary. That keeps a lightweight anchor of the conversation start while collapsing everything else.
• Summarization runs before the document-fetch check, so later steps see the compressed history.

Configuration

Variable	Role
use_summarization	True / False toggle
summarization_threshold	Token count that triggers summarization
summarization_query	Prompt template; must include {history}
summarization_encoder	tiktoken model name used only for counting

Leave a buffer between summarization_threshold and your LLM's true context window. The summarization prompt itself, the RAG instruction, and retrieved documents still need room after compression.

When to turn it on

For short, task-oriented chats (a handful of turns), summarization is usually unnecessary overhead — an extra LLM call on every long turn. It becomes valuable when:

• Users hold long investigative conversations over the same corpus.
• Your RAG instruction plus top-k chunks already consume a large fraction of the window.
• You care more about staying within limits than about perfect fidelity of early turns.

If you need perfect auditability of every prior user message, summarization is the wrong tool; persist the full transcript in your UI/database and only summarize the prompt view sent to the model — which is effectively what RAG Me Up's client/server split already enables.