Reranking

Bi-encoder retrieval (what your embedding model does) encodes the query and each chunk independently. That is why indexing can be offline and why query-time search is fast: you compare vectors that never attended to each other during encoding. The price is that fine-grained interactions between a specific question and a specific passage — negation, constraint words, cross-sentence references — are only approximated in the vector space.

A reranker encodes the query and a candidate passage together. In cross-encoder terms that means full cross-attention across query tokens and passage tokens in one forward pass. That joint representation is much better at deciding "is this passage actually useful for this question?" — and far too slow to run over your entire corpus.

Why not rerank the full collection?

Suppose you have 200 000 chunks. A cross-encoder forward pass per chunk at query time is prohibitive: latency goes from tens of milliseconds to minutes, and GPU/CPU cost explodes. Even 10 000 candidates is usually too many.

So production systems use a cascade:

1. Cheap hybrid retrieval recalls a broad candidate set (vector_store_k, often 10–30).
2. An expensive reranker scores only those candidates (cross-attention over query+passage).
3. You keep the top rerank_k (often 3–5) for the prompt.

flowchart LR
  C[All chunks in Postgres] -->|hybrid top-k| W[Wide candidate set]
  W -->|cross-encoder| N[Narrow set for the prompt]

Hybrid search optimizes for recall (don't miss the right chunk). Reranking optimizes for precision among survivors (put the best ones first, drop the rest).

Why cross-attention is more accurate

In a bi-encoder, similarity is roughly sim(encode(q), encode(d)). Negation in the query ("employees who are not eligible") cannot re-weight individual tokens inside d at comparison time — that information had to already be baked into both vectors.

In a cross-encoder, tokens from q and d sit in the same transformer input. Attention can align "not eligible" with the eligibility clause in the passage, or decide that a topically similar paragraph never states the asked fact. That is why rerankers usually lift nDCG / MRR on retrieval benchmarks relative to bi-encoder-only baselines — at the cost of compute linear in the number of candidates.

How RAG Me Up reranks

Reranking is optional (rerank=True). When enabled, RAGHelper constructs a Reranker that wraps FlashRank:

class Reranker:
    def __init__(self):
        self.reranker = Ranker(os.getenv("rerank_model"), cache_dir="flashrank")

    def rerank_documents(self, documents, prompt):
        passages = [
            {"id": i, "text": doc["content"], "metadata": doc["metadata"]}
            for (i, doc) in enumerate(documents)
        ]
        rerank_request = RerankRequest(query=prompt, passages=passages)
        rerank_results = self.reranker.rerank(rerank_request)
        rerank_results.sort(key=lambda x: x['score'], reverse=True)
        ...
        return rerank_results

In handle_documents:

documents = self.retriever.get_relevant_documents(prompt, prompt_embedding, datasets)
if os.getenv("rerank") == "True":
    documents = self.reranker.rerank_documents(documents, prompt)[:int(os.getenv("rerank_k"))]
else:
    documents = [{**document, "score": document['metadata']['distance']} for document in documents]

Without reranking, the hybrid score already stored as metadata['distance'] becomes score. With reranking, FlashRank scores take over and the list is truncated to rerank_k.

Which rerankers are supported?

RAG Me Up's current path is FlashRank-compatible checkpoints set via rerank_model. The .env.template default is:

rerank_model=ms-marco-MiniLM-L-12-v2

FlashRank ships lightweight ONNX-friendly ranking models (MiniLM-style MS MARCO trained models are the usual starting point). Model files are cached under the server's flashrank/ directory so subsequent startups do not re-download.

Practical choices:

Goal	Guidance
Speed / CPU	Smaller MiniLM FlashRank models (default is already in this family)
Accuracy	Larger FlashRank-compatible MS MARCO models if latency allows
Multilingual	Prefer a multilingual ranking checkpoint that FlashRank can load; validate on your language

If you fork the stack, older RAG Me Up generations also experimented with LangChain cross-encoders and ColBERT-style rerankers. The open-source server you have now is wired through FlashRank — stick to models that library accepts unless you change Reranker.py.

Power vs speed: when to use it

Use reranking when:

• Answer quality matters more than shaving ~50–200 ms.
• vector_store_k is comfortably larger than rerank_k (the cascade actually discards junk).
• Your eval set shows hybrid-only top-3 missing relevant chunks that appear at ranks 5–15.

Skip or disable reranking when:

• You are debugging retrieval in isolation and want raw hybrid ordering.
• You run on constrained CPU and already keep vector_store_k tiny (e.g. 3).
• Measured gains on your corpus are negligible — it happens for some FAQ-style collections.

A healthy production setup almost always has vector_store_k > rerank_k. Example: retrieve 15, keep 3. If the two values are equal, the reranker only reorders; it never discards weak hits.

Relation to provenance

The same Reranker instance can later be reused for provenance attribution when provenance_method=rerank. There the "query" becomes the answer (optionally plus the question): you are asking how relevant each chunk was to what was generated, not to what was asked.