LLM backends

Every generative step in RAG Me Up — answering, HyDE drafts, fetch-check decisions, rewrite judgements, summarization, LLM provenance — goes through LLMHelper. Understanding that class is understanding how provider choice, history handling, and streaming interact with the rest of the pipeline.

One helper, many providers

At construction time, LLMHelper reads boolean flags in .env and initializes exactly one client:

Flag	Provider	Extra env you must set
use_openai	OpenAI Chat Completions	OPENAI_API_KEY, openai_model_name
use_azure	Azure OpenAI	AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_CHAT_DEPLOYMENT_NAME, AZURE_OPENAI_API_VERSION
use_gemini	Google Gemini	GOOGLE_API_KEY, gemini_model_name
use_anthropic	Anthropic Messages API	ANTHROPIC_API_KEY, anthropic_model_name
use_ollama	Local Ollama	ollama_model_name

If none is True, initialization raises. Enable one provider; the first matching branch in initialize_client wins.

Instruct models only

As covered in An introduction to RAG, RAG Me Up expects instruct (chat) models. Foundation completion models that only continue text behave poorly with system/user/assistant threads. For Ollama, pick instruction-tuned tags (e.g. Gemma / Llama instruct variants).

How a call is assembled

generate_response(system_prompt, prompt, history):

1. If system_prompt is not None, it becomes (or replaces) the first system message.
2. Existing history is appended.
3. The new user prompt is appended.
4. The provider API runs with temperature from .env.

That is why prompt creation can pass system_prompt=None on follow-ups that should keep the previous system message: history already carries it.

Providers disagree on schemas; LLMHelper adapts:

• OpenAI / Azure — standard role / content dicts.
• Gemini — UserContent / ModelContent, system via system_instruction.
• Anthropic — top-level system=, content blocks for messages.
• Ollama — small OllamaClient wrapper around the ollama package.

Streaming

generate_response_stream mirrors the same thread construction but returns a generator of text chunks. Flask /chat_stream turns those into SSE token events so the UI can render partial answers. Anthropic requires max_tokens; Gemini uses generate_content_stream; call sites in RAGHelper stay provider-agnostic.

Cost multiplier — easy to underestimate

Each optional feature can add an LLM call on top of the final answer:

Feature	Extra call?
Summarization	Yes, when threshold exceeded
Fetch check	Yes, when history non-empty
HyDE	Yes, before retrieval
Rewrite judge + rewrite	Up to two calls
Final answer	Always
LLM provenance	One call per retrieved chunk

For production, pick a strong model for judges/rewrites if quality matters, and be deliberate about which optional stages stay on.

Reloading at runtime

RAGHelper.reload_llm reconstructs LLMHelper (and optionally reranker / provenance / summarization encoder) after config changes — useful when the UI's config page updates .env without restarting the process.

Attention provenance caveat

Attention-based provenance needs access to raw attention tensors from a local Hugging Face causal LM (output_attentions=True). Hosted APIs (OpenAI, Anthropic, Gemini) and typical Ollama HTTP usage do not expose those weights. If you need attention provenance, you must run an open-weight model in-process the way earlier RAG Me Up local backends did — see that page for the algorithm in detail.