RAG, Long Context or Tools? Deciding with Retrieval Evals

Three architectures, one set of questions

When a team asks me for "RAG over our docs", I first check whether retrieval is the right shape at all. There are three honest options, and most real systems use two of them:

• Long context: put the whole (permitted) corpus in the prompt on every request. No index, no chunking, nothing to go stale.
• RAG: retrieve a handful of chunks per question. Smaller prompts, but you now own an index, an ingestion pipeline and a retrieval quality problem.
• Tools: answer record questions ("what is the status of invoice 2041?") with a SQL query or an API call. Exact, live and permission-checked by the system that owns the data.

The choice falls out of corpus size against the context budget, per-user permissions, update frequency, the share of record lookups, and latency and cost per request. Once RAG is chosen, whether the retriever finds the right documents is a measurable question, and the measurement belongs in CI.

Everything below is a reference implementation I built and ran for this article, with pytest tests and no real model or embedding API calls.

Start with the token budget

"Does it fit in the context window?" is the wrong question. The right one is whether it fits in what is left after the answer, the system prompt, tool schemas, the user's question and a margin for tokenizer drift. I count with tiktoken because the provider's own count can differ from mine, and I would rather find out from a 10% margin than from a 400 error in production.

# ...
ENC = tiktoken.get_encoding("o200k_base")

def count_tokens(text: str) -> int:
    return len(ENC.encode(text))

@dataclass(frozen=True)
class ContextBudget:
    context_window: int          # model limit, input + output
    max_output_tokens: int       # what you reserve for the answer
    system_prompt_tokens: int    # instructions, tool schemas, few-shot examples
    max_question_tokens: int = 500
    safety_margin: float = 0.10  # tokenizer drift between your counter and the provider's

@property
    def available_for_documents(self) -> int:
        usable = int(self.context_window * (1 - self.safety_margin))
        return usable - self.max_output_tokens - self.system_prompt_tokens - self.max_question_tokens

def fits(self, corpus_tokens: int) -> bool:
        return corpus_tokens <= self.available_for_documents

@dataclass(frozen=True)
class Prices:
    """USD per million tokens. Pass your provider's current numbers; none are baked in."""
    input_per_mtok: float
    output_per_mtok: float
    cached_input_per_mtok: float | None = None

def request_cost(input_tokens: int, output_tokens: int, prices: Prices,
                 cached_input_tokens: int = 0) -> float:
    if cached_input_tokens > input_tokens:
        raise ValueError("cached tokens cannot exceed input tokens")
    cached_rate = prices.cached_input_per_mtok
    if cached_rate is None:
        cached_rate, cached_input_tokens = prices.input_per_mtok, 0
    uncached = input_tokens - cached_input_tokens
    return (uncached * prices.input_per_mtok
            + cached_input_tokens * cached_rate
            + output_tokens * prices.output_per_mtok) / 1_000_000

Prices are parameters on purpose: price sheets change more often than code, and a hardcoded price quietly turns a cost model into fiction. The cached-input rate matters because long context lives or dies on prompt caching, which most providers bill at a lower rate.

Latency follows the same token counts, because time to first token grows with input size. I do not guess a latency model: I measure p95 time to first token with the real provider at two prompt sizes and pass the largest input that meets the SLO as max_input_tokens_for_slo. RAG's extra hop (embedding call plus database query) gets measured the same way.

Permissions, freshness and records change the answer

The decision function takes a Workload: the corpus one user may see, budget, prices, daily volume, per-user permissions, structured-question share, update frequency, measured RAG context size, and the SLO and spend limits.

def recommend(w: Workload) -> Recommendation:
    reasons: list[str] = []
    prompt_overhead = w.budget.system_prompt_tokens + w.budget.max_question_tokens
    lc_input = w.corpus_tokens + prompt_overhead
    rag_input = w.rag_context_tokens + prompt_overhead

# A shared, cacheable prefix only exists when everyone sees the same corpus.
    lc_cached = 0 if w.per_user_permissions else w.corpus_tokens
    lc_cost = request_cost(lc_input, w.output_tokens, w.prices, lc_cached) * w.requests_per_day
    rag_cost = request_cost(rag_input, w.output_tokens, w.prices) * w.requests_per_day

if not w.budget.fits(w.corpus_tokens):
        primary = "rag"
        reasons.append(f"corpus {w.corpus_tokens} tokens > {w.budget.available_for_documents} available")
    elif lc_input > w.max_input_tokens_for_slo:
        primary = "rag"
        reasons.append(f"{lc_input} input tokens per request breaks the latency SLO")
    elif lc_cost > w.daily_budget_usd:
        primary = "rag"
        reasons.append(f"long context costs {lc_cost:.2f}/day, budget {w.daily_budget_usd:.2f}")
    else:
        primary = "long_context"
        reasons.append("fits, meets SLO and budget: no index to keep fresh")

if primary == "rag" and w.per_user_permissions:
        reasons.append("filter by ACL inside the retrieval query, not after it")
    if primary == "rag" and w.updates_per_day > 0:
        reasons.append("needs incremental ingestion; a stale index is a silent failure")
    if primary == "long_context" and w.updates_per_day > 0 and lc_cached:
        reasons.append("each update invalidates the cached prefix; re-check cost")

add_tools = w.structured_share >= 0.2
    if add_tools:
        reasons.append("route record lookups to a SQL/API tool, not retrieval")
    return Recommendation(primary, add_tools, lc_cost, rag_cost, reasons)

The tests pin down the scenarios that matter. With placeholder prices of 2.00 input, 0.50 cached input and 8.00 output per million tokens (arithmetic inputs, not anyone's price sheet), a 60,000-token handbook that every user may read, 2,000 requests a day and 400-token answers:

• Long context with a fully cached prefix costs 80.40 a day. RAG with five 400-token chunks costs 28.40 a day. Under a 150.00 daily budget, long context wins because it removes the index entirely.
• Turn on per-user permissions and the shared prefix disappears, since every user's prompt is different. Long context jumps to 260.40 a day, breaks the budget, and the recommendation flips to RAG with an ACL filter.

The cached figure is a best case: real prompt caches have time-to-live windows and minimum prefix lengths, and every document update invalidates the prefix.

Permissions are the factor people underrate. Filter after retrieval and top-k fills with documents the user cannot see, so recall quietly collapses for that user. The filter belongs inside the retrieval query. I cover that, and incremental ingestion, in keeping a knowledge base fresh.

Structured questions are the other trap. "How much did we invoice Acme last quarter?" is not retrieval: embedded rows are a stale, paraphrasable snapshot of a number the database knows exactly. It goes to a tool with its own permission check, which is the split I use for support agents in the LangGraph handoff article and in Semai AI Support (status: Built), where retrieval, tools and human handoff sit side by side.

Build the golden set before the retriever

The golden set is a list of real questions labelled with the documents that answer them. I write it before tuning, because afterwards labels drift towards whatever the current retriever returns.

For this article I generated a small synthetic help centre for a fictional invoicing product: 15 hand-written articles (refunds per plan, SSO per identity provider, exports, billing, account), 10 error-code pages generated from a table, and 30 seeded "release note" distractors that reuse the same vocabulary and mention error codes without explaining them. That is 55 documents and 1,950 tokens with o200k_base, which fits in any modern context window. The toy corpus exists to show the evaluation method, not to argue for RAG.

{"id": "q01", "kind": "identifier", "text": "What does E-1004 mean?", "relevant": {"error-e-1004": 2}}
{"id": "q02", "kind": "identifier", "text": "E-1009 when logging in with Okta", "relevant": {"error-e-1009": 2, "sso-okta": 1}}
{"id": "q07", "kind": "paraphrase", "text": "can I get my money back on the team plan", "relevant": {"refund-team": 2}}
{"id": "q08", "kind": "paraphrase", "text": "the API keeps throttling us", "relevant": {"error-e-1003": 2}}
{"id": "q17", "kind": "keyword", "text": "Okta SAML setup", "relevant": {"sso-okta": 2}}

The labelling rules I follow:

• Graded relevance. 2 means the document answers the question on its own. 1 means it helps. Binary labels cannot tell "found the answer" from "found something nearby".
• Every query has at least one grade-2 document. If nothing answers it, it is a content gap, not a retrieval test.
• A `kind` on every query. Here that is identifier, paraphrase and keyword (6, 9 and 9 queries). The slices matter more than the average, as the results below show.
• The loader validates labels against the corpus. When a document is deleted or renamed, load_golden raises instead of quietly scoring that query as a miss forever.

In a real system the queries come from tickets and search logs, and every production miss becomes a new golden query.

Recall@k, MRR and nDCG@k

Each metric answers a different question, so I track all three:

• Recall@k: did the right documents make it into the k chunks the model will see? If not, no prompt can fix the answer.
• MRR (mean reciprocal rank): how high is the first relevant hit? This matters when the prompt or reranker favours the top result.
• nDCG@k: is the ranking good once graded relevance is counted? It rewards putting the grade-2 document above the grade-1 one.

import math

def recall_at_k(ranked: list[str], relevant: dict[str, int], k: int) -> float:
    wanted = {d for d, g in relevant.items() if g > 0}
    return len(wanted & set(ranked[:k])) / len(wanted)

def reciprocal_rank(ranked: list[str], relevant: dict[str, int], k: int = 10) -> float:
    for i, doc_id in enumerate(ranked[:k], start=1):
        if relevant.get(doc_id, 0) > 0:
            return 1.0 / i
    return 0.0

def ndcg_at_k(ranked: list[str], relevant: dict[str, int], k: int) -> float:
    def dcg(grades: list[int]) -> float:
        return sum((2**g - 1) / math.log2(i + 2) for i, g in enumerate(grades))

actual = dcg([relevant.get(d, 0) for d in ranked[:k]])
    ideal = dcg(sorted(relevant.values(), reverse=True)[:k])
    return actual / ideal if ideal else 0.0

I test these against hand-computed values; swapping the grade-2 and grade-1 documents must give exactly (1 + 3/log2(3)) / (3 + 1/log2(3)). The evaluator averages each metric overall and per kind, so a regression in one slice cannot hide in the mean.

BM25, vector and hybrid on a toy corpus

I compared three retrievers behind the same search(query, k) interface:

• BM25 via rank-bm25's BM25Okapi, with a simple tokenizer that keeps identifiers like e-1004 whole and folds plurals.
• Vector search using a deterministic fake embedder, which needs an honest caveat.
• Hybrid with reciprocal rank fusion (RRF) over the top 20 from each.

The fake embedder folds a hand-written synonym table into shared concepts, then feature-hashes words and character trigrams into 256 dimensions. It is deterministic and offline, which suits tests, but it is not semantic: it "understands" exactly the synonyms I typed, and its mistakes come from hashing collisions and shared trigrams, not from meaning. In production the same embed method wraps a real embedding model, and its behaviour on your data has to be measured, not assumed from this toy.

SYNONYMS = {
    "money": "refund", "reimbursement": "refund", "refundable": "refund", "refunded": "refund",
    # ...
    "2fa": "two-factor", "authenticator": "two-factor",
}
# ... _bucket(): blake2b feature hashing to (index, sign)

class HashingEmbedder:
    """Deterministic stand-in for an embedding API: synonym folding + hashed words and trigrams."""

def __init__(self, dim: int = 256, trigram_weight: float = 0.3):
        self.dim, self.trigram_weight = dim, trigram_weight

def embed(self, text: str) -> np.ndarray:
        v = np.zeros(self.dim)
        for tok in tokenize(text):
            concept = SYNONYMS.get(tok, tok)
            i, s = _bucket("w:" + concept, self.dim)
            v[i] += s
            padded = f"#{concept}#"
            for j in range(len(padded) - 2):  # subwords blur near-identical identifiers
                i, s = _bucket("c:" + padded[j:j + 3], self.dim)
                v[i] += s * self.trigram_weight
        norm = np.linalg.norm(v)
        return v / norm if norm else v

def reciprocal_rank_fusion(rankings: list[list[str]], k_rrf: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k_rrf + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

class ExactIdentifierFirst:
    """If the query names an identifier, documents *titled* with it go first. Mentions elsewhere don't count."""
    # ...
    def search(self, query: str, k: int) -> list[str]:
        pinned = [d for ident in self.pattern.findall(query) for d in self.by_title.get(ident.upper(), [])]
        rest = [d for d in self.inner.search(query, k + len(pinned)) if d not in pinned]
        return (pinned + rest)[:k]

RRF fuses ranks, not scores, because BM25 scores and cosine similarities live on different scales and normalising them is fragile. The tie-break on document ID keeps output deterministic, so the CI gate cannot flicker.

These are the numbers I measured on the 55-document synthetic corpus and 24 golden queries, k=5. They illustrate the method and say nothing about which retriever is better on your data.

What the slices show:

• The vector paraphrase score is circular. It is perfect because I wrote synonyms for exactly those paraphrases. That demonstrates the mechanism, not the quality of dense retrieval.
• The vector identifier weakness comes from the trigrams. E-1003 and E-1006 share most of their trigrams, and in "E-1003 from the invoices endpoint" the common word "invoice" outweighs the code. Real dense models are known to struggle with rare tokens in a similar way, but that is something to check with your own golden set.
• BM25 found the error pages but did not always rank them first. Short release notes that mention a code ("Clearer message for E-1004") sat right behind the reference page. In "E-1006 on a new invoice", the words "new" and "invoice" lifted the card-update page ("the new card is charged on the next invoice date") above the error page. Real help centres have the same problem with changelogs, and it is fixed with document-type metadata or an exact lookup, not by switching to a different ranking algorithm.
• Hybrid had the best overall recall and was not best on either slice. RRF averages ranks. When one retriever is confidently right and the other confidently wrong, fusion splits the difference. On identifier MRR, hybrid sat below BM25, and on paraphrase MRR it sat below vector.

The slice results pointed to the fix. Queries that contain an error code should put the page titled with that code first. ExactIdentifierFirst does this, and identifier MRR went to 1.000. The remaining identifier recall gap comes from two grade-1 documents (the CSV export page for E-1008, the VAT page for E-1007), which is acceptable. A single average would have hidden both the problem and the fix.

Checking the answer, not just the retrieval

Good retrieval does not make the answer faithful, so I run two checks on recorded answers. The first is deterministic: the answer must cite at least one source, and every citation must point at a document that was actually retrieved. A citation to a document that was never retrieved is a fabricated source, which is worse than none.

The second is a faithfulness judge behind a Protocol, so the harness never knows whether it is talking to a model or a fake.

# ...
CITATION = re.compile(r"\[doc:([a-z0-9-]+)\]")

@dataclass(frozen=True)
class CitationCheck:
    cited: list[str]
    unknown: list[str]            # cited ids that were never in the retrieved context

@property
    def ok(self) -> bool:
        return bool(self.cited) and not self.unknown

def check_citations(answer: str, context_ids: list[str]) -> CitationCheck:
    cited = list(dict.fromkeys(CITATION.findall(answer)))
    return CitationCheck(cited, [c for c in cited if c not in context_ids])

class Verdict(BaseModel):
    faithful: bool
    unsupported_claims: list[str] = []

class FaithfulnessJudge(Protocol):
    def judge(self, question: str, answer: str, contexts: dict[str, str]) -> Verdict: ...

# ... JUDGE_PROMPT asks for {"faithful": bool, "unsupported_claims": [str]} as JSON only

class LLMFaithfulnessJudge:
    def __init__(self, model: BaseChatModel):
        self.model = model

def judge(self, question: str, answer: str, contexts: dict[str, str]) -> Verdict:
        sources = "\n".join(f"[doc:{i}] {t}" for i, t in contexts.items())
        msg = self.model.invoke([
            SystemMessage(JUDGE_PROMPT),
            HumanMessage(f"Question: {question}\n\nSources:\n{sources}\n\nAnswer:\n{answer}"),
        ])
        try:
            return Verdict.model_validate_json(msg.text)
        except (ValidationError, json.JSONDecodeError):
            # Fail closed: an unreadable verdict is not a pass.
            return Verdict(faithful=False, unsupported_claims=["judge output unparseable"])

In tests the model is GenericFakeChatModel(messages=iter([AIMessage('{"faithful": true, "unsupported_claims": []}')])) from langchain_core, which exercises the real prompt-building and parsing path. In production the wiring would be LLMFaithfulnessJudge(ChatOpenAI(model=..., temperature=0)), and that is configuration I did not execute here. One test feeds the judge "Looks fine to me!" and asserts that the verdict fails. A judge that passes whenever it cannot be parsed is not a check.

An LLM judge is itself a model with an error rate. Before trusting it in a gate, I label 30 to 50 answers by hand and measure how often the judge agrees. More detail on putting judges behind flags and shadow traffic is in shipping LLM features with eval gates.

Turning metrics into a CI gate

Metrics that live in a notebook do not stop regressions. I commit the floors next to the golden set, one per slice and metric, and a pytest test fails the build when any of them is breached.

# ...
def test_shipped_retriever_meets_every_floor(retrievers, golden, thresholds):
    results = evaluate(retrievers["shipped"], golden, k=thresholds["k"])
    failures = check_gate(results, thresholds["floors"])
    assert not failures, "retrieval regression:\n" + "\n".join(failures)

@pytest.mark.parametrize("name", ["bm25", "vector", "hybrid"])
def test_gate_rejects_weaker_configurations(retrievers, golden, thresholds, name):
    # If these ever pass, the floors are too loose to catch a real regression.
    results = evaluate(retrievers[name], golden, k=thresholds["k"])
    assert check_gate(results, thresholds["floors"])

The second test proves the gate has teeth: every weaker configuration must fail it, so loosening a floor until plain hybrid passes turns the build red. To see the failure message, I removed the identifier pinning from the shipped configuration and ran the gate:

E       AssertionError: retrieval regression:
E         all/recall@5: 0.917 < floor 0.95
E         all/mrr@10: 0.887 < floor 0.93
E         all/ndcg@5: 0.876 < floor 0.92
E         identifier/recall@5: 0.667 < floor 0.83
E         identifier/mrr@10: 0.743 < floor 0.95
FAILED tests/test_retrieval_gate.py::test_shipped_retriever_meets_every_floor
1 failed in 0.06s

The rules I keep for the floors:

• Floors sit just below the measured values. When a change improves a metric, the floor goes up in the same pull request, with the reason in the commit message.
• Small golden sets are coarse. With 24 queries, one query moves overall recall by about 4 points. On a 6-query slice it moves by nearly 17. I set slice floors knowing that, and grow the set from production misses.
• Everything is deterministic. Fake embedder, stable sorts and document-ID tie-breaks mean the gate never flakes. In a pipeline with a real embedding model, I pin the model version and cache corpus embeddings as a build artefact, so the gate measures my change and not a vendor's.
• A missing slice fails the gate (tested), so deleting the identifier queries cannot make the build pass.

If the decision is long context, the answer-level checks carry the gate. For a Postgres-backed retriever to plug in here, see RAG with Postgres and pgvector.

Checklist

• Count the permitted corpus with a real tokenizer and compare it with the budget after output, system prompt, question and a 10% margin.
• Pass prices in as parameters, and compute per-request and daily cost for long context (cached and uncached) and for RAG.
• Measure p95 time to first token at two prompt sizes, and gate on input tokens rather than guessing.
• If permissions differ per user, assume no shared prompt cache and filter by ACL inside the retrieval query.
• Send record questions (totals, statuses, counts) to a SQL or API tool, not to embeddings.
• Write a graded golden set with a kind per query before tuning, and validate labels against the live corpus.
• Report recall@k, MRR and nDCG@k per slice, not only overall.
• Compare BM25, vector and hybrid on your own golden set, and add exact lookup for identifiers when that slice is weak.
• Check that every answer cites a document that was actually retrieved, and run a faithfulness judge that fails closed.
• Commit per-slice floors and fail CI when any is breached. Add a test showing weaker configurations fail too.

Tested with Python 3.12.9, rank-bm25 0.2.2, tiktoken 0.14.0, numpy 2.5.3, langchain-core 1.6.5, pydantic 2.13.5 and pytest 9.1.1 on 2026-09-27.