# Case study: ask the file

Grounded assistant over this site's own corpus: the assistant running on the homepage. Five checked stages (gate, retrieve, scope, route, verify) streamed live over SSE; answers cite the file they came from or the system abstains. Python + FastAPI internals on a Cloudflare Worker.

**Live:** the ask panel on https://luisosio.dev · **Health:** https://luisosio.dev/v1/ask/health. Source stays private. Every run is inspectable through the public trace endpoint instead.

## Problem

Portfolio claims are unverifiable prose. A visitor should be able to interrogate the evidence, ask what was built, where, with what, and see the answer justified, not take the author's word for it. A demo that can't answer is decoration.

## Five checked stages

Every run emits per-stage telemetry over SSE, and the UI renders the same data the pipeline produced; nothing is staged for the demo.

| Stage          | What it does                                                                                          |
| -------------- | ------------------------------------------------------------------------------------------------------ |
| 01 gate        | One judge call scores intent (9 lanes), injection, and extra-task load before anything is retrieved    |
| 02 sources     | Dense cosine + BM25 fused by RRF; pinned canonical chunks guarantee the evidence window covers the ask |
| 03 scope       | Post-retrieval judge re-decides in_scope with the evidence in hand; a strong match counts as on-topic |
| 04 writer      | Routed model answers under a contract: claims plus cited chunk ids, nothing else                       |
| 05 check       | Per-claim groundedness verified against each claim's cited sources; numbers must appear in the source  |

## The gate

One System One call (vllm-sr/d3-flash) scores six criteria before anything is retrieved: intent across nine lanes, prompt injection, extra-task load, write-task asks, contact kind, and language. Scope is decided separately, after retrieval, by a second judge with the evidence in hand — a strong match counts as on-topic.

## Retrieval

Hybrid search over the corpus: BM25 keyword ranking fused with multilingual dense vectors through reciprocal rank fusion (k=60), plus an intent-family vote that keeps the evidence window on-topic. Chunks declare their own canonical intents; a pin inserts the right file only when similarity can't reach it — how "does he know rust?" still finds the stack list it never mentions. Closed-list enumerations (stack, credentials, languages) are appended past the window on every ask, flagged `appended` so context stays distinguishable from rank-earned evidence. The window starts at top 6 and widens to 12 when retrieval is thin.

## Routing

Answers are written through vLLM Semantic Router (https://vllm-sr.ai, https://github.com/vllm-project/semantic-router): vllm-sr/auto picks the model and reasoning effort per question. The routing decision streams in the trace, including when a failed model triggers a retry.

## Verification

The writer's output is a contract — claims plus the chunk ids they cite — checked before answer text renders. Every claim must be cited; every number must appear verbatim in a cited source (or echo the question itself — the asker's own data owes no source). Groundedness is scored per claim against each cited source and their union, so a padded citation can't dilute a true claim and a compound claim can draw on two files — though a claim citing several sources is first split into single-fact atoms, and the weakest atom sets the score and names itself to the repair pass. Absence on closed lists (stack, languages, credentials) is a deterministic string check. A failed check gets one repair pass, then the run abstains.

## Built with

Python + FastAPI inside a Cloudflare Python Worker: rank_bm25 for lexical search, Workers AI bge-m3 embeddings in production (sentence-transformers MiniLM locally), vLLM Semantic Router for the gate, the scope judge, and routed generation. Eval sets: 41 smoke + 60 held-out, replayed through the public trace endpoint — plus a 30-question persona-parity set replayed against a swapped identity (a María Vélez dossier) to prove no claim leaks the real name.

## Decisions and what they beat

- **Fixed stages, not an agent loop.** An agent lets the model pick its next action inside a loop; here the control flow is code and models only write or score at fixed points. Stage boundaries are where checks live — a loop has no checkpoints — worst-case cost is a fixed call count, and refusal is a threshold, not a mood the model can talk itself out of.
- **No tools on purpose.** "Answers only from the file" is only enforceable if every fact carries a chunk id — tool output has no id and would bypass the verifier entirely. A closed corpus also makes absence decidable: the file is enumerable, so "not covered" means absent — with a fetch tool it would only mean "not fetched yet".
- **Deterministic checks before the judge.** Citations are a string match, numbers a digit audit — exact, free, binary. Groundedness needs entailment, the one judgment only a model can make, so it runs last and scores claims it never wrote. Each layer exists because the other two can't do its job.
- **Verify before stream, not after.** The writer's output is a contract of claims and the chunk ids they cite, checked claim-by-claim before answer text reaches the visitor. Unverified text is never rendered: a failed check triggers one repair pass, then abstains.
- **Pins over hoping.** "Does he know rust?" never lexically matches the stack list; `me.stack` can't win a similarity contest against a word it doesn't contain. Intent pins force canonical chunks into the evidence window with their honest scores, so the writer always sees the source that can actually answer.
- **Deterministic absence on closed lists.** "Is rust in his stack?" is verifiable by string match, not by judge opinion: closed-list chunks (stack, languages, credentials) carry a flag and the `absent` field on a claim is checked mechanically against them.
- **Two gates, not one.** The first gate classifies pre-retrieval; a second judge re-decides scope *after* seeing what retrieval surfaced. Questions about the mechanism itself that never name the person still pass, because a strong evidence match is itself scope evidence.
- **Abstain is a feature.** Six named refusal reasons (out_of_scope, restricted, injection, not_in_corpus, unverified, absence) each produce a specific answer, not a generic error. The break-it suggestion lane exists to exercise them publicly.

## Numbers

- **5** checked stages, each timed and emitted live; cell totals partition the backend wall clock
- **9** intent lanes routed by the first gate
- **101** eval items (41 smoke + 60 held-out), run as controlled experiments, not tuning loops

## What I would do differently

- Dev and prod run different embedding models (local MiniLM vs Workers AI bge-m3), so scores and occasional rankings diverge. Parity or a shared embed binding closes it.
- The writer contract should have been first, not second: prose-first drafts hid failures the claims+citations contract made legible.
- Timing accounting was retrofitted: internal repair calls initially billed to nothing. Bill per call from the start; aggregates hide work.
