How it works
Calling every step an “agent” is how these systems get sold. It is also how they get built badly. Here is what each stage does, and — more usefully — which ones deliberately don’t involve a model at all.
The whole thing is a LangGraph state machine, checkpointed to Postgres. That is what lets a run stop at the human gate and resume later — possibly in a different process, after a restart.
Green: deterministic code. Red: a person.
The model issues its own searches against the corpus, reads what comes back, and decides whether it has enough. This is the one stage where open-ended exploration earns its cost.
Why it matters: formal document language rarely matches how a question is asked. Someone asks “do you rotate credentials?”; the control says “authenticators shall be refreshed”. One embedding of one query misses that. A research loop reformulates and tries again.
incident response SLA hours
security incident response time requirements
notification within 72 hours breach reporting timeframe
service level agreement escalation timeline Four searches the model chose on its own for a single question.
Retrieval underneath is hybrid RAG: dense vectors
(pgvector) for meaning, Postgres full-text search for exact
identifiers, fused by Reciprocal Rank Fusion. Embeddings blur precise references —
a query for AC-2(3) will happily return AC-2(7),
a different control with a different requirement. Keyword search catches
that; vectors catch the paraphrase. Neither alone is sufficient.
The model is given the retrieved passages and asked for structured output: a verdict, and a list of claims where each claim carries verbatim quotes and the passage each came from.
Three verdicts are available, and the second is not a failure mode:
| Verdict | Meaning |
|---|---|
grounded | Answered; every claim carries a quote. |
unsupported | The documents don’t cover this. Said explicitly, with a reason. |
escalate | Sources conflict, or the answer needs judgement the documents don’t settle. |
Each quote is matched against the chunk it was attributed to. Not
semantically — literally. If the text isn’t there, the quote is
discarded. If a claim loses all its quotes, the claim goes. If nothing
survives, the answer becomes unsupported.
Asking a model “does this string occur in that string?” is slower, costs money, and can be wrong. Software cannot be wrong about it. Using a model here would trade a guarantee for a probability, in the exact place the product’s value lives.
The check tolerates the ragged whitespace that PDF extraction produces — a newline the model rendered as a space is forgiven — but a reworded quote is not. What gets stored is the text as it appears in the source, not as the model typed it, so the citation quotes the document rather than the model’s recollection of it.
Discarded quotes aren’t swept away. They appear in the report and in the audit record, because a system that only shows what survived is asking to be trusted rather than demonstrating it should be.
A verified quote can still be attached to a claim it does not establish. The passage says credentials rotate every 90 days; the claim says the 30-day requirement is met; the quote is entirely genuine. A substring match passes it. A reader would not.
From an actual run, on a claim whose quote verified cleanly:
“The quote is truncated mid-sentence at ‘and those that focus’ — the passage never states what the other set of outcomes focuses on, so the claim is not shown by the quoted text.” Critic finding, NIST CSF supply-chain question.
The critic annotates rather than deletes. It marks the run for escalation and records the finding; a person decides. Letting a model silently remove claims would undo the point of having a deterministic verifier in front of it.
The run pauses before anything is published. A person approves, edits, or rejects, and their identity and timestamp are recorded against the run.
The pause happens even when everything verified cleanly. A gate that only triggers on suspicion is not a sign-off — and for a questionnaire answer or a filing, the signature is the product. State is checkpointed, so a reviewer can approve hours later, from a different machine, after a restart.
Only then does a report exist: PDF for attaching to a review, Markdown for a ticket, JSON for whatever you’re integrating with. Each carries the claims, the citations, what was discarded, and who signed.
Four model calls instead of one, so roughly four times the latency and spend of naive RAG. A run takes tens of seconds.
That is the right trade when a human signs the output and an auditor may read it, and the wrong trade for a chat assistant. If your failure mode is “mildly unhelpful answer”, you do not need this. If it is “we told a customer something untrue in writing”, you do.