// case_study
Answering security questionnaires without leaking the answers
A private RAG agent that drafts supplier security questionnaires, grounds every answer in your own policy, and flags the ones a human needs to look at.
Two hundred questions you have already answered
And the obvious fix, pasting them into someone else's API, is the thing a security questionnaire exists to stop.
Supplier security questionnaires are a tax on selling. A single one can run to two hundred questions, most of which you have answered before, in slightly different words, in a document nobody can find.
The obvious fix is to point an LLM at it. The obvious problem with the obvious fix is that a security questionnaire is a document about your security, and pasting it, along with your policies, into someone else’s API is a strange way to demonstrate that you take data protection seriously.
So the requirement writes itself: it has to be good enough to save real time, and it has to run somewhere you control.
Ground every answer, and make the model the replaceable part
Retrieval over policy you already approved, a confidence score on every draft, and a model you can swap for one on your own hardware.
Retrieval over your own policy documents, not a general model’s memory. Every answer is drafted from something you have actually written and approved, and it cites which document it came from, so a reviewer can check it in seconds rather than trusting it.
A confidence score on every answer, and a hard rule that low-confidence answers are never submitted silently. The agent is a drafting assistant with a paper trail, not an oracle.
The model is a swappable part. Retrieval, grounding, scoring and the audit log are the system; the thing that writes the sentence is a component behind an interface. Point it at a hosted API or at an open model on your own hardware and the pipeline does not change.
pipeline --trace
What happens to a single question
Five stages, from your documents to an answer a reviewer can check in seconds.
- 1
Ingest
Your policies, previous questionnaires and controls documentation, chunked and embedded into a private vector store.
- 2
Retrieve
For each question, pull the passages that actually bear on it, tuned on your content rather than on a demo corpus.
- 3
Draft
The model answers strictly from the retrieved passages, and cites them. If the passages do not support an answer, it says so instead of inventing one.
- 4
Score
Each answer gets a confidence score based on how well the retrieved evidence actually covers the question.
- 5
Review
Anything below the bar is routed to a human, with the evidence attached. The reviewer is checking, not writing from scratch.
What it proves
The model is the easy part. Everything that makes an AI answer checkable sits around it.
- ›That the hard part of a RAG agent is not the model. It is the retrieval, the grounding, the confidence scoring and the audit trail, and none of those care which model you plug in.
- ›That grounding and citation are not a nice-to-have. They are what make an AI answer checkable, and therefore usable, in a context where being wrong has consequences.
- ›That the honest move is to build the escape hatch in from the start: the system is designed to know when it does not know.
Got a process that looks like this?
If you are pasting your own documents into someone else's API to get work done, there is usually a better way. It starts with an honest look at what you are actually spending and sending.