RAG: AI that answers from your documents, not from memory

The most common request we hear about artificial intelligence goes roughly like this: "we want the model to know our data." Behind that sentence there is almost always a wrong picture of how AI works — as if there were a button that "loads" the company into the model, and from then on it knows everything. It does not work that way, and the good news is that you do not need it to.

What you need is called RAG — retrieval-augmented generation. The idea is simple: when a user asks a question, the system first finds the relevant pieces of your documents, and only then lets the language model compose an answer — but strictly from that retrieved material, citing where each claim came from. The model does not memorise your data; it reads it at the moment of the question.

What RAG is, without the mystique

Picture an experienced colleague you hand a question and a shelf of manuals. They do not answer from memory — they open the right manual, find the right page, read it, and then give you the answer while pointing at where it came from. RAG does exactly that, only at machine speed and across thousands of documents at once.

The key consequence is that the answer can be checked. When the system cites its source alongside the answer — which document, which section — a human can see within seconds whether the claim holds. That is the difference between an AI you trust on its word and an AI that shows you the evidence. For anything serious in an enterprise, the second kind is the only usable one.

Why fine-tuning is not the answer

When they hear "the model should know our data", many people think of fine-tuning — further training the model on their own documents. That is almost always the wrong tool for the goal, for several reasons worth separating.

Fine-tuning changes a model's style and behaviour, not its knowledge of facts. If you train the model on your documents, it will learn how you sound, but it will not reliably remember that item X costs exactly what it costs — and those are precisely the facts you need. Worse, when the price changes you would have to retrain, whereas RAG simply reads the new value from the source.

On top of that, fine-tuning knows nothing about permissions. Once a fact is baked into the model's weights, it is there for everyone — you cannot say the director may see salaries but the warehouse clerk may not. RAG keeps the data where it already lives, with its existing access rights, and retrieves it by the rules. Fine-tuning has its place when you want a particular tone or output format, or a specialised task — but "so the model knows our facts" is not that case.

The pipeline: from document to answer

RAG is not one model but a chain of steps, and the quality of the whole system is set by its weakest link, not by the model at the end. Here is what happens between a document and an answer.

Most people think RAG quality is chosen by picking the model in the last row. It is not chosen there. It is chosen in the middle of the table — in how the documents are chunked and labelled.

Chunking and metadata decide quality

This is the part most often underestimated. If you cut a document in the wrong places — so that a price table is split in half, or the sentence giving context lands in one piece and the figure in another — retrieval will fetch pieces that make no sense on their own. The model then receives fragments without context and answers poorly, however good it is.

Metadata — the labels on each piece, such as source, date, department or document version — matter just as much. Without them the system cannot tell the current version of a procedure from last year's, nor narrow the search to the right area. A good RAG spends more effort on how documents are prepared than on which model is picked, and that is the reverse of what most people expect.

Permissions are part of retrieval, not an afterthought

This is a line that must not be crossed: a user must never retrieve a document they are not allowed to open. If a RAG system searches every document regardless of access rights, it becomes a perfect data-leak machine — someone with no access to payroll asks a question, and the model politely composes an answer from exactly that payroll.

That is why permissions must be part of the retrieval itself, not a filter stuck on the end. In practice it means access rights are checked at search time: the system retrieves only from the set of documents this particular user is allowed to see, and only then generates. The same document yields different answers to different people, exactly as it would in a paper world. This is also why RAG is built on top of systems that already know who may see what — document management through DocDot relies on existing access rights rather than reinventing them.

How you measure RAG

A RAG system fails in two distinct ways, and conflating them is the most common reason quality does not improve. You have to measure them separately.

• Retrieval recall (retrieval recall) — whether retrieval found the piece of a document that actually holds the answer. If the answer was never retrieved, the model has no chance of being right.
• Answer faithfulness (faithfulness) — whether the generated answer stays within what was retrieved, or the model added something of its own. An answer can be faithful yet useless if retrieval missed; and correct material can still produce an unfaithful claim if the model wanders.

These two numbers are fixed by different means. Low recall is treated with better chunking, metadata and reranking. Low faithfulness is treated with a stricter instruction to the model and a cited source for every claim. If you measure only "is the user happy with the answer", you do not know which of the two levers to pull.

The order in which RAG is brought into a usable state:

1. Assemble a set of questions and correct answers from real queries, with a known source for each.
2. Measure recall first. For each question, check whether the right piece of the document was retrieved at all. Without that, the rest is meaningless.
3. Then measure faithfulness. Whether the answer stays within the retrieved material and cites a source that genuinely supports the claim.
4. Fix the pipeline, not the model. Chunking, metadata and reranking move the needle more than swapping the model.
5. Measure again after every change to the source. New documentation can break what worked yesterday.

Index freshness, and when RAG is not the answer

RAG is only as good as its index is fresh. If a procedure changes but the old version stays in the search, the system will confidently cite an outdated rule. That is why refreshing the index is not a one-off job but an ongoing process: when a document changes, its pieces are re-ingested and re-labelled, and the old ones withdrawn. A system that is filled once and then forgotten slowly starts to lie — not because the model is bad, but because it is reading yesterday's world.

And finally, the most important boundary: RAG is not the answer to every question. When the data is structured — stock levels, an invoice amount, the number of open complaints — do not search it as text; query it as a database. "How many units of item X are in stock" is not a job for meaning-based search but a plain database query, and trying to solve it through RAG gives a slower and less reliable result than a single row from the database.

In practice the best systems combine the two: RAG for questions over documents and policies, a direct query for numbers, and a clear rule about which question goes where. So NG Sara answers a question about delivery terms from the documents, and a question about the concrete status of an order from the database of the NG Commerce platform — every question from its own source. If you are not sure which questions in your case call for RAG and which call for the database, that is the first conversation worth having.