RAG and Knowledge Systems

RAG (retrieval-augmented generation) means: find the right documents first, then ask the model.

The model is only as good as the context you give it.

The pipeline

1. 1

   Chunk

   Split documents into pieces a model can actually use.

2. 2

   Embed / index

   Make those pieces searchable.

3. 3

   Retrieve

   Pull the top candidates for a query.

4. 4

   Rank

   Optional, but usually worth it — put the best snippets first.

5. 5

   Assemble + generate

   Build a prompt with citations, then generate an answer that stays inside that context.

Where RAG actually breaks

• chunks are too big (noise) or too small (no meaning)
• the query and the docs use different words
• you retrieve 20 snippets and dump all of them in
• you never check whether retrieval was relevant before blaming the model

RAG vs fine-tuning

I reach for RAG when the facts change or they are private. I reach for fine-tuning when the shape of the answer is what is wrong, and I have a pile of examples.

Fine-tuning does not replace retrieval. A fine-tuned model still does not know the doc you wrote yesterday.

RAG use cases that fit: support answers from your help center, “what does our policy say,” search over a private wiki. If the user needed a live action (refund the order), that is an agent, not RAG.

Ways to implement it

• Using SDKs directly — embed, query, stuff the prompt, generate. I can read this path. I prefer it.
• LangChain or LlamaIndex — faster wiring if the team already uses them. They do not fix bad chunks.
• OpenAI Assistant API as a RAG alternative — file search is hosted. Useful to learn the shape. I move off it when I need to see retrieval quality in my own logs.

Chunking decides more than the generator. Split on structure (headings, sections), overlap a little, and store the source URL in metadata so the answer can cite it.

Embedding uses one model for documents and queries. The vector database stores those vectors. The retrieval process is embed the question, filter by tenant, take top-k. Generation is a prompt that says: answer only from these chunks, and say you don’t know when they are not enough.

The useful test

If a human could not answer from the retrieved snippets, the model should not be expected to either.

1. Print the retrieved chunks
2. Ask: would I answer from these alone?
3. If no → fix retrieval, not the prompt poem

Next

When retrieval is not enough and the system needs to do things:

• Agents and Tool Use