RAG Explained: Building an AI Assistant That Knows Your Business

A general-purpose AI model knows a lot about the world and almost nothing about your business. It does not know your return policy, your product specs, your internal pricing rules or what changed in last month's HR handbook. Ask it anyway and it will often produce a confident, plausible and wrong answer.

Retrieval-augmented generation, usually shortened to RAG, is the standard way to fix this. Instead of relying on what the model memorized during training, you look up the relevant parts of your own documents at the moment a question is asked, and give those to the model as the material to answer from. This guide explains how a RAG knowledge base assistant works, where it tends to fail, and what it takes to build one your team will actually use.

How RAG works, step by step

There are two phases: preparing your content (done ahead of time) and answering a question (done each time someone asks).

Phase 1: Preparing the content

1. Collect sources. Help center articles, product manuals, policy documents, Notion or Confluence pages, Google Drive folders, past support macros, sometimes resolved tickets.
2. Clean and split. Each document is broken into chunks, typically a few paragraphs each, ideally along natural boundaries like headings. Each chunk keeps metadata: source, title, section, date, and who is allowed to see it.
3. Create embeddings. Each chunk is turned into an embedding, a list of numbers that represents its meaning. Chunks about similar topics end up with similar numbers.
4. Store them. Embeddings and text go into a database that can search by similarity. Postgres with the pgvector extension is a common, sensible choice. Dedicated vector databases exist too.

Phase 2: Answering a question

1. Understand the question. The user's question, sometimes rewritten by a model to be clearer or to include context from earlier in the conversation, is turned into an embedding.
2. Retrieve. The system finds the chunks most similar in meaning. Good systems combine this with keyword search, so exact terms like part numbers and policy names are not missed.
3. Rerank and filter. The candidates are sorted by relevance and filtered by permissions, so a customer never sees an internal-only document.
4. Generate. The top chunks are passed to a model like Claude or GPT with instructions such as "Answer only from the sources below. If they do not contain the answer, say so. Cite the source for each claim."
5. Return with citations. The answer comes back with links to the documents it used, so people can check.

That is the whole idea. The model does the reading and writing, but the knowledge comes from your content, which you control and can update without retraining anything.

Why not just fine-tune a model on our documents?

This comes up often. Fine-tuning adjusts a model's behavior using examples. It is good for teaching style, format or a narrow task. It is a poor way to teach facts that change. If your shipping policy changes, a RAG system picks up the new document the next time it is indexed. A fine-tuned model would need retraining, and you would have no easy way to confirm it forgot the old policy.

RAG also gives you citations and access control, which fine-tuning does not. For a knowledge assistant, RAG is almost always the right starting point.

Where RAG assistants go wrong

A demo RAG assistant can be built in an afternoon. A reliable one takes more care. Most problems fall into a few buckets.

1. The content is the problem

If your help center has three conflicting articles about warranty periods, the assistant will find them and may quote the wrong one. RAG makes outdated or contradictory content visible very quickly. Budget time for a content cleanup: archive old documents, merge duplicates, and add dates and owners to the ones that remain.

2. Bad chunking

Split documents too small and chunks lose context ("This does not apply to them" means nothing on its own). Split too large and retrieval gets vague. Tables, FAQs and step-by-step procedures each need slightly different handling. This is unglamorous work, but it makes a big difference to answer quality.

3. Retrieval misses the right chunk

Pure meaning-based search can miss exact identifiers like SKUs, error codes or contract clause numbers. Hybrid search (meaning plus keywords) and a reranking step usually fix this. When an answer is wrong, check retrieval first: was the right chunk even found?

4. The model answers beyond the sources

Even with good retrieval, models sometimes fill gaps with general knowledge. Clear instructions, requiring citations, and testing with questions that the documents do not answer all help. The assistant should be comfortable saying "I could not find that in our documentation" and offering a handoff.

5. Permissions are an afterthought

If the HR policy folder and the public help center go into one index without access metadata, someone will eventually get an answer they should not see. Decide early who can see what, store that with every chunk, and filter at retrieval time based on the user asking.

A worked example: an internal support assistant

Consider a hypothetical manufacturer with a technical support team of eight. Engineers answer distributor questions about installation, compatibility and error codes. The answers live in a mix of PDF manuals, a Confluence space, and the heads of two senior engineers.

A practical build might look like this:

• Sources: current PDF manuals, the Confluence troubleshooting space, and a curated set of a few hundred resolved Zendesk tickets that the senior engineers have reviewed and tagged as good answers.
• Audience: internal support staff first, not customers. That keeps the risk low while the team learns where it is weak.
• Interface: a Slack bot and a sidebar in Zendesk that drafts a suggested reply with citations for every new ticket.
• Feedback: thumbs up or down on every answer, with a reason. Weekly review of the thumbs-down list to fix content or retrieval.
• Later: once draft acceptance is consistently high, expose a narrower version to distributors in their portal, limited to public manuals.

This "internal first, external later" path is one we recommend often. Your team catches the mistakes before customers do, and the knowledge gaps it surfaces are useful in their own right.

When RAG becomes an agent

A pure RAG assistant reads documents. Many real questions also need live data: "Is this part in stock?", "What is the status of my claim?", "When does my contract renew?" Answering those means connecting the assistant to your systems through tools, which turns it into a simple agent. The same principles apply: limit what it can access, log what it does, and make the human handoff easy. Our customer support agent work usually combines both: RAG for policies and how-to questions, tool calls for order and account data.

Choosing models and infrastructure

A few practical notes:

• Models. Claude, GPT and other strong models all work well for RAG. Pick based on answer quality on your own test questions, cost at your volume, and your data handling requirements. Design the system so the model can be swapped later.
• Embeddings. Use a good general embedding model unless you have a specialized domain with lots of jargon. If you switch embedding models later, you must re-index everything.
• Storage. If you already run Postgres, pgvector keeps things simple. Your vector data then lives next to your permissions and metadata.
• Sync. Decide how content stays fresh. Nightly re-indexing is fine for manuals. Help center articles may need updates within minutes of publishing.
• Logging. Store each question, the retrieved chunks, the answer and the feedback. Without this you cannot improve anything.

If you want this inside an existing product or internal tool rather than as a standalone bot, that falls under LLM integration: the same RAG pipeline, exposed through your own UI and APIs.

How to evaluate a RAG assistant

Do not judge it by trying ten questions in a demo. Build a test set:

1. Collect 50 to 150 real questions from tickets, Slack threads and sales calls.
2. For each, write down the correct answer and which document contains it.
3. Include questions the documents cannot answer, to test that the assistant says so.
4. Include questions that should be blocked by permissions.
5. Run the set after every significant change and track: did retrieval find the right document, was the answer correct, was it properly cited.

This test set is the most valuable thing you will build. It turns "it feels better" into something you can measure, and it lets you safely try a new model or chunking approach later.

A launch checklist

• Sources are identified, owned and cleaned of obvious duplicates and outdated pages
• Access rules are defined and stored with each chunk
• The assistant cites sources and says when it does not know
• There is a clear handoff to a human, with the conversation attached
• A test set exists and passes at a level the team is comfortable with
• Feedback is captured and someone reviews it weekly
• Content updates flow into the index on a known schedule

Getting started

The best first RAG project is usually narrow: one team, one well-defined body of content, and a group of internal users who will tell you honestly when it is wrong. If you would like help scoping a knowledge base assistant for your support team or another function, have a look at our process or start a conversation about what you have in mind.