Adding an AI Assistant to Your Product: A Practical Guide to RAG

Many businesses want an assistant that can answer questions about their own content: product documentation, policies, contracts, support history. A general chat model does not know any of that, and fine-tuning one is slow and costly. The approach most teams use instead is retrieval-augmented generation, usually shortened to RAG.

The idea in one paragraph

Instead of hoping the model remembers your content, you look up the relevant passages at the moment someone asks, and hand them to the model together with the question. The model then writes an answer based on those passages. The knowledge lives in your own search index, where you can update it, restrict it and audit it.

The architecture

flowchart TD
    subgraph Ingest["Ingestion pipeline"]
        A["Documents"] --> B["Chunk"]
        B --> C["Embed"]
        C --> D["Vector store"]
    end

    subgraph Query["Query pipeline"]
        E["Question"] --> F["Embed"]
        F --> G["Search"]
        D --> G
        G --> H["Top chunks"]
        H --> I["LLM"]
        E --> I
        I --> J["Answer with sources"]
        J --> K["Guardrails check"]
    end

There are two separate flows.

Two flows: index the documents once, answer questions live.

Ingestion (offline)

1. Collect your documents.
2. Split them into chunks of a few hundred words.
3. Turn each chunk into an embedding, a list of numbers that captures its meaning.
4. Store the embeddings and the original text in a vector store.

Query (live)

1. Embed the user's question the same way.
2. Search the store for the most similar chunks.
3. Send the top chunks and the question to the language model.
4. Check the answer, then return it with sources.

Details that decide quality

Most disappointing RAG projects are not model problems. They are retrieval problems.

Most bad RAG is bad retrieval, not a bad model.

Chunking

Chunks that are too small lose context. Chunks that are too large dilute the match. Split on natural boundaries such as headings and paragraphs, keep a little overlap between neighbours, and store the title and section with each chunk.

Retrieval

Pure vector search misses exact terms such as product codes and names. Hybrid search, combining keyword and vector matching, usually works better. A re-ranking step on the top results often improves accuracy further.

The prompt

Tell the model plainly what to do:

Answer using only the context below.
If the context does not contain the answer, say you don't know.
Cite the source title for each claim.

Context:
{retrieved_chunks}

Question: {question}

An assistant that says "I don't know" is far better than one that sounds confident and is wrong.

A minimal query function

def answer(question, user):
    q_vec = embed(question)
    hits = vector_store.search(q_vec, top_k=6, filter={"allowed_roles": user.roles})
    context = "\n\n".join(f"[{h.title}] {h.text}" for h in hits)
    reply = llm.generate(prompt(context, question))
    return {"answer": reply, "sources": [h.url for h in hits]}

Note the filter. It leads to the next point.

Permissions and privacy

An assistant that can read everything will happily repeat it to anyone who asks. Enforce access control at retrieval time: store the permissions with each chunk and filter by the current user before anything reaches the model. Do not rely on the prompt to keep secrets.

Filter by permission before the model sees anything.

Also decide up front:

• Where is the data processed and stored, and does that meet your legal and customer requirements?
• Does the provider you use keep or train on your data? Check the terms and choose the setting that does not.
• Should personal data be masked before it is sent?
• What do you log, and for how long?

Guardrails

Add checks around the model, not just inside the prompt:

• Input checks for prompt injection, where text in a document or a user message tries to override your instructions.
• Output checks that the answer is grounded in the retrieved text and does not leak restricted data.
• Scope limits so the assistant refuses topics it should not handle.
• A human fallback, such as "talk to a person", when confidence is low.

Measure it

You cannot improve what you do not measure. Build a small evaluation set of 50 to 100 real questions with known good answers, and run it every time you change chunking, retrieval or prompts. Track:

• Did the right passage appear in the top results?
• Was the answer correct and supported by the sources?
• How often did it correctly say it did not know?
• Latency and cost per question.

User feedback buttons, a thumbs up or down on every answer, give you a steady stream of real cases to learn from.

Common pitfalls

1. Indexing everything. Out-of-date and duplicate documents produce confusing answers. Clean first.
2. No refresh process. Content changes, and the index must follow.
3. Skipping evaluation. Without a test set, every change is a guess.
4. Ignoring cost. Long contexts and many calls add up. Cache common answers and limit context size.
5. Treating it as a finished product. It needs ongoing maintenance, like search does.

When RAG is not the answer

If you need the assistant to do things, such as booking, refunds or updating records, you need tool use with strict permissions and confirmations, not just retrieval. If your data is structured, a direct database query may beat a language model. Choose the simplest tool that solves the problem.

Conclusion

RAG is a practical way to put your own knowledge behind an AI assistant without retraining a model. The model is the easy part. The work is in clean content, good retrieval, access control and measurement. Start with one well-defined use case, such as support or internal documentation, measure it honestly, and expand once it is reliably useful.