You have a pile of your own documents. Reports, contracts, meeting notes, intake forms, an 800 page wiki. And you wonder: can I just point an AI at this so my team can ask questions? The short answer is yes, and the technique is called RAG. The longer answer: there is more under the hood than a ChatGPT-style UI suggests, and skipping steps gets you hallucinations instead of usable answers. This article explains how an AI agent over your own data works, which components you need, what it costs and which pitfalls we have hit in SMB projects over the past 18 months. With real numbers and a case study from a client where we built exactly this.
What is RAG and why do you need it?
RAG stands for Retrieval Augmented Generation. In plain language: an AI model receives the relevant pieces of your documents alongside every question, and only then generates an answer. The model does not “know” what is in your reports on its own, it looks it up at the moment you ask. Without retrieval you have to stuff everything into the prompt, and without generation you only get chunks of text without synthesis.
Why not just ChatGPT? Three reasons. One: ChatGPT does not know your internal data. The model is trained on public text and has no idea what is in your client contracts or consultancy reports. Two: a 200,000 token context window sounds like a lot, but if you push 50,000 documents through it monthly, three search queries already put you above €100 in API costs per question. Three: ChatGPT does not cite sources. For SMBs that process legal, financial or medical information, that is not an option. If you want a clear view of the difference between standalone chatbots and real agents, read AI agents vs chatbots.
RAG solves those three problems at once. You keep your own data local, you only pay for the pieces that are actually relevant, and you can show on every answer which document the information came from. For the broader context of AI in the SMB segment, we also recommend our AI automation guide.
Which components does a RAG system have?
Under the hood of every working RAG setup there are five components. They are interconnected and you cannot drop any of them without losing quality.
- Ingestion (document intake). Your RAG system needs to know which sources belong to it. That can be PDFs, Google Docs, Notion pages, SharePoint, an intranet or a database. During ingestion you pull out the raw text plus metadata such as author, date and source URL.
- Chunking (cutting into pieces). You cannot feed long documents to an AI model whole. You cut them into pieces of 300 to 800 tokens (roughly 200 to 500 words). Chunks that are too big give imprecise retrieval, chunks that are too small lose context.
- Embedding (turning into vectors). Each chunk goes through an embedding model that converts the text into a list of numbers (a vector of, say, 1,536 dimensions). The OpenAI embeddings documentation is a solid starting point.
- Vector database (storage and search). You store the embeddings in a vector database that can find “similar” vectors very fast. We usually go with pgvector or Qdrant because they give us full control over hosting and data location.
- Retrieval and generation (assembling the answer). When a question comes in, the system fetches the top 5 to 10 most relevant chunks, pastes them in front of an LLM prompt, and lets the model formulate an answer with explicit source references.
Some teams add a reranking step between retrieval and generation: a second model that reorders the top 20 chunks by actual relevance to the specific question. That improves answer quality a lot, especially once your dataset goes above 10,000 chunks.
Which tools do we use?
An honest comparison of the tools we have used in production for Dutch and international SMB clients over the past 18 months.
| Category | Option | Strong at | We pick it for |
|---|---|---|---|
| Framework | LangChain | Biggest ecosystem, many integrations | POCs and complex agents |
| Framework | LlamaIndex | Strong in data ingestion and query engines | Document-heavy setups |
| Vector DB | Pinecone | Managed, scalable, low maintenance | Clients without an ops team, US data acceptable |
| Vector DB | Qdrant | Open source, fast, EU hosting possible | GDPR-sensitive projects, self-hosted |
| Vector DB | pgvector | Works inside existing Postgres | SMB clients already running Postgres |
| LLM | OpenAI (GPT-4o) | Best quality on most tasks | Production systems judged on answer quality |
| LLM | Anthropic (Claude Sonnet) | Strong with long context and analytical reasoning | Legal and consultancy cases |
| LLM | Mistral (EU) | EU-hosted, open-weight models | Clients with strict data location requirements |
You do not need the most expensive tool to get good results. A setup with LlamaIndex, pgvector and GPT-4o-mini runs in production for less than €60 per month at typical SMB volume. A setup with LangChain, Pinecone and GPT-4o quickly hits €250 per month. The difference in answer quality is not measurable in 8 out of 10 cases.
Concrete case: Planit Consulting
At Planit Consulting, a British consultancy, the consultants kept hitting the same problem. Searching their own archives (reports, meeting notes, deliverables) cost on average two hours per consultant per week. We built them an AI assistant on top of RAG: all deliverables through an ingestion pipeline, 600 token chunks in Qdrant on a Frankfurt server, embeddings via OpenAI, generation via Claude Sonnet. Every answer comes with direct links to the source documents. All data inside the EU. Result: two hours per consultant per week saved.
What does a RAG setup cost?
Scope sets the price, and we deliberately sit at the lower end of what the market charges for this. A POC is ready in two weeks; a full production setup for 30 staff, with the necessary attention to security, performance and monitoring, we deliver in around four weeks. That is roughly half the lead time you would spend elsewhere.
| Component | POC | Production-grade RAG |
|---|---|---|
| Ingestion (50 to 500 documents) | €1,800 | €4,500 |
| Chunking + embedding pipeline | €1,200 | €2,200 |
| Vector database setup | €1,000 (pgvector) | €2,800 (Qdrant cluster) |
| Retrieval + reranking | €1,400 | €3,300 |
| UI / chat interface | €2,200 | €4,500 |
| Eval suite and monitoring | €400 | €2,700 |
| One-time total | €8,000 | €20,000 |
| Tooling per month | €25 to €50 | €80 to €200 |
Monthly running costs are often surprisingly low. A production RAG for 30 users at typical volume (200 questions per day) costs around €120 per month all-in. Compare that to a per-seat SaaS license at €25 per user per month on comparable tools (€750 for 30 people), and it pays back within four months.
What does a simple RAG flow look like in code?
Under the hood, a first RAG flow is surprisingly compact. The Python example below shows the core in 10 lines using LangChain, a vector store and an LLM.
from langchain.vectorstores import Qdrant
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Qdrant.from_documents(docs, embeddings, location=":memory:")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa = RetrievalQA.from_chain_type(llm=llm, retriever=vectorstore.as_retriever(k=5))
answer = qa.run("Which methodology did we use at client X in 2024?")What you do not see in this example: the real complexity sits in document parsing for the first line, prompt engineering for consistent answers, and evaluation of whether retrieval actually picks the right chunks. For the broader trade-off, see no-code vs custom automation.
Which common pitfalls should you avoid?
- Chunks that are too coarse. Test at least three chunk sizes before you go live. At Planit we moved from 1,000 to 600 tokens and saw retrieval precision climb 22 percent.
- No reranking. A reranker reorders the top 20 chunks by real relevance. It costs 50 milliseconds per query and lifts answer quality significantly. Cohere Rerank and BGE Rerank are popular options.
- Accepting hallucinations. Build into your prompt: “Answer only based on the given sources. If the sources do not contain the answer, say so.” That removes 80 percent of hallucinations.
- No citations. Have the model name the source chunk after every claim, and show a clickable link to the original in the UI. Without source attribution your user cannot verify an answer.
- No eval suite. Build a set of 50 to 200 questions with known correct answers, and run it automatically after every change. Without evaluation you are flying blind.
How do you measure whether RAG actually works?
- Retrieval precision. Of the top 5 chunks the system fetches, how many are actually relevant? A good system sits above 80 percent. Below 60 percent means your chunking or embeddings are off.
- Answer accuracy. Is the answer factually correct? Above 85 percent is good, below 70 percent means something is fundamentally wrong.
- Latency. For interactive use you want to stay under 3 seconds. Above 5 seconds, users drop off. Split the latency between retrieval, reranking and generation.
On top of that, in production you track two more things: cost per question and usage patterns. For the broader setup of reporting, see our data and reporting solution.
When is RAG overkill for your use case?
- Static FAQ under 50 items. A regular chatbot or a simple search function is cheaper and faster. Read AI agents vs chatbots.
- Documents that rarely change and are rarely consulted. A good folder structure with search will do.
- Highly structured data. For questions about numbers or relationships between entities, a SQL query or a dashboard is faster and more reliable.
How do you get started concretely?
- Week 1: scope and data audit. Document 20 example questions with known correct answers. That is your first eval set.
- Week 2: ingestion + indexing. Build the pipeline that picks up documents, cleans them, chunks them and generates embeddings.
- Week 3: UI and prompt engineering. Build a minimal chat interface. Test with three to five end users.
- Week 4: eval, monitoring and launch. Grow the eval suite to 100+ questions. Add logging. Launch for an initial group.
Book a free intake call if you want us to take a look with you. Our way of working and guarantees sit in our terms and conditions. For the broader context of what we build, see custom platforms and AI assistant.
