AI agents with your own data: how RAG works

An AI agent using your own data can retrieve relevant passages from your documents and use them to formulate an answer. This approach is called RAG. It doesn’t require retraining the language model, but you do need to decide which sources it may use and how to check its answers. Our guide to AI automation covers the wider approach; our explanation of agents and chatbots discusses when actions in other systems enter the picture.

How does RAG work with your own documents?

RAG stands for retrieval augmented generation. The application finds information relevant to a question and supplies it to the language model, which then formulates a response. IBM describes this connection to external knowledge, without needing to train the model again whenever documents change.

Consider a hypothetical installation business. An employee asks which maintenance agreement applies to a particular customer. A useful application retrieves the valid contract and any amendments, shows the passages used and identifies missing information. A general maintenance manual isn’t enough.

That’s also the difference from a search box alone: the answer can bring together information from several passages. You need to test whether that extra step adds value. Sometimes an employee can already find the right document quickly with a good search function, and a generated summary contributes little.

What needs to be built or configured?

Discuss the complete route from source file to answer. An impressive chat window tells you little about what happens behind it.

Component	What happens?	What should you be able to check as the client?
Connect sources	The application retrieves permitted files and their attributes	Which folders, versions and permissions are included?
Process content	Text, tables and other content are made readable	Do headings, conditions and references stay together?
Populate a search index	Content is stored in a searchable form	How are withdrawn or replaced documents removed?
Retrieve information	A question leads to a selection of passages	Do they match the customer, topic and user?
Generate an answer	The model uses the selected information	Do the source references really support the answer?

Ask for a demonstration using a difficult file from your work. It might be a scanned attachment containing an exception. If that exception disappears during ingestion, the eventual answer can be well written yet describe the wrong agreement.

Also establish what happens when a file cannot be processed. Silently skipping it is a poor outcome: an administrator should see that a source is missing. Users don’t need to read a technical log, but they do need to understand that the answer might not cover all relevant material.

Does everything need a vector database?

Semantic search often uses embeddings: numbers representing text, allowing similar passages to be found. But exact search remains useful. A product code, case number or name may need a literal match. Microsoft explains how hybrid search combines text and vector results.

Choose the search method around questions your team actually asks. In the installation example, the application needs both the correct customer number and the relevant contract clause. A passage about maintenance for a different customer isn’t a useful result.

Long documents are often split into passages, known as chunks. There’s no universally correct length. Microsoft describes different ways of splitting content, including approaches based on document structure and meaning. A short passage can omit a condition; an entire chapter can contain too much unrelated information.

So don’t ask only which database brand a supplier chooses. Ask how they’ll test whether the right information is retrieved from your document types. An additional step that ranks results again may be worth investigating when relevant passages appear too far down the results. It isn’t a compulsory addition that automatically produces better answers.

How do you prevent staff from seeing the wrong documents?

Check permissions before retrieved information reaches the language model. An employee without access to a customer file shouldn’t receive a summary of it either. Microsoft describes document permissions and security filters that exclude unauthorised search results.

Turn this into an acceptance test. Ask the same question through two accounts with different permissions. Then revoke access and check when the change takes effect. Examine stored conversations and source links as well: access control must cover more than the initial search.

The term RAG says nothing by itself about where your data is processed. Source documents, the search index, model requests, conversation history and logs can be handled in different places. Request a map of that route and the arrangements for each service. “The database is in Europe” doesn’t explain where a passage goes when an answer is generated.

Hosting the system yourself also requires maintenance. Agree who revokes access, installs updates and tests recovery. Don’t choose a deployment model on its label alone; assess whether your organisation can carry out the agreed work.

What happens with old sources and conflicting information?

Retrieval is only part of the job. An old agreement can closely match a question, while a later amendment contains the relevant exception. Keep identifiable attributes such as customer, document type, version and validity status with the source.

Agree which information takes precedence with the person responsible for the content. A model shouldn’t guess which of two conflicting agreements applies. In the installation example, the application should flag a missing amendment or unclear effective date so an employee can investigate.

Include new and withdrawn documents in the trial. Replace an agreement, delete a file and repeat the question. Check the answer and its source link. That tests the maintenance process, rather than only a demonstration using an unchanging folder.

Also distinguish source content from instructions to the application. OWASP describes indirect prompt injection: retrieved content can contain instructions that try to change a model’s behaviour. A sentence in a document mustn’t grant permission to skip checks or send information to someone.

How do you measure whether answers are useful?

Assess retrieval and the final answer separately. This follows the step-by-step evaluation approach described by Microsoft. Otherwise, when an answer is wrong, you won’t know whether the wrong source was found or the right passage was misinterpreted.

Put together a test collection with someone who knows the work. Include ordinary questions, but also questions with no available answer, documents containing exceptions and situations with restricted access. Record which sources each question requires and why the expected answer is correct.

Then assess more than the wording:

• Was the correct source found, including the condition that changes the answer?
• Does the answer say anything the source doesn’t support?
• Can an employee open the reference and locate the passage?
• How much checking and correction remains before the result is usable?

There isn’t a universal success percentage that makes every application suitable for use. Agree thresholds for each task and type of error. An awkwardly worded summary carries different consequences from a misstated contract condition. Repeat relevant tests when sources, search settings or the model change.

What does an existing RAG case show?

Our Planit Consulting case study describes a search and chat application for an archive of planning documents. The recorded implementation includes document classification, passages with metadata, ChromaDB and answers through Claude with references to the original files.

The case illustrates why document preparation is a separate part of the project. It doesn’t prove that the same configuration will suit your files. Compare the kinds of documents and questions, rather than simply noting that both projects have a chat window.

Which costs should a proposal include?

Ask the supplier to separate initial setup from recurring usage and maintenance. Document processing, integrations, access control, user screens and testing involve different work from paying for model requests. These choices determine the scope; a fixed price for “a RAG system” tells you little without them.

Include changes too. What does it take to connect a new source, process a document type differently or switch models? Who investigates errors and maintains the test collection? Our AI agent cost guide helps you discuss those items in a proposal.

Compare the trial with the current method, including checking time. Faster retrieval is useful only if the employee can work with the answer. Sometimes improving document organisation is the first investment, with RAG coming later.

How do you actually get started?

Choose a recurring information question and collect the permitted sources needed to answer it. Describe a good response, an incomplete response and a situation where the system should report that it doesn’t know. Assign someone who can assess the content.

Then run a limited trial with recognisable source references and access control. Before extending its use, agree how sources are updated and who investigates errors. If you also want actions in other systems, describe those separately when commissioning AI agent development. A good search answer doesn’t grant permission to change a contract or order.

Book a call and bring an anonymised example. TopDevs can help identify the components you need and what you should establish first.