Adding RAG to an AI Agent
In the previous articles, we gradually improved our AI Agent.
Our Agent can now:
- Understand user requests
- Use an AI model
- Use tools
- Perform calculations
- Maintain conversation context
- Use memory
However, there is another important problem.
An AI model does not automatically know the private information contained in our documents, company databases, manuals, or internal knowledge.
For example, imagine that we have a company document containing:
Company Software Development Guide
The company uses C# and ASP.NET Core.
All applications must use SQL Server.
Unit testing is performed with xUnit.
If we ask our Agent:
What database does our company use?
The Agent cannot automatically know the answer simply because the document exists on our computer.
We need a way to give the Agent access to this information.
This is where RAG becomes useful.
What Is RAG?
RAG stands for:
Retrieval-Augmented Generation
RAG is a technique that allows an AI application to retrieve relevant information from an external knowledge source and provide that information to the AI model when generating an answer.
In simple terms:
RAG allows an AI Agent to search relevant information from external knowledge and use it to answer the user’s question.
A simplified architecture looks like this:
User
|
v
AI Agent
|
v
Search Knowledge
|
v
Retrieve Relevant Information
|
v
LLM
|
v
Answer
Why Do We Need RAG?
Large Language Models are trained using large amounts of information, but they do not automatically have access to every private or newly created document.
For example, an organization might have:
- Internal documentation
- Product manuals
- Employee guides
- Technical documentation
- Customer information
- Company policies
- Project documentation
- Databases
- PDF documents
We may want our Agent to use this information.
Instead of putting all the information directly into every prompt, we can use RAG.
A Simple Example
Imagine that we have a document called:
company-development-guide.txt
The document contains:
The company uses C# for backend development.
ASP.NET Core is the standard framework for web applications.
SQL Server is the standard relational database.
All new applications should include automated unit tests.
Now the user asks:
What framework does our company use for web applications?
The Agent can retrieve the relevant part of the document:
ASP.NET Core is the standard framework for web applications.
The retrieved information is then provided to the AI model.
The model can answer:
The company uses ASP.NET Core as its standard framework for web applications.
RAG Architecture
A typical RAG system contains several stages:
Documents
|
v
Split Documents
|
v
Create Embeddings
|
v
Vector Database
|
|
User Question
|
v
Create Query Embedding
|
v
Search Similar Information
|
v
Retrieve Relevant Text
|
v
LLM
|
v
Final Answer
This may look complicated at first.
Let’s break it down step by step.
Step 1 – Collect the Knowledge
First, we need information that the Agent should be able to use.
For example:
documents/
│
├── company-guide.txt
├── development-guide.txt
└── security-guide.txt
These documents become our knowledge source.
Step 2 – Split the Documents
Large documents are usually divided into smaller pieces called chunks.
For example:
Large Document
|
+---- Chunk 1
|
+---- Chunk 2
|
+---- Chunk 3
|
+---- Chunk 4
Why?
Because when the user asks a question, we usually don’t want to send the entire document to the AI model.
We want to find the parts that are relevant to the question.
Step 3 – Create Embeddings
The text chunks can be converted into numerical representations called embeddings.
An embedding represents the semantic meaning of text as a vector.
For example:
"ASP.NET Core is used for web development."
|
v
Embedding Model
|
v
[0.12, -0.43, 0.81, ...]
The exact numbers are not important for our understanding.
The important idea is that semantically similar text can be represented in a way that allows a system to search for related information.
Step 4 – Store the Embeddings
The embeddings can be stored in a vector database or another system that supports semantic search.
For example:
Vector Database
--------------------------------
Chunk 1 → Vector
Chunk 2 → Vector
Chunk 3 → Vector
Chunk 4 → Vector
Popular technologies used for vector search include:
- PostgreSQL with vector extensions
- Elasticsearch
- Pinecone
- Weaviate
- Chroma
- Other vector databases and search systems
The choice depends on the application.
Step 5 – Ask a Question
Now the user asks:
What framework does the company use for web applications?
The question is also converted into an embedding.
User Question
|
v
Embedding
|
v
Vector Search
The system searches for chunks that are semantically related to the question.
Step 6 – Retrieve Relevant Information
The search may return:
Chunk 2:
ASP.NET Core is the standard framework
for web applications.
This is the information we need.
Step 7 – Give the Information to the LLM
The retrieved information is then provided to the AI model together with the user’s question.
Conceptually:
User Question
+
Retrieved Information
|
v
LLM
|
v
Final Answer
The model can now generate an answer based on the retrieved information.
The Complete RAG Process
The complete process can therefore be summarized as:
Documents
|
v
Split into Chunks
|
v
Embeddings
|
v
Vector Storage
|
|
User Question ------+
|
v
Semantic Search
|
v
Relevant Information
|
v
LLM
|
v
Answer
RAG and AI Agents
Now we can add RAG to our Agent architecture.
Previously we had:
AI Agent
|
+-- LLM
|
+-- Tools
|
+-- Memory
Now we can add a knowledge retrieval capability:
AI Agent
|
+-- LLM
|
+-- Tools
|
+-- Memory
|
+-- RAG / Knowledge Retrieval
The Agent can now use:
- The AI model for reasoning and generating responses
- Tools for performing actions
- Memory for maintaining useful information
- RAG for retrieving external knowledge
RAG vs Memory
It is important to understand the difference.
Memory is generally concerned with information about the Agent, user, session, or previous interactions.
For example:
User prefers C#.
RAG is primarily concerned with retrieving information from an external knowledge source.
For example:
Company documentation says that
ASP.NET Core is the standard framework.
A simple comparison is:
Memory
|
+-- What do we remember about the user?
+-- What happened previously?
+-- What information should persist?
RAG
|
+-- What information exists in our documents?
+-- Which information is relevant to this question?
+-- What knowledge should we retrieve?
RAG vs Training the AI Model
Another common misunderstanding is that RAG means training the AI model with our documents.
It does not.
With RAG:
Your Documents
|
v
Knowledge Retrieval
|
v
Relevant Information
|
v
AI Model
The model itself does not need to be retrained every time a document changes.
This makes RAG useful for applications where information changes regularly.
Example: Company Knowledge Agent
Imagine a company has thousands of technical documents.
We could create an Agent that answers questions about those documents.
For example:
What database technology does our company use?
The Agent retrieves the relevant information.
Or:
What is the procedure for deploying an application?
The Agent searches the company documentation and retrieves the relevant deployment instructions.
The architecture might look like:
User
|
v
AI Agent
|
+------------+------------+
| | |
v v v
LLM Memory Tools
|
|
v
RAG Search
|
v
Company Documents
This is a practical example of an enterprise AI Agent.
Building a Simple RAG Example
For our tutorial, we can create a small knowledge base.
For example:
knowledge/
│
├── company.txt
└── development.txt
company.txt:
Our company develops software applications.
The main backend programming language is C#.
The company uses ASP.NET Core for web applications.
development.txt:
All new applications should use automated testing.
The standard testing framework is xUnit.
SQL Server is the standard relational database.
Our Agent can then answer questions based on these documents.
Example Question
User:
What programming language does the company use?
The RAG system searches the knowledge base.
It finds:
The main backend programming language is C#.
The Agent then responds:
The company uses C# as its main backend programming language.
Another Example
User:
What testing framework should I use?
The system retrieves:
The standard testing framework is xUnit.
The Agent responds:
The standard testing framework is xUnit.
What Happens If the Information Is Not Found?
This is an important part of a good RAG system.
Suppose the user asks:
What cloud provider does the company use?
But our documents do not contain this information.
The Agent should not simply invent an answer.
It should respond something like:
I could not find information about the company’s cloud provider in the available documentation.
This is one of the advantages of designing the Agent to use a controlled knowledge source.
RAG Does Not Automatically Eliminate Hallucinations
RAG can improve the reliability of answers by providing relevant external information, but it does not guarantee that every answer will be correct.
For example:
- The retrieved document may be outdated.
- The search may retrieve the wrong information.
- The Agent may misunderstand the retrieved information.
- The documents may contain incorrect information.
Therefore, production RAG systems should include appropriate testing, validation, monitoring, and source handling.
RAG and Tools Can Work Together
RAG does not replace tools.
They can work together.
For example:
User:
What database does the company use,
and how many customers registered today?
The Agent may need two different capabilities.
AI Agent
|
+--------------+--------------+
| |
v v
RAG Search Database Tool
| |
v v
Company Documents Current Data
| |
+--------------+--------------+
|
v
LLM
|
v
Answer
The Agent can retrieve company documentation using RAG and retrieve current data using a database tool.
RAG in Real-World Applications
RAG can be used in many applications.
Examples include:
Customer Support
An Agent searches product manuals and support documentation before answering customers.
Software Development
An Agent searches API documentation, project documentation, and coding standards.
Enterprise Knowledge
An Agent searches internal company policies and procedures.
Education
An Agent searches course materials and educational documents.
Legal and Compliance
An Agent can retrieve relevant documents and policies from an authorized knowledge base.
Technical Support
An Agent searches troubleshooting guides and technical manuals.
Security and Access Control
RAG systems also need security.
Imagine that a company has:
Public Documents
Internal Documents
Confidential Documents
Not every user should be able to search all documents.
The RAG system should therefore consider:
User
|
v
Authentication
|
v
Authorization
|
v
Allowed Documents
|
v
RAG Search
For example, an employee might be allowed to search internal technical documentation but not confidential financial documents.
Keeping the Knowledge Up to Date
Another important consideration is document maintenance.
Suppose a company changes its database technology.
The old document says:
SQL Server is the standard database.
Later, the company moves to PostgreSQL.
If the old document remains in the knowledge base, the Agent may retrieve outdated information.
Therefore, a production RAG system should include processes for:
- Updating documents
- Removing outdated documents
- Adding new documents
- Tracking document versions
- Controlling access
- Monitoring retrieval quality
What We Have Added to Our Agent
Our Agent has now evolved considerably.
At the beginning:
Agent
|
+-- LLM
After adding tools:
Agent
|
+-- LLM
|
+-- Tools
After adding memory:
Agent
|
+-- LLM
|
+-- Tools
|
+-- Memory
Now:
Agent
|
+-- LLM
|
+-- Tools
|
+-- Memory
|
+-- RAG
|
+-- Knowledge Base
+-- Search
+-- Retrieved Information
This is becoming a much more realistic AI Agent architecture.
What We Have Learned
In this article, we introduced RAG – Retrieval-Augmented Generation.
We learned that RAG allows an AI Agent to retrieve relevant information from external knowledge sources and provide that information to an AI model when generating an answer.
We explored:
- What RAG means
- Why RAG is useful
- Documents and knowledge bases
- Document chunks
- Embeddings
- Vector search
- Retrieval
- RAG and memory
- RAG and tools
- Enterprise applications
- Security and access control
The most important concept is:
External Knowledge
↓
Retrieval
↓
Relevant Information
↓
LLM
↓
Answer
Conclusion
RAG is one of the most important technologies used when building practical AI applications and AI Agents.
It allows an Agent to work with information that is outside the model’s built-in knowledge, such as company documentation, manuals, project files, and other authorized knowledge sources.
When RAG is combined with tools and memory, an Agent becomes significantly more capable:
AI Agent
|
+---------------+---------------+
| | |
v v v
LLM Memory Tools
|
v
RAG
|
v
External Knowledge
In the next article, we will explore MCP – Model Context Protocol, and see how it can provide a standardized way for AI applications and Agents to connect with external tools, data sources, and services.
← Back to AI Agents – Step-by-Step