More and more companies are looking to adopt artificial intelligence to improve internal processes, automate tasks and make use of the information they already hold. Yet a model like ChatGPT — or any generic LLM — knows nothing about an organisation's documents, contracts, procedures or internal systems. This is where RAG (Retrieval-Augmented Generation) comes in: the architecture used to build enterprise AI solutions capable of answering from a company's own data.

This article explains, from an engineering standpoint, what RAG is, how it works, what infrastructure it requires, how permissions and security are handled in a private AI deployment, which use cases deliver real value — technical documentation, ERP, CRM, support, human resources — and where its scope ends. It also clears up two frequent misconceptions: the difference between RAG and fine-tuning, and the relationship between RAG and MCP (Model Context Protocol), two complementary pieces in modern AI agent architectures.

In this guide you will find:

• What RAG is and why it underpins enterprise artificial intelligence.
• How it works across documents, ERP, CRM and internal systems.
• How it differs from ChatGPT, fine-tuning and MCP.
• The recommended architecture for taking it to production.
• Real use cases, and the problems it does not solve.
• How to implement a secure corporate AI, with permissions and traceability.

What Is RAG and What Does It Solve for a Company?

RAG is an architectural pattern that combines two capabilities: retrieving relevant information from the organisation's sources and generating an answer grounded exclusively in that material. Instead of expecting the model to "remember" corporate data, the system looks it up at query time and supplies it as verifiable context.

The usual analogy is an open-book exam. A language model with no access to documentation answers from memory, with the attendant risk of invention; the same model with the right fragment in front of it answers from evidence and can cite its source. That difference — answering with documentation rather than about documentation it never saw — is what turns an interesting demo into a tool a team actually uses every day.

In business terms, the concrete outcome is an enterprise AI assistant that answers real operational questions: what the current procedure says, what terms a contract carries, how a similar support case was resolved, what the internal purchasing policy establishes. And it does so by indicating which document each statement came from, which is the only way a team accountable for that data will agree to rely on it.

Why Doesn't ChatGPT or a Generic LLM Know Your Company?

A commercial language model is essentially a reasoning engine trained on public information up to a cut-off date. It never had access to the company's ERP, document management system or ticketing platform. When asked about an internal policy, an active contract or the status of an order, it lacks the information required to answer — and unless the system is designed to prevent it, it will produce a plausible answer regardless.

That behaviour, commonly called hallucination, is not an occasional flaw to be fixed with better instructions: it is the predictable consequence of asking a probabilistic system to complete information it never received. The operational conclusion is straightforward: the problem is not the model's capability, it is context. A mid-tier model with access to the right document consistently outperforms a state-of-the-art model answering from memory.

Designing an enterprise AI solution therefore looks far less like a model selection exercise and far more like a classic information architecture and systems integration problem: where the data lives, how it is accessed, who is allowed to see it, and at what latency.

How Does RAG Work? The Path of a Single Query

The full path of a query in an AI solution built on proprietary data runs through six stages:

1. Upfront indexing. Company documents and records are processed once: text is extracted, split into fragments, and each fragment is converted into a numerical vector (an embedding) representing its meaning.
2. User query. The question is transformed using the same embedding model, so that it is expressed in the same semantic space as the corpus.
3. Retrieval. The engine searches for the fragments closest to the query, combining semantic and keyword search while applying the user's permission filters.
4. Reranking. A reranking model evaluates the retrieved candidates and prioritises the genuinely relevant ones, discarding superficial matches.
5. Generation. The model receives the question together with the selected fragments and an explicit instruction: answer only from that material and acknowledge when the information is not available.
6. Delivery with citations. The answer is presented alongside its source documents, so any user can verify it against the original.

Final quality depends far more on stages 1, 3 and 4 than on the model chosen at stage 5. A poorly segmented corpus or imprecise retrieval will produce weak answers even with the best model on the market.

Components of a RAG Architecture in Production

There is a substantial engineering gap between a prototype that answers well across twenty PDFs and a system that sustains thousands of daily queries against a company's entire document repository. A serious implementation involves, at minimum:

• Ingestion and normalisation: connectors to the real sources — document manager, intranet, SharePoint, Google Drive, ERP, knowledge base, email — with reliable text extraction from heterogeneous formats, including scanned PDFs requiring OCR and spreadsheets with implicit structure.
• Chunking: splitting content into fragments that preserve semantic units. Mechanical splitting by character count cuts contractual clauses and tables in half, and degrades quality more than any other decision in the pipeline.
• Embeddings and vector database: a numerical representation of each fragment, stored alongside metadata for source, version, date and permissions.
• Hybrid retrieval and reranking: semantic search combined with lexical search — essential for product codes, standard numbers or internal nomenclature — followed by relevance-based reordering.
• Orchestration and application layer: the service that handles authentication, assembles the prompt, controls costs and surfaces the functionality inside the tools people already use.
• Observability and evaluation: logging of queries, retrieved fragments and generated answers, plus a reference question set to measure quality continuously.

Which Vector Database Should You Choose?

There is no universal answer, and in practice the decision is driven by infrastructure context rather than benchmarks. Most projects do not need a dedicated vector database in their early stages: starting with PostgreSQL + pgvector usually simplifies operations and reduces cost. If the organisation already runs PostgreSQL, the extension avoids adding a new component to the stack and is more than adequate for mid-sized volumes. If a search engine is already deployed, OpenSearch or Elasticsearch provide hybrid search with no additional infrastructure. For large volumes or specific filtering and latency requirements, dedicated engines such as Qdrant, Weaviate or Milvus offer better performance; managed services reduce operational load in exchange for vendor dependency.

Our criterion is deliberately conservative: the vector database is rarely the bottleneck in an enterprise AI project. Ingestion quality, chunking and the permission model are. Choose the engine that adds the least complexity to the existing cloud architecture and concentrate effort where the outcome is actually decided.

Permissions, Security and Private AI

This is consistently what separates a proof of concept from a corporate AI fit to operate in production. An assistant that retrieves information from the entire repository without replicating the permission model of the source systems is a first-order security risk: it makes queryable in natural language exactly what the organisation had carefully restricted by role.

Permission filtering must be applied at the retrieval stage, not on the final answer, and must be based on the real identity of the user making the query. This is an architectural decision, not a later adjustment: if the index does not store access control lists alongside each fragment, no prompt instruction will fix the problem.

The second recurring question is where the data ends up. There are three routes, depending on the regulatory framework and the sensitivity of the information: enterprise agreements with commercial models under zero-retention and no-training clauses; deployment within the organisation's private cloud through services such as Azure OpenAI, Amazon Bedrock or Vertex AI; and open-source models running entirely on its own infrastructure, the usual choice when strict data residency requirements apply. A well-designed private AI deployment also logs every query and every retrieved fragment, which is what makes internal or regulatory audits possible.

Common Use Cases for AI on Documents, ERP and CRM

Projects that deliver early returns share a pattern: a large body of written information, many repeated queries, and a high cost of searching manually. It is what many companies search for today as "ChatGPT with documents" or an enterprise chat over internal documents. The most frequent scenarios are:

• Intelligent search over technical documentation: manuals, specifications, drawings, product sheets and system documentation — AI over SharePoint, Google Drive or whatever document manager the company already uses.
• Internal assistant for employees: a single point of enquiry for policies, procedures, benefits and internal regulations.
• Contract review: terms, expiry dates, renewal clauses and penalties, citing the exact document and section. For example: "which contracts expire this quarter?" or "what is the early-termination penalty with this supplier?".
• Quality and compliance: quality manuals, ISO standards, certified procedures and audit records.
• Support knowledge base and help desk: assisted resolution based on ticket history, surfacing analogous cases already solved.
• ERP queries: stock, order status, commercial terms or purchase history in natural language, over data read through an API integration. For example: "which orders are pending for customer Acme?" or "what was this supplier's last purchase?".
• CRM queries: customer history, opportunities, prior interactions and preparation for sales meetings. For example: "what conversations did we have with this account before the renewal?".
• Human resources: onboarding, collective agreements, leave, expense policies and answers to frequent staff questions.
• AI automation of administrative tasks: classifying and routing incoming requests, extracting data from documents and pre-filling forms.

In every case, the value comes not from the sophistication of the model but from the process automation of work that previously depended on someone knowing where to look.

What RAG Does Not Solve

Defining the boundaries prevents misplaced expectations and projects that fail through poor framing. RAG does not replace:

• an ERP or any transactional system: it does not execute operations or guarantee accounting consistency;
• a CRM: it does not manage the sales cycle, it only queries and summarises what the CRM records;
• a document management system: it does not version documents, control validity or administer their lifecycle;
• a workflow or BPM engine: it does not orchestrate processes with states, deadlines and owners;
• a data warehouse or BI tool: for consistent aggregate metrics, SQL over a dimensional model remains superior;
• a predictive model: forecasting demand, default risk or churn is a classic machine learning problem, not a text retrieval one.

There is one further limitation worth stating plainly: RAG does not fix an organisation's documentation. If procedures are outdated, duplicated or contradictory, the system will faithfully retrieve that disorder. In many projects, the first useful outcome of an enterprise AI pilot is not the assistant itself but a precise diagnosis of which documentation is missing or obsolete.

RAG, Prompting, Fine-Tuning and Agents: When to Use Each

The costliest misconception at the evaluation stage is assuming fine-tuning is the route to incorporating proprietary knowledge. Retraining a model adjusts its style and behaviour, but it is expensive to maintain against information that changes daily, it provides no traceability for the origin of each statement, and it forces the process to be repeated with every meaningful update to the corpus. For dynamic knowledge, retrieval wins on cost, on update speed and, above all, on auditability. Both approaches can nonetheless coexist within the same solution.

RAG vs MCP: What's the Difference?

RAG and MCP (Model Context Protocol) are frequently confused. Although both are used alongside language models, they solve different problems. RAG contributes knowledge by retrieving relevant information from documents and databases, whereas MCP standardises how a model interacts with tools, services and external systems.

In a modern architecture they coexist without conflict: RAG provides the context, MCP enables the safe execution of actions against corporate applications. An internal assistant can answer a policy question through RAG and, in the same conversation, check an order's status in the ERP through an MCP server that exposes that operation with permissions, limits and audit logging. From a microservices architecture perspective, MCP is above all a contract layer: it defines how available tools are described and invoked, reducing coupling between the model and each individual integration.

AI Agents: From Answering to Acting

Once corporate knowledge can be queried reliably, the natural evolution is to move from a system that answers to one that acts: an agent able to query a management system, generate a document, record an operation or trigger a workflow. The leap is technologically accessible and strategically delicate, because an error stops being an imprecise answer and becomes an incorrect transaction in a production system.

The criteria we apply at INTWAY to define an agent's scope combine three variables: reversibility of the action, cost of error and volume of the operation. High-volume, low-individual-impact, easily reversible tasks — classifying and routing requests, pre-filling forms, drafting response templates, reconciling records and flagging discrepancies — are solid candidates for direct automation. Operations with contractual, financial or regulatory effect must retain explicit human validation: the agent prepares, justifies and proposes; a person authorises.

None of this works outside conventional software engineering. An agent in production needs well-defined interfaces with existing systems, idempotency control to prevent duplication, spending and iteration limits, explicit error handling, observability over every executed action, and a rollback procedure. These are exactly the same requirements as any critical integration, with the added complication that the deciding component is probabilistic.

Conclusions: How to Move Forward with Enterprise AI

If your company is evaluating how to apply artificial intelligence to documentation, ERP, CRM or any internal data source, the main challenge is not choosing the most powerful model but designing an architecture that combines information retrieval, security, permissions and integration with existing systems.

Before choosing a model or a provider, three questions are worth answering: where is the data?, who is allowed to access it? and which processes are actually worth automating? Getting those three answers right usually has far more impact on the project's success than any change of AI model.

At INTWAY we build enterprise AI solutions based on RAG and AI agents that leverage each organisation's current infrastructure and make it possible to move from pilot to production with traceability and control, backed by more than 25 years of experience in custom software development for demanding production environments.