Fine-tuning vs RAG what your AI agent needs to know, and what it needs to become

I

Three distinct layers: knowledge, know-how, infrastructure

Research literature distinguishes two memories in a language model. Lewis and his co-authors, in 2020, speak of “parametric” memory (what the model learned during its training, stored in its weights) and “non-parametric” memory (an index of documents consulted at the time of answering)¹.

This distinction is enough to clarify the fine-tuning vs RAG debate. RAG acts on what the AI agent can consult: your product pages, your terms and conditions, your procedures. Fine-tuning acts on how the model behaves: its tone, its response formats, its vocabulary, the way it follows your rules. A third question remains: where this model runs, and what you keep if you change provider.

II

System instructions: the first lever, often sufficient

OpenAI’s model optimization guide places work on instructions before fine-tuning, which it presents as optional, and states that this step may be enough to get good results⁵.

System instructions are text placed upstream of every conversation. They set the AI agent’s role, its scope, the topics it does not cover, the point at which it offers a human handoff. A business team can review and correct them the same day. Their limit lies in their nature: they steer a model without teaching it new reflexes, and each added instruction lengthens the request, which has a cost with every exchange⁵. As long as a problem can be fixed by a clear sentence in the instructions, it calls for neither retraining nor infrastructure.

III

RAG: giving the AI agent access to your documents

The principle

RAG (Retrieval-Augmented Generation) was formalized by Lewis et al. in the paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, presented at NeurIPS 2020¹. The system pairs a generative model with an index of documents queried by a neural search engine. For each question, it retrieves the relevant passages, then writes its answer based on them.

The authors started from an observation: models that rely only on their parameters cannot easily revise their memory, explain their answers poorly and can produce “hallucinations”¹. They observe, qualitatively, that their RAG models hallucinate less and more often produce factually correct text than the baseline model without retrieval¹.

Its strengths: freshness and traceability

Freshness is RAG’s first strength, and the original paper demonstrates it. The authors queried their model on 82 world leaders who changed between 2016 and 2018. With the 2016 index, the model answers 70% of questions about 2016 correctly; with the 2018 index, 68% of questions about 2018; with a mismatched index, accuracy falls to 12% and 4%¹. Their conclusion: the system’s knowledge can be updated simply by replacing its index, without retraining¹. For customer experience, a change in pricing or return policy becomes available as soon as the source document is updated.

Traceability is the second strength. Because the answer relies on identified passages, it becomes possible to display the source, check that it is up to date and know which document to fix in case of error. The authors stress that, in a hybrid model, the knowledge consulted can be inspected and interpreted¹.

Its limits: corpus, chunking, retrieval

RAG is only as good as the corpus. An outdated procedure or two contradictory documents will be returned with the same confidence as correct information. Document quality becomes an operational responsibility.

Chunking matters next. Before indexing, documents are split into passages: a passage that is too short loses its context (a pricing exception separated from the rule it modifies), a passage that is too long dilutes the useful information.

Retrieval, finally, can fail silently. If the right passage is not found, the model answers with what it has. And even when it is found, its position matters: Liu et al. (2023) show that models make better use of information placed at the beginning or end of a long context than in the middle, including for models designed for long contexts².

IV

Fine-tuning: changing how the model behaves

Full or efficient

Fine-tuning continues the training of an existing model on chosen examples. In its full version, all of the model’s parameters are updated, which requires computing power and storage proportional to its size.

So-called efficient methods have changed the economics of this operation. LoRA (Low-Rank Adaptation), introduced by Hu and colleagues in 2021, freezes the weights of the original model and trains only small matrices added to its layers³. On GPT-3 175B, the authors reduce the number of trainable parameters by a factor of 10,000 and the GPU memory requirement by a factor of 3, with no additional inference latency³. In their configuration, the checkpoint goes from about 350 GB to 35 MB³. QLoRA (Dettmers and colleagues, 2023) took the approach further by making it possible to fine-tune a 65-billion-parameter model on a single 48 GB GPU⁴.

The same base model can thus carry several interchangeable adaptations: Hu et al. calculate that storing 100 adapted models requires about 354 GB instead of 35 TB³.

What it really brings

OpenAI’s documentation lists the uses for which fine-tuning is relevant: producing content in a precise format, adopting the right tone and style, and fixing failures to follow instructions⁵. It also mentions the ability to cover more examples than fit in a single request, and to train a smaller, cheaper, faster model to excel at a task where a large model isn’t cost-effective⁵.

In business terms, fine-tuning teaches the AI agent how your company speaks: register, answer structure, technical vocabulary, reflexes when facing a complaint.

What it doesn’t provide

Fine-tuning is not the right tool for injecting facts, let alone facts that change. Ovadia and colleagues (2023) compare RAG with unsupervised fine-tuning on knowledge-intensive tasks: RAG consistently wins, both for knowledge seen during training and for entirely new knowledge⁶. Gekhman et al. (EMNLP 2024) go further: examples carrying new knowledge are learned more slowly than those that confirm what the model already knows, and once learned, they increase the model’s tendency to hallucinate⁷. Their reading is that models acquire most of their factual knowledge during pre-training, and that fine-tuning mainly teaches them to use it better⁷.

For a catalog, pricing or delivery times, the conclusion is straightforward: this information belongs to RAG and tool connections, not to the model’s weights.

V

Combine the two rather than choose

The “fine-tuning vs RAG” question doesn’t force a choice. Balaguer et al. (2024), on an agricultural use case, measure an accuracy gain of more than 6 points from fine-tuning, then 5 additional points by adding RAG⁸. The effects add up.

Some methods even fine-tune a model specifically to make better use of RAG. RAFT (Zhang et al., 2024) trains the model to ignore retrieved documents that don’t help answer and to cite the relevant passage⁹.

The target architecture can therefore be summed up like this: clear instructions for the framework, RAG for facts, tool connections for real-time data (orders, stock, appointments), and a fine-tuned model for behavior.

VI

Open models or APIs: a question of license and control

Open-weight models can be hosted wherever you decide, but their licenses vary. Mistral AI’s documentation, accessed in October 2026, lists several models under the Apache 2.0 license, such as Mistral Small 4, others under a modified MIT license, and one model under the non-commercial CC BY-NC 4.0 license¹⁰. Meta’s Llama 4 license requires displaying the “Built with Llama” notice when a product integrates it, and requires authorization from Meta for companies whose services exceeded 700 million monthly active users on the model’s release date¹¹. “Open” therefore doesn’t mean “unconditional”: each license should be read before choosing the model.

Models accessed through an API spare you any infrastructure to manage, at the price of depending on the provider’s decisions. On May 7, 2026, OpenAI announced the gradual shutdown of its fine-tuning platform available directly to developers: closed since that date to organizations that had never used it, it will no longer let active customers start new training runs from January 6, 2027¹². Models that were already fine-tuned remain usable until their base model is deprecated¹². An adaptation built on a closed model remains tied to that model’s life cycle.

VII

The dedicated model: where it runs changes what it’s worth

The European Data Act, applicable since September 12, 2025, sets a framework that lets customers actually switch data processing service providers¹³.

In practice, there are four hosting levels for a fine-tuned model. Shared infrastructure splits computing resources among several customers, with logical separation of data and adaptations. The managed private model is reserved for a single customer and operated by the provider. The dedicated GPU also reserves the hardware. On-premises installation places the model within the company’s own walls.

Each level balances three concerns. Confidentiality: the stronger the isolation, the smaller and the more documentable the scope of access to conversations and training data. Cost: sharing spreads the cost of computing, whereas dedicated hardware is paid for whether it is used or not. Reversibility: an open model fine-tuned with LoRA produces a compact adaptation file (35 MB in the configuration the authors tested on GPT-3³) that the company can keep, provided the contract allows it.

VIII

Evaluate before deploying, then continuously

OpenAI’s documentation recommends systematically measuring performance, establishing a baseline and testing on inputs comparable to those in production⁵. It even suggests writing the evaluations before the instructions, in an approach close to behavior-driven development⁵.

For a customer relationship AI agent, this translates into a business test set: real, anonymized questions, with the expected answer validated by your teams. It includes common cases, edge cases (a refund request past the deadline), questions the AI agent must not answer, and those that call for a human handoff. Every change to instructions, corpus or model reruns this test set.

Measuring hallucinations requires separating two causes. The Ragas evaluation framework (Es et al., 2023) proposes evaluating separately the system’s ability to retrieve relevant passages, the faithfulness with which the model uses them, and the quality of the answer itself, without requiring human reference annotations¹⁴. This separation shows where to act: corpus and retrieval, or model.

IX

Tomorrow: the fine-tuned model becomes a company asset

What follows is a forward-looking analysis, not an observation. An AI agent’s conversations contain what companies struggle to formalize: the phrasings that reassure, the recurring objections, the answers that convert, the cases that require a human. Fine-tuning a model on these exchanges, once reviewed and validated by your teams, amounts to codifying know-how. With methods like LoRA, this know-how takes the form of a compact adaptation file³, and the validated data that produced it can be used to fine-tune the next base model again. Our hypothesis is that this file will become an asset managed like a knowledge base or a CRM.

This outlook has a framework. In its June 19, 2025 recommendations, the CNIL states that reusing the future conversations of a conversational agent’s users to improve a model can rely on legitimate interest, provided strong safeguards are in place: informing individuals, an unconditional right to object, and limiting processing to certain pseudonymized or anonymized data¹⁵. The European Data Protection Board considers, in its Opinion 28/2024 of December 18, 2024, that the anonymity of a model trained on personal data is assessed case by case¹⁷. In July 2025, the CNIL pointed out that the GDPR often applies to these models because of their memorization capabilities, and notably recommends robust filters at the level of the system that wraps the model¹⁶.

X

What this means for your organization

Four decisions can be made right now. Appoint an owner for the document corpus, with a review rhythm: it’s the condition for any reliable RAG. Build your business test set before choosing a method, so you can compare the options on your own cases. Check in your contracts the ownership and portability of model adaptations, and the license of the base model. Involve your data protection officer as soon as reusing conversations for training is on the table, relying on the CNIL’s recommendations¹⁵¹⁶. Your teams stay in control at every step: they are the ones who validate what the AI agent learns.

XI

Where QLOA stands

XII

Frequently asked questions

What is the difference between fine-tuning and RAG?

RAG gives the model access to documents at the moment it answers: it retrieves the relevant passages and relies on them, which allows up-to-date, sourced answers. Fine-tuning continues training the model on chosen examples to change its behavior: tone, format, vocabulary, instruction-following. The first acts on what the AI agent knows at the moment of answering, the second on how it answers.

Can fine-tuning teach an AI agent my product catalog?

It’s not the right method. A 2023 comparative study shows that RAG outperforms unsupervised fine-tuning for injecting knowledge, whether new or not, and that learning new facts through fine-tuning can increase hallucinations. A catalog, prices or stock change constantly: they must be looked up in your documents and tools at the moment of the question, not written into the model’s weights.

What is LoRA in artificial intelligence?

LoRA (Low-Rank Adaptation) is a fine-tuning method published in 2021. It freezes the original model’s weights and trains only small matrices added to its layers. On GPT-3 175B, it reduces the number of trainable parameters by a factor of 10,000 and the required GPU memory by a factor of 3, without slowing down responses. In the configuration tested by its authors, the adaptation fits in a file of about 35 MB, versus 350 GB for the full model.

Can you combine RAG and fine-tuning?

Yes, and the two methods complement each other, because they don’t act on the same thing. A 2024 study on an agricultural use case measures more than 6 points of accuracy gained through fine-tuning and 5 additional points with RAG. Some methods, like RAFT, even fine-tune the model so it learns to discard retrieved documents that don’t serve the answer.

Can you train an AI agent on its customers’ conversations?

It’s possible under conditions. Since June 2025, the CNIL has stated that reusing a conversational agent’s conversations to improve a model can rely on legitimate interest, with strong safeguards: informing individuals, an unconditional right to object, and limiting use to certain pseudonymized or anonymized data. According to the CNIL, the GDPR often applies to the model itself, because of its memorization capabilities. Involve your data protection officer from the start.

Is an open-source model free for commercial use?

Not always. Licenses vary from one model to another, even within the same publisher. Mistral AI publishes models under Apache 2.0, under a modified MIT license and under a non-commercial license. The Llama 4 license requires an attribution notice and provides for specific authorization beyond 700 million monthly active users. Each license should be read before choosing the base model.

XIII

Sources

1. 1

   Lewis P., Perez E., Piktus A. et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 2020, arXiv, May 2020. arxiv.org

2. 2

   Liu N. F., Lin K., Hewitt J. et al., “Lost in the Middle: How Language Models Use Long Contexts,” TACL, arXiv, July 2023. arxiv.org

3. 3

   Hu E. J., Shen Y., Wallis P. et al., “LoRA: Low-Rank Adaptation of Large Language Models,” arXiv, June 2021. arxiv.org

4. 4

   Dettmers T., Pagnoni A., Holtzman A., Zettlemoyer L., “QLoRA: Efficient Finetuning of Quantized LLMs,” arXiv, May 2023. arxiv.org

5. 5

   OpenAI, “Model optimization,” developer documentation, accessed October 23, 2026. developers.openai.com

6. 6

   Ovadia O., Brief M., Mishaeli M., Elisha O., “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs,” arXiv, December 2023. arxiv.org

7. 7

   Gekhman Z., Yona G., Aharoni R. et al., “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?,” EMNLP 2024, arXiv, May 2024. arxiv.org

8. 8

   Balaguer A. et al., “RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture,” arXiv, January 2024. arxiv.org

9. 9

   Zhang T., Patil S. G., Jain N. et al., “RAFT: Adapting Language Model to Domain Specific RAG,” arXiv, March 2024. arxiv.org

10. 10

    Mistral AI, “Models,” official documentation, accessed October 23, 2026. docs.mistral.ai

11. 11

    Meta, “Llama 4 Community License Agreement,” official page of the Llama 4 Maverick model on Hugging Face, accessed October 23, 2026. huggingface.co

12. 12

    OpenAI, “Deprecations,” section “Update to OpenAI’s self-serve fine-tuning,” May 7, 2026. developers.openai.com

13. 13

    European Commission, “Data Act,” policy page, accessed October 23, 2026. digital-strategy.ec.europa.eu

14. 14

    Es S., James J., Espinosa-Anke L., Schockaert S., “Ragas: Automated Evaluation of Retrieval Augmented Generation,” arXiv, September 2023. arxiv.org

15. 15

    CNIL, “Development of AI systems: the CNIL publishes its recommendations on legitimate interest”, June 19, 2025. cnil.fr

16. 16

    CNIL, “AI: the CNIL finalizes its recommendations on the development of AI systems and announces its future work”, July 22, 2025. cnil.fr

17. 17

    European Data Protection Board (EDPB), “EDPB opinion on AI models: GDPR principles support responsible AI,” Opinion 28/2024, December 18, 2024. edpb.europa.eu