Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Artificial intelligence

LLM fine-tuning: should you adjust the model, or stick to the prompt and RAG?

In brief

Fine-tuning an LLM continues the training of an existing language model on selected examples in order to change its behaviour lastingly: a format, a register, a narrow task. It brings neither up-to-date knowledge nor verifiable sources: Management Square sets out here the criteria that justify fine-tuning and the cases where the prompt or RAG are enough.

Page contents

Your assistant built on a language model does more or less what you expect, and it is this “more or less” that keeps you busy. The format of the answers drifts, the business vocabulary remains approximate, the prompt grows longer with every correction. Someone then suggests fine-tuning the LLM.

Fine-tuning an LLM, or adjusting the model, continues the training of an existing model on your own examples and changes its parameters. It lastingly changes a behaviour: a format, a register, a narrow task. It brings no up-to-date knowledge and cites no source. The decision is therefore taken in a precise order: work on the prompt, provide the right documents, and adjust the model only if a behavioural gap, measured on an evaluation set, resists the first two levers.

Prompt, RAG or fine-tuning: locate the problem before choosing the lever

The three levers do not act in the same place. The instruction (prompt) changes what the model is asked. Document search changes what it reads before answering. Fine-tuning changes the model itself. Our glossary entries describe how an LLM works and how RAG works, including a comparison with fine-tuning. What remains is the diagnosis: what is the nature of the gap you observe?

What you observe Nature of the problem Lever to try first
The assistant does not know an internal or recent fact Missing knowledge Document search (RAG)
It relies on an outdated version of a procedure Poorly maintained corpus Correction of the document base
The output format varies despite a precise prompt Behaviour Examples in the prompt, enforced structured output, then fine-tuning
The business vocabulary or register is poorly kept Behaviour Examples in the prompt, then fine-tuning if the gap persists
The very long prompt is sent with every call at high volume Cost and response time Shortening the prompt, then fine-tuning a smaller model or distillation
Reasoning fails on complex cases Model capability Breaking down the task or changing model

The last row deserves a clarification. Fine-tuning specialises a model on what its examples show. When difficult cases fail for lack of reasoning capability, it rarely provides the answer: that is more often to be found in how the task is broken down or in the choice of model.

Fine-tuning an LLM: full fine-tuning, LoRA, distillation

The term covers several techniques, which involve neither the same resources nor the same operation.

Full fine-tuning updates all the model’s parameters from pairs associating an input with the expected output. It produces a complete copy of the model and requires the greatest computing capacity. It also exposes the model to forgetting: by specialising, the model can lose part of its general abilities.

Parameter-efficient methods freeze the base model and train only a small set of added parameters. LoRA (low-rank adaptation) is a common example. The correction made to certain layers of the model is represented there by the product of two low-rank matrices, which alone are trained. The result is a small adapter, loaded on top of the base model. The same model can thus serve several tasks, with one adapter per task.

Distillation answers a different question. A smaller “student” model is trained to reproduce the outputs of a larger “teacher” model on the target task. The aim is to keep a behaviour acquired on a defined scope with a model that is lighter to host. Outside that scope, the student does not inherit the teacher’s abilities. When the teacher is a provider’s model, its terms of use should be read before reusing its outputs for training.

These techniques are applied either through a provider’s fine-tuning service, for the models it opens to this possibility, or on an open-weight model that you host. The choice determines who holds the fine-tuned model and who ensures its continuity.

The criteria that justify adjusting the model

The nature of the gap. Fine-tuning is justified for a behavioural gap: format, register, compliance with a convention, a repetitive and narrow task such as classification or extraction. It is not justified for a knowledge gap.

A measured baseline. You have an evaluation set and a starting result, obtained with the base model, a carefully written prompt and a few examples. Without this baseline, nothing allows you to say that fine-tuning improves anything.

Examples that you can produce and use. Training requires pairs of inputs and expected outputs, validated by business experts and representative of real requests, edge cases included. A share of them is set aside for evaluation and is never used for training. When these examples come from real exchanges, they often contain personal data: their use falls under the GDPR, Regulation (EU) 2016/679, and a model can reproduce elements of its training data. Our article on data anonymisation covers this point.

A stable task. A fine-tuned model freezes what it has learnt. If the business rules change often, each change calls for new training, whereas a prompt or a document can be corrected without training.

A team to operate it. Fine-tuning adds an artefact to maintain. The last section of this article gives a measure of it.

The cases where it is better to refrain

  • The need concerns facts, procedures or prices that change. A fine-tuned model does not cite a source and is not updated without retraining.
  • The information is not meant to be read by everyone. What a model has learnt, it can reproduce for any user. A RAG system can filter documents according to access rights at the moment of the search; a fine-tuned model offers no equivalent filtering.
  • No evaluation set exists. Fine-tuning then becomes a gamble, and its result an impression.
  • The examples are scarce, heterogeneous or disputed by the experts themselves. The model learns the flaws of its examples as diligently as their qualities.
  • The prompt has not been seriously worked on. Rewritten instructions, a few well-chosen examples and a structured output often remove format gaps.
  • No one is appointed to take over the training when the base model is withdrawn or replaced.

After go-live: one more artefact to operate

A fine-tuned model is tied to a precise version of its base model. When that version is replaced, the adapter cannot be carried over: you have to retrain, then re-evaluate. The training set, the adapter and the evaluation results are versioned together, and each change replays the reference set. Two checks are added to those of an ordinary assistant: comparison with the base model on the same cases, to verify the gain, and a test on general requests, to spot what the model may have unlearnt. This discipline belongs to the chain described on our MLOps and LLMOps page.

Fine-tuning can also shift your position under Regulation (EU) 2024/1689, which provides for a specific regime for general-purpose AI models. The question is examined with AI governance before the first training run.

The next step requires no training. Build the evaluation set with the business experts, measure what the prompt enriched with examples and then document search produce, and classify the remaining gaps according to their nature. This is the approach we follow to design an assistant, described on our generative AI in the enterprise page. If the gap that remains is behavioural, you will know which examples to train the model on and by which measure to judge the result.

Insights

Further reading

All insights

FAQ

Frequently asked questions

Another question? Ask us

Do you have to choose between fine-tuning an LLM and RAG?

No, the two can be combined because they do not address the same problem. RAG gives the model the documents to read at the moment of the question. Fine-tuning changes the way the model answers. A model fine-tuned to keep to a strict output format can perfectly well receive its facts from a document search.

How many examples are needed to fine-tune a language model?

There is no universal threshold: the need depends on the task, the technique and the gap to be closed. The quality and representativeness of the examples matter more than their number. The prudent method is to start with a small set, validated by business experts, then to add to it as long as the result improves on the evaluation set.

What is LoRA?

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method. The base model remains frozen. Only small matrices added to some of its layers are trained. The result is a lightweight adapter, loaded on top of the base model, which is stored and versioned separately.

Can a proprietary model be fine-tuned?

That depends on the provider. Some offer a fine-tuning service for some of their models: the fine-tuned model then remains hosted by them and you do not hold its parameters. The terms of use of the examples sent, how long the base model will remain available and reversibility should be checked in the contract. An open-weight model that you host yourself removes these dependencies, at the cost of the infrastructure and its operation.

Does fine-tuning reduce hallucinations?

Not on facts. A fine-tuned model still produces the most plausible text, and it does not cite its sources. It can learn to respect a refusal instruction or a format better, which limits certain behavioural errors. For answers based on verifiable facts, the lever remains document search, supplemented by a check on outputs.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.