Your assistant built on a language model does more or less what you expect, and it is this “more or less” that keeps you busy. The format of the answers drifts, the business vocabulary remains approximate, the prompt grows longer with every correction. Someone then suggests fine-tuning the LLM.
Fine-tuning an LLM, or adjusting the model, continues the training of an existing model on your own examples and changes its parameters. It lastingly changes a behaviour: a format, a register, a narrow task. It brings no up-to-date knowledge and cites no source. The decision is therefore taken in a precise order: work on the prompt, provide the right documents, and adjust the model only if a behavioural gap, measured on an evaluation set, resists the first two levers.
Prompt, RAG or fine-tuning: locate the problem before choosing the lever
The three levers do not act in the same place. The instruction (prompt) changes what the model is asked. Document search changes what it reads before answering. Fine-tuning changes the model itself. Our glossary entries describe how an LLM works and how RAG works, including a comparison with fine-tuning. What remains is the diagnosis: what is the nature of the gap you observe?
| What you observe | Nature of the problem | Lever to try first |
|---|---|---|
| The assistant does not know an internal or recent fact | Missing knowledge | Document search (RAG) |
| It relies on an outdated version of a procedure | Poorly maintained corpus | Correction of the document base |
| The output format varies despite a precise prompt | Behaviour | Examples in the prompt, enforced structured output, then fine-tuning |
| The business vocabulary or register is poorly kept | Behaviour | Examples in the prompt, then fine-tuning if the gap persists |
| The very long prompt is sent with every call at high volume | Cost and response time | Shortening the prompt, then fine-tuning a smaller model or distillation |
| Reasoning fails on complex cases | Model capability | Breaking down the task or changing model |
The last row deserves a clarification. Fine-tuning specialises a model on what its examples show. When difficult cases fail for lack of reasoning capability, it rarely provides the answer: that is more often to be found in how the task is broken down or in the choice of model.
Fine-tuning an LLM: full fine-tuning, LoRA, distillation
The term covers several techniques, which involve neither the same resources nor the same operation.
Full fine-tuning updates all the model’s parameters from pairs associating an input with the expected output. It produces a complete copy of the model and requires the greatest computing capacity. It also exposes the model to forgetting: by specialising, the model can lose part of its general abilities.
Parameter-efficient methods freeze the base model and train only a small set of added parameters. LoRA (low-rank adaptation) is a common example. The correction made to certain layers of the model is represented there by the product of two low-rank matrices, which alone are trained. The result is a small adapter, loaded on top of the base model. The same model can thus serve several tasks, with one adapter per task.
Distillation answers a different question. A smaller “student” model is trained to reproduce the outputs of a larger “teacher” model on the target task. The aim is to keep a behaviour acquired on a defined scope with a model that is lighter to host. Outside that scope, the student does not inherit the teacher’s abilities. When the teacher is a provider’s model, its terms of use should be read before reusing its outputs for training.
These techniques are applied either through a provider’s fine-tuning service, for the models it opens to this possibility, or on an open-weight model that you host. The choice determines who holds the fine-tuned model and who ensures its continuity.
The criteria that justify adjusting the model
The nature of the gap. Fine-tuning is justified for a behavioural gap: format, register, compliance with a convention, a repetitive and narrow task such as classification or extraction. It is not justified for a knowledge gap.
A measured baseline. You have an evaluation set and a starting result, obtained with the base model, a carefully written prompt and a few examples. Without this baseline, nothing allows you to say that fine-tuning improves anything.
Examples that you can produce and use. Training requires pairs of inputs and expected outputs, validated by business experts and representative of real requests, edge cases included. A share of them is set aside for evaluation and is never used for training. When these examples come from real exchanges, they often contain personal data: their use falls under the GDPR, Regulation (EU) 2016/679, and a model can reproduce elements of its training data. Our article on data anonymisation covers this point.
A stable task. A fine-tuned model freezes what it has learnt. If the business rules change often, each change calls for new training, whereas a prompt or a document can be corrected without training.
A team to operate it. Fine-tuning adds an artefact to maintain. The last section of this article gives a measure of it.
The cases where it is better to refrain
- The need concerns facts, procedures or prices that change. A fine-tuned model does not cite a source and is not updated without retraining.
- The information is not meant to be read by everyone. What a model has learnt, it can reproduce for any user. A RAG system can filter documents according to access rights at the moment of the search; a fine-tuned model offers no equivalent filtering.
- No evaluation set exists. Fine-tuning then becomes a gamble, and its result an impression.
- The examples are scarce, heterogeneous or disputed by the experts themselves. The model learns the flaws of its examples as diligently as their qualities.
- The prompt has not been seriously worked on. Rewritten instructions, a few well-chosen examples and a structured output often remove format gaps.
- No one is appointed to take over the training when the base model is withdrawn or replaced.
After go-live: one more artefact to operate
A fine-tuned model is tied to a precise version of its base model. When that version is replaced, the adapter cannot be carried over: you have to retrain, then re-evaluate. The training set, the adapter and the evaluation results are versioned together, and each change replays the reference set. Two checks are added to those of an ordinary assistant: comparison with the base model on the same cases, to verify the gain, and a test on general requests, to spot what the model may have unlearnt. This discipline belongs to the chain described on our MLOps and LLMOps page.
Fine-tuning can also shift your position under Regulation (EU) 2024/1689, which provides for a specific regime for general-purpose AI models. The question is examined with AI governance before the first training run.
The next step requires no training. Build the evaluation set with the business experts, measure what the prompt enriched with examples and then document search produce, and classify the remaining gaps according to their nature. This is the approach we follow to design an assistant, described on our generative AI in the enterprise page. If the gap that remains is behavioural, you will know which examples to train the model on and by which measure to judge the result.