From prototype to service: where models come to a halt
Two situations recur. In the first, the model achieves good results on its test set, the demonstration was convincing, and nothing is in production. In the second, the model is in service, deployed by hand by the person who designed it, and no one can say whether it is still working correctly. MLOps addresses both.
The gap between prototype and service stems from the nature of a model. Its behaviour depends on three elements that evolve separately: the code, the training data and the parameters. It can degrade without a single line of code changing, because the reality it describes has moved. And its real performance is often known only after a delay, once the prediction can finally be compared with what actually happened.
Applications based on language models add their own difficulties. Their outputs vary from one call to the next, there is no single right answer to check, and the model is most often supplied by a third party that changes its versions.
MLOps and LLMOps: a common foundation, real differences
The two disciplines share the same requirement: trace, test, monitor. They differ in the objects they handle.
| Subject | MLOps (predictive models) | LLMOps (language models) |
|---|---|---|
| What is versioned | Code, training data, parameters, trained model | Prompts, model configuration, document base and index, fine-tuned model if there is one |
| Evaluation | Metrics calculated on a labelled test set | A set of reference cases, multiple criteria, a share of human judgement |
| What drifts | Input data, the relationship between inputs and outcome | User requests, document content, the version of the supplied model |
| Dominant cost | Training and hosting | Usage, proportional to the volumes of text processed |
| Specific risks | Bias, silent degradation | Invented answers, prompt injection, disclosure of information |
For a general presentation of the discipline, see our glossary entry on the definition of MLOps.
Version, deploy, monitor: the pipeline to put in place
Versioning and tracing. Every model in production must be able to be rebuilt: code version, dataset, parameters, evaluation results. A model registry, kept in a tool such as MLflow, shows which version serves which use and who approved it. A model card describes the intended use, the data used and the known limitations.
Deploying without disruption. Production release goes through an automated pipeline: tests, validation against a performance threshold, promotion between environments. Two techniques limit the risk. Shadow deployment submits the new model to real requests without using its answers. Progressive deployment initially entrusts it with only a fraction of requests. In both cases, reverting to the previous version is prepared in advance. These mechanisms extend the practices of DevOps and platform engineering.
Monitoring drift. Data drift appears when the inputs received depart from those used in training. Concept drift appears when the relationship between the inputs and the outcome changes. Monitoring therefore covers three levels: the service (latency, errors), the distributions of inputs and predictions, then real performance as soon as the observed outcome is available.
Responding. An alert with no associated action is useless. Each threshold refers to a course of action: investigation, retraining, rollback. Retraining can be automated, but putting the retrained model into service remains subject to validation.
Evaluating a language model: the heart of LLMOps
Without an evaluation set, every change to a prompt is a gamble. The framework rests on four elements.
- A set of reference cases, drawn from real requests. It includes the difficult cases and those where the application must refuse to answer. It is enriched with every incident.
- Explicit criteria: accuracy, faithfulness to sources, completeness, compliance with format and refusal instructions.
- Three complementary scoring modes: automated checks for what can be verified mechanically, a model used as a judge after calibration on expert scores, human review on a sample.
- A non-regression rule: no change to the prompt, the document base or the model version goes into service without the reference set having been rerun.
For a RAG system, retrieval and generation are evaluated separately. The first question is whether the right passages were retrieved. The second is whether the answer sticks to those passages. Confusing the two prevents corrections from being made in the right place. The design of the assistant itself (framing, corpus, architecture) falls under our generative AI in the enterprise offering.
In operation, monitoring covers the traces of exchanges, user feedback, cost per request and latency. Retention of traces is decided in line with the rules applicable to personal data.
Our approach: a first model end to end
We start from a real model, already prototyped or in service, and our consultants help your teams take it through a complete pipeline: versioning, automated deployment, monitoring, response procedure. This first journey sets the standards and reveals the obstacles specific to your organisation, whether technical or linked to the division of roles between data, operations and business teams.
The level of automation is proportionate. It depends on the number of models, how often they are updated and how critical they are. Shared components are only built when several teams need them.
The pipeline also produces evidence. For high-risk AI systems, the AI Act, Regulation (EU) 2024/1689, provides for requirements on technical documentation (Article 11), automatic recording of events (Article 12) and post-market monitoring by providers (Article 72). The registry, model cards and logs contribute to these. The qualification of systems and the overall framework fall under AI governance.
The pitfalls of going into production
- Building the platform before the first model. A complete foundation with no model to operate freezes choices that the first real cases will contradict.
- Monitoring the infrastructure and not the model. A service can respond quickly and wrongly. Technical indicators say nothing about correctness.
- Putting a retrained model into service without validation. A corrupted dataset then produces a degraded model, deployed automatically.
- Changing a prompt directly in production. An improvement on one case can degrade others, which only the reference set reveals.
- Depending on one person. A model that only its author knows how to restart is an operational risk.
- Discovering the cost of usage after success. For a language model, spending follows adoption. It is measured per request from the pilot onwards.
The indicators of a pipeline that works
- The time between the validation of a model and its production release.
- The share of models in service that can be rebuilt and have a named owner.
- The frequency of deployments, their success rate and the time needed for a rollback.
- The share of degradations detected by monitoring before being reported by a user.
- For language models: the result on the reference set at each version, the cost per request and the share of answers flagged.
These indicators are defined at framing and tracked by the team that operates the models. They also serve to decide when to retire a model: a model whose use no longer justifies its operating cost is decommissioned like any other service.