Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Data

MLOps and LLMOps: industrialising your AI models

A model that works in a notebook is not a service in production. Our consultants help your teams establish the pipeline that makes it possible to deliver, monitor and correct your models, predictive and generative alike.

Our principles

  • Reproducible, traceable models, each with an owner
  • Degradation detected before users notice
  • Language models evaluated on your real cases
In brief

MLOps is the set of practices that make it possible to put machine learning models into production, monitor them and evolve them reliably and reproducibly; LLMOps extends them to applications based on language models. Management Square consultants help your teams set up this pipeline: versioning, deployment, drift monitoring and evaluation.

Scope

How we contribute

  1. Diagnostic of the path to production

    Analysis of a model's current journey, from experimentation to operation: manual steps, dependencies on individuals, lack of monitoring, waiting times.

  2. Versioning and model registry

    Traceability of code, training data, parameters and evaluation results, with a registry that says which version serves which use.

  3. Deployment pipeline

    Automation of testing, validation and promotion between environments, with progressive deployment and rollback planned before going into service.

  4. Monitoring and drift management

    Tracking of input data, predictions and real performance, with alerts linked to an action: investigation, retraining or reverting to the previous version.

  5. Language model evaluation

    A set of reference cases, scoring criteria and a regression protocol for assistants, RAG systems and agents, before and after going into service.

  6. Shared foundation

    When several teams produce models: common components, project templates and shared rules, sized to the real number of models to be operated.

Page contents

From prototype to service: where models come to a halt

Two situations recur. In the first, the model achieves good results on its test set, the demonstration was convincing, and nothing is in production. In the second, the model is in service, deployed by hand by the person who designed it, and no one can say whether it is still working correctly. MLOps addresses both.

The gap between prototype and service stems from the nature of a model. Its behaviour depends on three elements that evolve separately: the code, the training data and the parameters. It can degrade without a single line of code changing, because the reality it describes has moved. And its real performance is often known only after a delay, once the prediction can finally be compared with what actually happened.

Applications based on language models add their own difficulties. Their outputs vary from one call to the next, there is no single right answer to check, and the model is most often supplied by a third party that changes its versions.

MLOps and LLMOps: a common foundation, real differences

The two disciplines share the same requirement: trace, test, monitor. They differ in the objects they handle.

Subject MLOps (predictive models) LLMOps (language models)
What is versioned Code, training data, parameters, trained model Prompts, model configuration, document base and index, fine-tuned model if there is one
Evaluation Metrics calculated on a labelled test set A set of reference cases, multiple criteria, a share of human judgement
What drifts Input data, the relationship between inputs and outcome User requests, document content, the version of the supplied model
Dominant cost Training and hosting Usage, proportional to the volumes of text processed
Specific risks Bias, silent degradation Invented answers, prompt injection, disclosure of information

For a general presentation of the discipline, see our glossary entry on the definition of MLOps.

Version, deploy, monitor: the pipeline to put in place

Versioning and tracing. Every model in production must be able to be rebuilt: code version, dataset, parameters, evaluation results. A model registry, kept in a tool such as MLflow, shows which version serves which use and who approved it. A model card describes the intended use, the data used and the known limitations.

Deploying without disruption. Production release goes through an automated pipeline: tests, validation against a performance threshold, promotion between environments. Two techniques limit the risk. Shadow deployment submits the new model to real requests without using its answers. Progressive deployment initially entrusts it with only a fraction of requests. In both cases, reverting to the previous version is prepared in advance. These mechanisms extend the practices of DevOps and platform engineering.

Monitoring drift. Data drift appears when the inputs received depart from those used in training. Concept drift appears when the relationship between the inputs and the outcome changes. Monitoring therefore covers three levels: the service (latency, errors), the distributions of inputs and predictions, then real performance as soon as the observed outcome is available.

Responding. An alert with no associated action is useless. Each threshold refers to a course of action: investigation, retraining, rollback. Retraining can be automated, but putting the retrained model into service remains subject to validation.

Evaluating a language model: the heart of LLMOps

Without an evaluation set, every change to a prompt is a gamble. The framework rests on four elements.

  • A set of reference cases, drawn from real requests. It includes the difficult cases and those where the application must refuse to answer. It is enriched with every incident.
  • Explicit criteria: accuracy, faithfulness to sources, completeness, compliance with format and refusal instructions.
  • Three complementary scoring modes: automated checks for what can be verified mechanically, a model used as a judge after calibration on expert scores, human review on a sample.
  • A non-regression rule: no change to the prompt, the document base or the model version goes into service without the reference set having been rerun.

For a RAG system, retrieval and generation are evaluated separately. The first question is whether the right passages were retrieved. The second is whether the answer sticks to those passages. Confusing the two prevents corrections from being made in the right place. The design of the assistant itself (framing, corpus, architecture) falls under our generative AI in the enterprise offering.

In operation, monitoring covers the traces of exchanges, user feedback, cost per request and latency. Retention of traces is decided in line with the rules applicable to personal data.

Our approach: a first model end to end

We start from a real model, already prototyped or in service, and our consultants help your teams take it through a complete pipeline: versioning, automated deployment, monitoring, response procedure. This first journey sets the standards and reveals the obstacles specific to your organisation, whether technical or linked to the division of roles between data, operations and business teams.

The level of automation is proportionate. It depends on the number of models, how often they are updated and how critical they are. Shared components are only built when several teams need them.

The pipeline also produces evidence. For high-risk AI systems, the AI Act, Regulation (EU) 2024/1689, provides for requirements on technical documentation (Article 11), automatic recording of events (Article 12) and post-market monitoring by providers (Article 72). The registry, model cards and logs contribute to these. The qualification of systems and the overall framework fall under AI governance.

The pitfalls of going into production

  • Building the platform before the first model. A complete foundation with no model to operate freezes choices that the first real cases will contradict.
  • Monitoring the infrastructure and not the model. A service can respond quickly and wrongly. Technical indicators say nothing about correctness.
  • Putting a retrained model into service without validation. A corrupted dataset then produces a degraded model, deployed automatically.
  • Changing a prompt directly in production. An improvement on one case can degrade others, which only the reference set reveals.
  • Depending on one person. A model that only its author knows how to restart is an operational risk.
  • Discovering the cost of usage after success. For a language model, spending follows adoption. It is measured per request from the pilot onwards.

The indicators of a pipeline that works

  • The time between the validation of a model and its production release.
  • The share of models in service that can be rebuilt and have a named owner.
  • The frequency of deployments, their success rate and the time needed for a rollback.
  • The share of degradations detected by monitoring before being reported by a user.
  • For language models: the result on the reference set at each version, the cost per request and the share of answers flagged.

These indicators are defined at framing and tracked by the team that operates the models. They also serve to decide when to retire a model: a model whose use no longer justifies its operating cost is decommissioned like any other service.

Typical work

What our consultants contribute to

Deployment pipeline

Training, validation and production release pipelines, documented and taken over by your teams.

Registry and model cards

An inventory of models in service with, for each, the intended use, the data used, the known limitations and the owner.

Monitoring framework

Dashboard, alert thresholds and response procedures for each model in production.

Evaluation protocol

A set of reference cases, a grid of criteria and regression rules applicable to every change.

Management Square in figures

300+
projects completed
43+
clients supported
Since2015
serving our clients
  • Allianz
  • Euler Hermes
  • Groupe BPCE
  • ODDO BHF
  • Candriam
  • Ledger
  • Saur
  • Veolia
  • Voltalia
  • emeis
  • Bonpoint
  • Legal Suite

FAQ

Frequently asked questions

Another question? Ask us

What is MLOps?

MLOps (Machine Learning Operations) applies to machine learning models the automation and operations principles that come from DevOps, adding what is specific to models: versioning of data and parameters, evaluation, drift monitoring and retraining. Its aim is for a model to be delivered, monitored and corrected as a service, and not as an experiment.

What is the difference between MLOps and LLMOps?

MLOps concerns the models the organisation trains itself, most often predictive ones. LLMOps concerns applications built on language models, generally supplied by a third party: what is versioned there is prompts and document bases rather than training sets, and evaluation bears on texts for which there is no single right answer. The principles of traceability, testing and monitoring remain the same.

What is model drift?

Drift refers to the degradation of a model caused by changes in its environment, without any modification of its code. A distinction is made between data drift, when the inputs received in production depart from those used in training, and concept drift, when the relationship between the inputs and the outcome to be predicted changes. Both are detected through continuous monitoring.

How do you evaluate an application based on a language model?

By building a set of reference cases drawn from real requests, with the expected answers or behaviours, then rerunning it at every change. Scoring combines automated checks, a model used as a judge and calibrated on expert scores, and human review on a sample. For a RAG system, document retrieval and generation are evaluated separately.

Do you need a complete MLOps platform to get started?

No. The level of tooling is proportionate to the number of models, how often they are updated and how critical they are. For a first model, rigorous versioning, simple automated deployment and basic monitoring are enough. Shared components are justified when several teams encounter the same needs.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.