Definition of LLMOps: a term of common usage
LLMOps, for large language model operations, refers to the practices and tools used to put into production, evaluate, monitor and evolve an application built on a large language model (LLM): an assistant, a document search system, an agent.
The word is formed on the model of MLOps. It is one of common usage, and its scope varies from one source to another. This entry takes the meaning of an extension of MLOps, organised around four subjects: prompts, evaluation, monitoring in production and costs.
What makes operating an LLM-based application distinctive
The behaviour of such an application does not depend on its code alone. It depends on the prompt, the version of the model called, the generation parameters and, where there are any, the documents provided to the model. A change to one of these elements can change the answers without a single line of code having moved.
Three other traits are added. The output is free text, which can vary from one call to another for the same request. When the model is provided by a third party, its versions evolve according to a schedule the organisation does not set. Finally, when access to the model is billed on usage, the cost grows with the volume of text read and produced, counted in tokens.
Prompts, evaluation, monitoring, costs: the four components
| Component | What it organises |
|---|---|
| Prompt management | Versioning of instructions and of call configuration (model version, parameters), their review and their release like code |
| Evaluation | A set of reference cases, with expected results fixed in advance, submitted to the application before each change of prompt, model or corpus, then a sample of real exchanges reviewed after release |
| Monitoring in production | Traces of each exchange: request, passages provided, answer, latency, number of tokens, user feedback, guardrail triggered |
| Cost control | Measuring cost per request and per use, and the levers that reduce it: context length, number of calls per answer, choice of model according to the task |
A trace contains the text of the request, which may include personal data: what it retains, and for how long, must be decided before release.
LLMOps, MLOps, prompt engineering, RAG: not to be confused
MLOps concerns models that the organisation trains, and our entry on MLOps describes their lifecycle. LLMOps takes up its principles (versioning, testing, monitoring) and shifts their object. Training leaves the cycle when the model is used as is. The model registry becomes a registry of prompts and configurations. Measuring performance on labelled data gives way to assessing free text.
The two meet when a language model is fine-tuned on examples specific to the organisation: a trained artefact returns to the pipeline. Our article on LLM fine-tuning sets out this case.
Prompt engineering is the art of writing an instruction. LLMOps deals with what surrounds it: version, test, deployment, monitoring. Finally, RAG is an application architecture, whose document index becomes one more object to version and evaluate.
Example: the announced withdrawal of a model version
Take an assistant that answers employees’ questions on leave rules, based on the company’s agreements. The model provider announces the withdrawal of the version in use. The team replays the set of reference cases on the next version. The answers remain accurate, but the assistant now answers out-of-scope questions it previously refused.
The prompt is corrected, recorded under a new version, then re-evaluated. The cost per request is compared between the two configurations. The new version is opened to a portion of users, and the traces are reviewed before general rollout. The old configuration remains available until the withdrawal.
Setting up this pipeline is the subject of our page on MLOps and LLMOps. The design of the assistant itself (task, corpus, architecture) falls under generative AI in the enterprise.