Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Cloud and platforms

Generative AI observability adds cost, answer quality and drift to latency

In brief

Generative AI observability refers to the measures taken on an application based on a language model in production: latency, cost per request, evaluated answer quality, tool call failure rate, drift. Management Square sets out here how to establish each measure, the reader it serves and the blind spot it leaves.

Page contents

Nothing in the return code of a call to a language model says whether the answer is correct. A generative AI application can respond on time, without a technical error, and be wrong. It can also cost more from one week to the next, or change behaviour without a single line of code having changed.

Generative AI observability therefore adds five measures to an application’s traces, metrics and logs: latency, broken down by operation; cost per request; answer quality, which is evaluated; the tool call failure rate; drift. Each one is related below to its calculation method, its reader and its blind spot.

The measure names cited come from the semantic conventions for generative AI of the OpenTelemetry project, read on 12 November 2026 in the semantic-conventions-genai repository, to which the opentelemetry.io site points. They have “Development” status there: these names may still change.

Latency and cost per request are read in the trace

A request addressed to such an application breaks down into operations: passage retrieval, one or more calls to the model, tool calls. The trace keeps the duration of each.

Latency is measured end to end, then by operation. For a model call, the conventions distinguish the total duration (gen_ai.client.inference.duration) and, when the answer is streamed as it is produced, the time to the first chunk (gen_ai.client.inference.time_to_first_chunk). It is read in percentiles rather than as an average, as our entry on SLO, SLI and SLA points out. The product owner reads in it the user’s wait, the operations team the operation that causes it. Latency says nothing about correctness, and compares poorly between answers of different lengths: a brief refusal is fast.

Cost per request is obtained, for a model billed on usage, by multiplying the input tokens and the output tokens by their respective prices, then adding up all the calls of the request. Suppose a request that triggers three calls, each reading 4,000 tokens and producing 500: it consumes 12,000 input tokens and 1,500 output tokens, to be valued at the provider’s rate. The conventions provide separate counters for input tokens, output tokens and those served from a cache.

Tracking tools perform this calculation. Langfuse’s documentation states that the cost is either passed with the model’s response or derived from the tokens and a price list per model; MLflow’s, that its traces record latency and token usage at each step (pages consulted on the same day). This measure serves the person who holds the application’s budget. It ignores human review time, and its average can mask long requests, where an agent chains calls.

Answer quality goes through an evaluation

Quality cannot be read from a duration or a counter. It is scored, according to written criteria: accuracy, faithfulness to sources, compliance with refusal instructions. Three ways of scoring coexist, each with its limit.

Mode How the measure is established Its limit
Reference set Requests with an expected result set in advance, replayed at each change; the share of compliant cases is recorded It says nothing about the requests it does not contain
Scoring by people A sample of real exchanges, reviewed by domain experts with a rubric It only holds for the sample, and two reviewers may diverge
Scoring by a model A second model, known as a judge, scores each answer against a rubric The judge’s agreement with the experts remains unknown until it has been measured

Langfuse’s documentation presents this last mode as using a model to assess the outputs of another application, and recommends checking the judge against annotated examples before relying on it. Those of MLflow and Phoenix (Arize) also place automated judges alongside other modes: evaluation sets and human feedback for the former, evaluators written in code for the latter. The OpenTelemetry conventions provide for an event, gen_ai.evaluation.result, which links a score to the evaluated operation.

A quality score remains an assessment. It depends on the rubric, the sample and, for a judge, the model and the prompt that make it work. It serves the product owner and the business experts, who use it to decide on a go-live. For a documentary assistant, retrieval and drafting are scored separately, as described on our MLOps and LLMOps page.

Failed tool calls and drift over time

The tool call failure rate relates, for each tool and over a period, the calls ending in error to the calls issued. The conventions describe one span per tool execution (gen_ai.execute_tool), whose error.type attribute is filled in when the operation ends in error. The calculation convention remains to be written: do you count invalid arguments produced by the model, empty results, retries? The team that maintains the tool and the team that designs the agent read this rate together. A successful call to the wrong tool, or with valid but incorrect arguments, does not appear in it.

Drift here refers to the shift of a measure without any code change. Our MLOps and LLMOps page identifies three sources: users’ requests, the documentary content, the version of the supplied model. It is tracked by replaying the reference set at a fixed interval and at each version change, and by recording the distribution of real requests: topics, length, share of refusals. Our LLMOps entry gives an example, the announced retirement of a model version. The application owner finds in it the signal to investigate. The measure observes a gap without giving its cause, and a stable score on an ageing reference set says nothing about new requests.

Generative AI observability may retain users’ words

The traces of such an application may contain the requests and answers in clear text. The OpenTelemetry conventions class input and output messages among the optional attributes (“Opt-In”) and flag that they are likely to contain sensitive information. Whether or not to capture this content, masking it, setting its retention period and its readers: these choices come before collection. When personal data appear in it, they are made with the data protection officer, in light of the articles read in our article GDPR and artificial intelligence.

This article gives no threshold: an objective is set for each application, from a baseline survey, according to the method on our SRE and observability page. Latency and, by convention, the share of answers scored as compliant can become service level indicators there, with their alerts. The design of the assistant itself falls under generative AI in business.

None of these five measures can be interpreted alone. A falling cost per request may come from shorter answers, and therefore a quality to be rechecked; a rising latency, from tool calls that fail and then retry. Finally, each quality survey carries the version of the model, the prompt and the index on which it was established: after a change, it describes the application as it was before.

FAQ

Frequently asked questions

Another question? Ask us

What is the difference between generative AI observability and that of a conventional application?

The starting signals are the same: traces, metrics and logs. A generative AI application adds measures that these signals do not carry by themselves: the cost of each request, counted in tokens, the quality of the answers, which requires an evaluation, the tool call failure rate and the drift of behaviour without any code change. An answer can indeed arrive on time, without a technical error, and be wrong.

How do you calculate the cost per request of a generative AI application?

For a model billed on usage, the cost of a call is the sum of the input tokens multiplied by their price and the output tokens multiplied by theirs. The cost of a request adds up all the calls it triggers: a request that causes three calls of 4,000 input tokens and 500 output tokens consumes 12,000 input tokens and 1,500 output tokens. The chosen convention specifies whether it includes evaluation calls, embedding computation and retries.

Can the evaluation of answer quality be entrusted to another model?

Yes, provided you know its limit. A model used as a judge scores each answer against a rubric, which makes it possible to score a larger volume of exchanges than human review. Its score depends on the rubric, the prompt and the model that make it work: it is checked against examples annotated by experts, and again when the judge model changes. It complements a reference set and human review of a sample.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.