Nothing in the return code of a call to a language model says whether the answer is correct. A generative AI application can respond on time, without a technical error, and be wrong. It can also cost more from one week to the next, or change behaviour without a single line of code having changed.
Generative AI observability therefore adds five measures to an application’s traces, metrics and logs: latency, broken down by operation; cost per request; answer quality, which is evaluated; the tool call failure rate; drift. Each one is related below to its calculation method, its reader and its blind spot.
The measure names cited come from the semantic conventions for generative AI of the OpenTelemetry project, read on 12 November 2026 in the semantic-conventions-genai repository, to which the opentelemetry.io site points. They have “Development” status there: these names may still change.
Latency and cost per request are read in the trace
A request addressed to such an application breaks down into operations: passage retrieval, one or more calls to the model, tool calls. The trace keeps the duration of each.
Latency is measured end to end, then by operation. For a model call, the conventions distinguish the total duration (gen_ai.client.inference.duration) and, when the answer is streamed as it is produced, the time to the first chunk (gen_ai.client.inference.time_to_first_chunk). It is read in percentiles rather than as an average, as our entry on SLO, SLI and SLA points out. The product owner reads in it the user’s wait, the operations team the operation that causes it. Latency says nothing about correctness, and compares poorly between answers of different lengths: a brief refusal is fast.
Cost per request is obtained, for a model billed on usage, by multiplying the input tokens and the output tokens by their respective prices, then adding up all the calls of the request. Suppose a request that triggers three calls, each reading 4,000 tokens and producing 500: it consumes 12,000 input tokens and 1,500 output tokens, to be valued at the provider’s rate. The conventions provide separate counters for input tokens, output tokens and those served from a cache.
Tracking tools perform this calculation. Langfuse’s documentation states that the cost is either passed with the model’s response or derived from the tokens and a price list per model; MLflow’s, that its traces record latency and token usage at each step (pages consulted on the same day). This measure serves the person who holds the application’s budget. It ignores human review time, and its average can mask long requests, where an agent chains calls.
Answer quality goes through an evaluation
Quality cannot be read from a duration or a counter. It is scored, according to written criteria: accuracy, faithfulness to sources, compliance with refusal instructions. Three ways of scoring coexist, each with its limit.
| Mode | How the measure is established | Its limit |
|---|---|---|
| Reference set | Requests with an expected result set in advance, replayed at each change; the share of compliant cases is recorded | It says nothing about the requests it does not contain |
| Scoring by people | A sample of real exchanges, reviewed by domain experts with a rubric | It only holds for the sample, and two reviewers may diverge |
| Scoring by a model | A second model, known as a judge, scores each answer against a rubric | The judge’s agreement with the experts remains unknown until it has been measured |
Langfuse’s documentation presents this last mode as using a model to assess the outputs of another application, and recommends checking the judge against annotated examples before relying on it. Those of MLflow and Phoenix (Arize) also place automated judges alongside other modes: evaluation sets and human feedback for the former, evaluators written in code for the latter. The OpenTelemetry conventions provide for an event, gen_ai.evaluation.result, which links a score to the evaluated operation.
A quality score remains an assessment. It depends on the rubric, the sample and, for a judge, the model and the prompt that make it work. It serves the product owner and the business experts, who use it to decide on a go-live. For a documentary assistant, retrieval and drafting are scored separately, as described on our MLOps and LLMOps page.
Failed tool calls and drift over time
The tool call failure rate relates, for each tool and over a period, the calls ending in error to the calls issued. The conventions describe one span per tool execution (gen_ai.execute_tool), whose error.type attribute is filled in when the operation ends in error. The calculation convention remains to be written: do you count invalid arguments produced by the model, empty results, retries? The team that maintains the tool and the team that designs the agent read this rate together. A successful call to the wrong tool, or with valid but incorrect arguments, does not appear in it.
Drift here refers to the shift of a measure without any code change. Our MLOps and LLMOps page identifies three sources: users’ requests, the documentary content, the version of the supplied model. It is tracked by replaying the reference set at a fixed interval and at each version change, and by recording the distribution of real requests: topics, length, share of refusals. Our LLMOps entry gives an example, the announced retirement of a model version. The application owner finds in it the signal to investigate. The measure observes a gap without giving its cause, and a stable score on an ageing reference set says nothing about new requests.
Generative AI observability may retain users’ words
The traces of such an application may contain the requests and answers in clear text. The OpenTelemetry conventions class input and output messages among the optional attributes (“Opt-In”) and flag that they are likely to contain sensitive information. Whether or not to capture this content, masking it, setting its retention period and its readers: these choices come before collection. When personal data appear in it, they are made with the data protection officer, in light of the articles read in our article GDPR and artificial intelligence.
This article gives no threshold: an objective is set for each application, from a baseline survey, according to the method on our SRE and observability page. Latency and, by convention, the share of answers scored as compliant can become service level indicators there, with their alerts. The design of the assistant itself falls under generative AI in business.
None of these five measures can be interpreted alone. A falling cost per request may come from shorter answers, and therefore a quality to be rechecked; a rising latency, from tool calls that fail and then retry. Finally, each quality survey carries the version of the model, the prompt and the index on which it was established: after a change, it describes the application as it was before.