In a distributed system, each service crossed by a request records its part of the story, and nothing links these fragments as long as no common identifier travels with the request. What is missing is the thread that says that this error in the payment service and this slowness in the gateway belong to the same order.
Distributed systems observability is built by laying this thread, one journey at a time. The method follows an order: choose a journey and map it, propagate a trace context from one service to the next, produce readable traces, attach logs and metrics to them, then decide what to keep. Each step produces a verifiable result and has an owner. The open standard OpenTelemetry and the W3C Trace Context recommendation provide the vocabulary and the formats.
A journey, its map and its owners
Should you instrument, or is the monitoring in place enough? Our article observability vs monitoring deals with this question. Here we start from the decision already taken.
The starting point is a journey whose failure is visible: an order, a transfer, the opening of a case. Its map lists what a request crosses, in order: entry point, services called synchronously, message queues, deferred processes, databases, third-party interfaces. Against each element appear the team responsible for it, the language used and the protocol of each exchange.
This map provides two lists: the boundaries that the context will have to cross, and the components that nobody can instrument (software package, third-party service). The product owner chooses the journey. An architect or the platform team maintains the map.
The trace context must cross every boundary
Distributed tracing relies on propagation: each outgoing call passes to the next service what it needs to attach itself to the same trace. The W3C Trace Context recommendation, in its version of 23 November 2021, standardises this transmission for HTTP by means of two headers. The first, traceparent, brings together a version, the trace identifier, the identifier of the calling operation and a flags field: the only one defined by this version signals that the caller may have recorded the trace. The second, tracestate, carries information specific to each tracing tool.
According to its documentation, OpenTelemetry uses these headers in its default propagator, and instrumentation libraries in principle take care of propagation. The breaks lie elsewhere. The recommendation requires any compliant tool to forward at least the two headers, even when it does not take part in the trace: a proxy or a gateway that removes them cuts the thread. A message queue falls outside the scope of the text, defined for HTTP, which refers to extensions for other protocols. The context is then written into the message metadata on sending and extracted from it on receipt. A deferred process can be connected to the original trace by a link between the two traces, which the OpenTelemetry documentation calls a span link.
At the organisation’s boundaries, the OpenTelemetry documentation calls for caution: a context received from an uncontrolled source may have been forged, and a context sent to a third party reveals internal identifiers.
The result can be verified: a request launched at the entry point carries the same trace identifier in the last service of the journey.
Spans named and described the same way everywhere
A trace is a tree of spans, each representing an operation: a name, a parent, a start and an end, attributes, a status. OpenTelemetry also distinguishes the kind of span (client, server, producer, consumer, internal), which helps the analysis tool assemble the two sides of a call.
Instrumentation without code changes provides the spans of common libraries. It reports, according to the documentation, on what happens at the edges of the application. The inside is added through code: the business steps of the journey and the attributes that will be used for sorting (customer type, channel, version).
Without common rules, these traces cannot be compared from one service to another. The OpenTelemetry semantic conventions propose shared attribute names. Each service also declares its identity in its resource attributes: without configuration, the development kits give the service name the value unknown_service. An error status filled in consistently will then be used for sampling.
This step delivers, for a request, the sequence of its operations with their duration and outcome. The conventions are better provided by the internal platform, following the logic of our DevOps and platform engineering offering.
Logs and metrics attached to the trace
Correlation consists of writing the trace and span identifiers where they are missing. For logs, the OpenTelemetry data model provides these two fields in each record, and its development kits can fill them in automatically when a trace is active. At the time of writing, log support there is stable for some languages and in development for others: it is checked for each service on the map.
For metrics, the link goes through exemplars: a recorded value, with which the identifier of the trace in progress at the time of measurement can be associated. An abnormal point on a response time histogram can thus lead to a trace that illustrates it. The three signals must still carry the same resource attributes (service, version, environment). The metrics of a cluster, detailed in our article on Kubernetes monitoring, are attached through these same attributes.
The outcome is a reading path: from a metric to a trace, from a span to the logs emitted during its execution.
A written sampling rule, with a known cost
Keeping every trace of a very busy journey has a storage and processing cost. The OpenTelemetry documentation describes two families of decision. Head sampling decides as early as possible, for example from the trace identifier and a target proportion: in this form, it is simple and keeps whole traces, but cannot guarantee that traces in error will be retained. Tail sampling decides after seeing all or part of the spans: it makes it possible to keep errors and slow requests, at the price of a stateful component, which stores a large volume and must itself be monitored. The two can be combined.
The same documentation points out that a system that produces few traces can do without sampling. The rule chosen is written down: what is kept for certain, the proportion of the rest, the retention period, and who can change it.
Verifying distributed systems observability on a real request
| Step | What it produces | Who is responsible |
|---|---|---|
| Journey map | The list of boundaries to cross and of components that cannot be instrumented | Product owner, architect |
| Context propagation | The same trace identifier from the entry point to the last service | Each team, for its service |
| Traces | The operations of a request, with duration, outcome and common attributes | Development teams |
| Correlation | Moving from a metric to a trace, then to the logs | Platform team |
| Sampling | A written retention rule and its cost | Platform team, budget owner |
The check is done on real evidence. Take a request that failed yesterday and trace it back from the alert to the trace, then to the logs of the faulty service, noting each place where you had to search by hand. These places point to an uncrossed boundary, an unnamed service or a log without an identifier.
Limits remain. A non-instrumented component only appears through the duration of the call addressed to it. The signals also carry data: the W3C recommendation forbids tracing tools from placing personal or sensitive information in traceparent and tracestate, and the OpenTelemetry documentation advises against putting any in baggage, the key-value pairs propagated with the context.
Once the first journey is readable end to end, service level objectives can rely on its signals: this is the subject of our SRE and observability offering. The next journey reuses the conventions and the collection, and only its own boundaries remain to be instrumented.