Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Cloud and platforms

Distributed systems observability begins with a journey traced end to end

In brief

Distributed systems observability relies on distributed tracing: an identifier propagated from service to service links the operations of a single request, then the logs and metrics that relate to it. Management Square describes here the method for instrumenting a journey end to end, from context propagation to sampling, with what each step produces and who is responsible for it.

Page contents

In a distributed system, each service crossed by a request records its part of the story, and nothing links these fragments as long as no common identifier travels with the request. What is missing is the thread that says that this error in the payment service and this slowness in the gateway belong to the same order.

Distributed systems observability is built by laying this thread, one journey at a time. The method follows an order: choose a journey and map it, propagate a trace context from one service to the next, produce readable traces, attach logs and metrics to them, then decide what to keep. Each step produces a verifiable result and has an owner. The open standard OpenTelemetry and the W3C Trace Context recommendation provide the vocabulary and the formats.

A journey, its map and its owners

Should you instrument, or is the monitoring in place enough? Our article observability vs monitoring deals with this question. Here we start from the decision already taken.

The starting point is a journey whose failure is visible: an order, a transfer, the opening of a case. Its map lists what a request crosses, in order: entry point, services called synchronously, message queues, deferred processes, databases, third-party interfaces. Against each element appear the team responsible for it, the language used and the protocol of each exchange.

This map provides two lists: the boundaries that the context will have to cross, and the components that nobody can instrument (software package, third-party service). The product owner chooses the journey. An architect or the platform team maintains the map.

The trace context must cross every boundary

Distributed tracing relies on propagation: each outgoing call passes to the next service what it needs to attach itself to the same trace. The W3C Trace Context recommendation, in its version of 23 November 2021, standardises this transmission for HTTP by means of two headers. The first, traceparent, brings together a version, the trace identifier, the identifier of the calling operation and a flags field: the only one defined by this version signals that the caller may have recorded the trace. The second, tracestate, carries information specific to each tracing tool.

According to its documentation, OpenTelemetry uses these headers in its default propagator, and instrumentation libraries in principle take care of propagation. The breaks lie elsewhere. The recommendation requires any compliant tool to forward at least the two headers, even when it does not take part in the trace: a proxy or a gateway that removes them cuts the thread. A message queue falls outside the scope of the text, defined for HTTP, which refers to extensions for other protocols. The context is then written into the message metadata on sending and extracted from it on receipt. A deferred process can be connected to the original trace by a link between the two traces, which the OpenTelemetry documentation calls a span link.

At the organisation’s boundaries, the OpenTelemetry documentation calls for caution: a context received from an uncontrolled source may have been forged, and a context sent to a third party reveals internal identifiers.

The result can be verified: a request launched at the entry point carries the same trace identifier in the last service of the journey.

Spans named and described the same way everywhere

A trace is a tree of spans, each representing an operation: a name, a parent, a start and an end, attributes, a status. OpenTelemetry also distinguishes the kind of span (client, server, producer, consumer, internal), which helps the analysis tool assemble the two sides of a call.

Instrumentation without code changes provides the spans of common libraries. It reports, according to the documentation, on what happens at the edges of the application. The inside is added through code: the business steps of the journey and the attributes that will be used for sorting (customer type, channel, version).

Without common rules, these traces cannot be compared from one service to another. The OpenTelemetry semantic conventions propose shared attribute names. Each service also declares its identity in its resource attributes: without configuration, the development kits give the service name the value unknown_service. An error status filled in consistently will then be used for sampling.

This step delivers, for a request, the sequence of its operations with their duration and outcome. The conventions are better provided by the internal platform, following the logic of our DevOps and platform engineering offering.

Logs and metrics attached to the trace

Correlation consists of writing the trace and span identifiers where they are missing. For logs, the OpenTelemetry data model provides these two fields in each record, and its development kits can fill them in automatically when a trace is active. At the time of writing, log support there is stable for some languages and in development for others: it is checked for each service on the map.

For metrics, the link goes through exemplars: a recorded value, with which the identifier of the trace in progress at the time of measurement can be associated. An abnormal point on a response time histogram can thus lead to a trace that illustrates it. The three signals must still carry the same resource attributes (service, version, environment). The metrics of a cluster, detailed in our article on Kubernetes monitoring, are attached through these same attributes.

The outcome is a reading path: from a metric to a trace, from a span to the logs emitted during its execution.

A written sampling rule, with a known cost

Keeping every trace of a very busy journey has a storage and processing cost. The OpenTelemetry documentation describes two families of decision. Head sampling decides as early as possible, for example from the trace identifier and a target proportion: in this form, it is simple and keeps whole traces, but cannot guarantee that traces in error will be retained. Tail sampling decides after seeing all or part of the spans: it makes it possible to keep errors and slow requests, at the price of a stateful component, which stores a large volume and must itself be monitored. The two can be combined.

The same documentation points out that a system that produces few traces can do without sampling. The rule chosen is written down: what is kept for certain, the proportion of the rest, the retention period, and who can change it.

Verifying distributed systems observability on a real request

Step What it produces Who is responsible
Journey map The list of boundaries to cross and of components that cannot be instrumented Product owner, architect
Context propagation The same trace identifier from the entry point to the last service Each team, for its service
Traces The operations of a request, with duration, outcome and common attributes Development teams
Correlation Moving from a metric to a trace, then to the logs Platform team
Sampling A written retention rule and its cost Platform team, budget owner

The check is done on real evidence. Take a request that failed yesterday and trace it back from the alert to the trace, then to the logs of the faulty service, noting each place where you had to search by hand. These places point to an uncrossed boundary, an unnamed service or a log without an identifier.

Limits remain. A non-instrumented component only appears through the duration of the call addressed to it. The signals also carry data: the W3C recommendation forbids tracing tools from placing personal or sensitive information in traceparent and tracestate, and the OpenTelemetry documentation advises against putting any in baggage, the key-value pairs propagated with the context.

Once the first journey is readable end to end, service level objectives can rely on its signals: this is the subject of our SRE and observability offering. The next journey reuses the conventions and the collection, and only its own boundaries remain to be instrumented.

FAQ

Frequently asked questions

Another question? Ask us

What is distributed tracing?

Distributed tracing consists of following a single request through all the services it crosses. Each service records its operations as spans, attached to a common trace identifier passed from call to call. Together they form a tree that shows the order of the operations, their duration and their outcome.

What is the traceparent header for?

The traceparent header is defined by the W3C Trace Context recommendation to pass the trace context in an HTTP request. It contains a version, the trace identifier, the identifier of the calling operation and a flags field, which includes the sampling flag. The called service uses it to attach its own operations to the current trace. A second header, tracestate, carries information specific to each tracing tool.

Should all the traces of a distributed system be kept?

It depends on the volume produced. The OpenTelemetry documentation distinguishes head sampling, decided as early as possible, for example from the trace identifier, and tail sampling, decided after examining the spans, which makes it possible to keep errors and slow requests. The second requires a stateful component, to be operated and monitored. A system that produces few traces can do without sampling.

Is OpenTelemetry enough to make a distributed system observable?

OpenTelemetry provides programming interfaces, software development kits, a protocol and a collector to produce and transport traces, metrics and logs. Its documentation specifies that it is not itself a storage or visualisation tool: these functions fall to other tools. The teams remain responsible for instrumenting the business code, naming conventions and context propagation at the boundaries that libraries do not cover.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.