Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Data

Under what conditions is a real-time data architecture worth its cost?

In brief

A real-time data architecture processes data as it arrives in order to meet a delay, the one separating an event from the decision it informs, when batch processing can no longer meet it. Management Square reviews the questions of an investment committee: which decision, which delay, which step between batch and stream, which operating burden, which place for batch.

Page contents

Suppose a packaging printer whose industrial management asks for a real-time data architecture: line stoppages, known today through the next morning’s report, should be known without waiting. The investment committee to which the request is submitted does not have to judge a technology. Its job is to establish which decision would be taken earlier, by whom, and what the difference is worth.

Real time is defined here by a delay: the one separating an event from the decision it informs. A real-time data architecture processes data as it arrives in order to meet this delay when scheduled batch processing can no longer meet it. It is justified use by use, when an identified decision loses value by waiting for the next batch. Elsewhere, a more frequent batch is enough and is simpler to operate. The committee’s questions are dealt with in the order in which they commit expenditure.

Which decision would be taken earlier?

Fresher data is only worth the decision it brings forward. The committee therefore asks, for each candidate stream, who acts, through which action, and what changes if the information arrives earlier.

The answer depends on the recipient. A system that reacts on its own (blocking, alert, replenishment) makes use of every instant gained. A person on shift, a team leader or a maintenance technician, acts within the time their organisation allows. An indicator read at a weekly meeting advances no decision if it is refreshed continuously. In the opening scenario, the line stoppage alert falls into the second case, the monthly overall equipment effectiveness into the third.

A further precaution applies. The most immediate reaction, stopping a machine on a fault, belongs to the controller and the shop-floor supervision system, which the analytics platform need not replace. When events are published by the applications themselves, the subject also goes beyond analytical data: it is an event-driven architecture, whose exchange rules fall under IT architecture.

How long between the event and the decision?

Usage applies “real time” to very different delays. In “Questioning the Lambda Architecture”, an article published on O’Reilly Radar on 2 July 2014, Jay Kreps speaks of a low latency that he places between a few seconds and a few hours. The need is therefore written as a delay, use by use: the maximum time between the event and the moment its recipient can act, the share of events that must meet it, the course of action when it is exceeded.

This delay is a chain, in which the slowest link weighs the most: emission by the source, transport, processing, display, then the time the person takes to see the information and act. Let us assume that the maintenance technician takes ten minutes to reach the line. Whether the alert reaches them in ten seconds or in one minute matters little compared with those ten minutes; an alert available the next day is no longer of use for the intervention. The target is deduced from the human or mechanical link that data will not shorten.

Would a more frequent batch be enough?

Between the overnight batch and the continuous stream, several steps exist, and each adds operating constraints to the previous one:

  • a closer-spaced batch, that is, the same processing scheduled every hour or every few minutes;
  • a micro-batch, where the stream is cut into small successive batches;
  • change data capture, where each modification of a source table becomes an event;
  • a continuous stream, where each event is processed on arrival, by an event streaming platform and a stream processing engine.

The step to choose is, in our view, the first one that meets the delay written down in answer to the previous question. This choice of ingestion mode, according to the freshness actually needed, is part of what our data platform offering frames.

The documentation of two stream processing engines, Spark Structured Streaming and Apache Flink, read on 12 November 2026, shows that the boundary is not clear-cut: the first describes an engine that, by default, processes the stream as a series of small batches; the second presents batch as the processing of a bounded, that is finite, stream.

What does a real-time data architecture demand of those who operate it?

A batch that fails overnight is restarted before opening time. A stream has no night: a failure at three in the morning is dealt with at three in the morning, by an on-call rota, or else the organisation accepts in writing that the delay will not be met.

Other burdens stem from the nature of streaming, as described in the documentation of Apache Kafka, an event streaming platform, and Apache Flink, read on the same day. Processing keeps state, for example running aggregates, which Flink saves through checkpoints; recovery requires a source capable of replaying events, which Kafka allows by retaining them for a period set by configuration. The time of an event is not the time it is processed: Flink distinguishes the two and expects events to arrive late. A displayed figure may therefore be revised afterwards, which its readers need to know. Kafka guarantees read order within a partition; across partitions, order has to be handled in the design.

The rest is a matter of day-to-day operation. A format change at the source reaches consumers without delay, hence the value of a data contract per stream. A corrected rule means recalculating the past, a point Kreps’s article stresses. Detecting a situation from several events, the subject of complex event processing, adds its own state to monitor. Finally, computation may run permanently, whether events are flowing in or not.

What becomes of batch processing?

It remains. History is reloaded in batches, a period close is calculated on a frozen state, and some sources only deliver files. The committee is interested in the coexistence, and first of all in whether the same calculation rule will be written twice.

Jay Kreps’s article describes two set-ups. In the lambda architecture, which he attributes to a blog post by Nathan Marz, the same data feeds a batch system and a stream system in parallel, the logic is written in each and the results are merged at read time. Kreps, who mentions in his article that he took part in building Kafka, sees it as a maintenance burden. He proposes keeping only the stream and replaying the retained events when the code changes, a set-up he suggests calling the kappa architecture.

Whatever the set-up, two figures can coexist for the same quantity: the stream figure, immediate and revisable, and the batch figure, consolidated. The committee has it written down which one is authoritative, and for which use.

Which stream should you start with, and when should you stop?

With a single stream: the one whose decision is named, whose recipient is on shift and whose delay is written down. It is built end to end, then measured on real events: observed delay between event and action, share of events outside the delay, on-call interventions, compute consumed. Monitoring its freshness draws on the signals of data observability.

Stopping is a normal outcome. If the closer-spaced batch meets the delay, the project stops there. If the action does not happen earlier even though the information arrives earlier, the obstacle lies in the organisation, and a second stream will not remove it.

In the starting scenario, the request comes back before the committee with the targeted decision, the delay measured on the first stream and the operating burden observed: enough to authorise the next stream, or to close the file.

FAQ

Frequently asked questions

Another question? Ask us

What does real time mean for data?

For data, we define "real time" by the delay between an event and the moment its recipient can act, with no general threshold. This delay depends on the use: that of an alert sent to a technician is not that of a monthly indicator. The need is therefore written down use by use, with a maximum delay, the share of events that must meet it and the course of action when it is exceeded.

What is the difference between batch processing and stream processing?

Batch processing applies to a finite set of data, at a scheduled time, and can wait until it has received everything before producing a result. Stream processing applies to events that arrive endlessly and produces its results as it goes. Streaming additionally requires managing state, late-arriving events and uninterrupted operation.

What is the lambda architecture, and what is the kappa architecture?

In an article of 2 July 2014 published on O'Reilly Radar, Jay Kreps describes the lambda architecture, which he attributes to Nathan Marz: the same data feeds a batch system and a stream system in parallel, and the computation logic is written in each. He proposes an alternative, which he suggests calling the kappa architecture: a single stream system, which replays the retained events when the code changes.

Does a real-time architecture replace the data warehouse?

No. Streaming serves decisions that cannot wait for the next batch. The data warehouse, or the lakehouse, keeps the history, the consolidated figures and the analyses covering long periods. The two coexist, and the organisation writes down, for each quantity calculated on both sides, which of the two figures is authoritative.

Does a dashboard need real-time data?

Only if someone watches it continuously and can act on what they read there, as on a shop-floor screen or in a control room. A dashboard consulted in a meeting, every week or every month, advances no decision if it is refreshed continuously. A batch scheduled at the frequency of consultation is enough and is simpler to operate.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.