Suppose a packaging printer whose industrial management asks for a real-time data architecture: line stoppages, known today through the next morning’s report, should be known without waiting. The investment committee to which the request is submitted does not have to judge a technology. Its job is to establish which decision would be taken earlier, by whom, and what the difference is worth.
Real time is defined here by a delay: the one separating an event from the decision it informs. A real-time data architecture processes data as it arrives in order to meet this delay when scheduled batch processing can no longer meet it. It is justified use by use, when an identified decision loses value by waiting for the next batch. Elsewhere, a more frequent batch is enough and is simpler to operate. The committee’s questions are dealt with in the order in which they commit expenditure.
Which decision would be taken earlier?
Fresher data is only worth the decision it brings forward. The committee therefore asks, for each candidate stream, who acts, through which action, and what changes if the information arrives earlier.
The answer depends on the recipient. A system that reacts on its own (blocking, alert, replenishment) makes use of every instant gained. A person on shift, a team leader or a maintenance technician, acts within the time their organisation allows. An indicator read at a weekly meeting advances no decision if it is refreshed continuously. In the opening scenario, the line stoppage alert falls into the second case, the monthly overall equipment effectiveness into the third.
A further precaution applies. The most immediate reaction, stopping a machine on a fault, belongs to the controller and the shop-floor supervision system, which the analytics platform need not replace. When events are published by the applications themselves, the subject also goes beyond analytical data: it is an event-driven architecture, whose exchange rules fall under IT architecture.
How long between the event and the decision?
Usage applies “real time” to very different delays. In “Questioning the Lambda Architecture”, an article published on O’Reilly Radar on 2 July 2014, Jay Kreps speaks of a low latency that he places between a few seconds and a few hours. The need is therefore written as a delay, use by use: the maximum time between the event and the moment its recipient can act, the share of events that must meet it, the course of action when it is exceeded.
This delay is a chain, in which the slowest link weighs the most: emission by the source, transport, processing, display, then the time the person takes to see the information and act. Let us assume that the maintenance technician takes ten minutes to reach the line. Whether the alert reaches them in ten seconds or in one minute matters little compared with those ten minutes; an alert available the next day is no longer of use for the intervention. The target is deduced from the human or mechanical link that data will not shorten.
Would a more frequent batch be enough?
Between the overnight batch and the continuous stream, several steps exist, and each adds operating constraints to the previous one:
- a closer-spaced batch, that is, the same processing scheduled every hour or every few minutes;
- a micro-batch, where the stream is cut into small successive batches;
- change data capture, where each modification of a source table becomes an event;
- a continuous stream, where each event is processed on arrival, by an event streaming platform and a stream processing engine.
The step to choose is, in our view, the first one that meets the delay written down in answer to the previous question. This choice of ingestion mode, according to the freshness actually needed, is part of what our data platform offering frames.
The documentation of two stream processing engines, Spark Structured Streaming and Apache Flink, read on 12 November 2026, shows that the boundary is not clear-cut: the first describes an engine that, by default, processes the stream as a series of small batches; the second presents batch as the processing of a bounded, that is finite, stream.
What does a real-time data architecture demand of those who operate it?
A batch that fails overnight is restarted before opening time. A stream has no night: a failure at three in the morning is dealt with at three in the morning, by an on-call rota, or else the organisation accepts in writing that the delay will not be met.
Other burdens stem from the nature of streaming, as described in the documentation of Apache Kafka, an event streaming platform, and Apache Flink, read on the same day. Processing keeps state, for example running aggregates, which Flink saves through checkpoints; recovery requires a source capable of replaying events, which Kafka allows by retaining them for a period set by configuration. The time of an event is not the time it is processed: Flink distinguishes the two and expects events to arrive late. A displayed figure may therefore be revised afterwards, which its readers need to know. Kafka guarantees read order within a partition; across partitions, order has to be handled in the design.
The rest is a matter of day-to-day operation. A format change at the source reaches consumers without delay, hence the value of a data contract per stream. A corrected rule means recalculating the past, a point Kreps’s article stresses. Detecting a situation from several events, the subject of complex event processing, adds its own state to monitor. Finally, computation may run permanently, whether events are flowing in or not.
What becomes of batch processing?
It remains. History is reloaded in batches, a period close is calculated on a frozen state, and some sources only deliver files. The committee is interested in the coexistence, and first of all in whether the same calculation rule will be written twice.
Jay Kreps’s article describes two set-ups. In the lambda architecture, which he attributes to a blog post by Nathan Marz, the same data feeds a batch system and a stream system in parallel, the logic is written in each and the results are merged at read time. Kreps, who mentions in his article that he took part in building Kafka, sees it as a maintenance burden. He proposes keeping only the stream and replaying the retained events when the code changes, a set-up he suggests calling the kappa architecture.
Whatever the set-up, two figures can coexist for the same quantity: the stream figure, immediate and revisable, and the batch figure, consolidated. The committee has it written down which one is authoritative, and for which use.
Which stream should you start with, and when should you stop?
With a single stream: the one whose decision is named, whose recipient is on shift and whose delay is written down. It is built end to end, then measured on real events: observed delay between event and action, share of events outside the delay, on-call interventions, compute consumed. Monitoring its freshness draws on the signals of data observability.
Stopping is a normal outcome. If the closer-spaced batch meets the delay, the project stops there. If the action does not happen earlier even though the information arrives earlier, the obstacle lies in the organisation, and a second stream will not remove it.
In the starting scenario, the request comes back before the committee with the targeted decision, the delay measured on the first stream and the operating burden observed: enough to authorise the next stream, or to close the file.