Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Data

Data platform: architecture, build and operation

Warehouse, lake, lakehouse, data mesh: the right choice depends on your uses, your organisation and your constraints, not on the trend of the moment. Our consultants help your teams design a platform they know how to operate and whose cost remains clear.

Our principles

  • An architecture chosen on your real uses
  • Pipelines that are tested, observed and replayable
  • An operating cost tracked use by use
In brief

A data platform is the set of components that collect, store, transform and expose an organisation's data for management, analysis and AI. Management Square consultants contribute, within your teams, to defining its architecture, building it and organising its operation, starting from the uses to be served rather than from a catalogue of technologies.

Scope

How we contribute

  1. Framing uses and constraints

    Inventory of the uses to be served, the sources, the volumes, the freshness and sensitivity requirements. This framing sets the criteria against which the architecture will be judged.

  2. Target architecture

    Choice of model (warehouse, lake, lakehouse), layering, security, environments and cost assumptions. Each decision is recorded with the options set aside and the reasons why.

  3. Ingestion and integration

    Batch, change data capture or continuous streaming pipelines, according to the real need for freshness. Each source receives a contract describing what it provides.

  4. Modelling and transformation

    Data layers, naming conventions, models intended for analysis and transformations that are versioned, tested and documented like application code.

  5. Operation and observability

    Orchestration, monitoring of freshness and volumes, incident management, backup and cost tracking. A platform that is not operated loses the trust of its users.

  6. Migration of the existing estate

    A path for taking over an ageing warehouse or lake: inventory of what is still used, switchover order, a period of parallel running and decommissioning.

Page contents

A platform delivered is not a platform used

The scenario is familiar. An organisation invests in a modern foundation, loads most of its sources into it, then finds that the business teams carry on working from their own extracts. The data platform exists, it works, and it serves only a handful of users.

Three causes recur. The first is the order of operations: the foundation was built for “all the data”, with no priority use to set its requirements. The second is trust: loads that fail silently or tables without descriptions are enough to send users back to their spreadsheets. The third is operation: a platform delivered as a project, with no team to operate it, deteriorates from the first change to a source application.

A more recent trap adds to these. The vocabulary (lake, lakehouse, data mesh) is often used as an answer, whereas it refers to options that do not solve the same problems.

Warehouse, lake, lakehouse, data mesh: four options that are not mutually exclusive

The first three are technical architectures. The fourth is an organisational model, compatible with each of them.

Option Principle Suitable when Point of attention
Data warehouse Structured data, modelled before being exposed, queried in SQL Management and reporting dominate Poorly suited to unstructured data and machine learning
Data lake Raw files of any format on object storage, structure applied on read Volumes are high and uses exploratory Without a catalogue or rules, the lake becomes a dump that nobody can read
Lakehouse Open table formats (Apache Iceberg, Delta Lake) on top of the lake’s storage, with transactions and schema management The organisation wants to serve analysis and AI without maintaining two systems Requires real engineering skills and discipline in table management
Data mesh Each business domain publishes its data as products, on a self-service platform and under federated governance A central team has become a bottleneck for numerous domains Requires domain teams able to sustain their products over time

The choice is not made on how modern the option is. It is made on four criteria: the nature of the uses (management reporting, exploration, models), the variety of the data, the skills available to operate the whole, and the degree of autonomy of the business domains. Our glossary entries detail the definition of the lakehouse and that of data mesh.

The four layers of a data platform

Whichever option is chosen, the whole is designed as a chain. A weakness in one layer cancels out the quality of the others.

Ingestion. Three modes coexist: batch loading, change data capture in the source databases and continuous event streaming. The right mode depends on the freshness the use really requires. Each source deserves an explicit contract: schema, frequency, owner, expected behaviour when it changes.

Storage and modelling. Common practice separates a raw layer, faithful to the source, a cleansed and conformed layer, and an exposed layer, modelled for the uses. This separation makes it possible to replay processing without calling on the source applications again.

Transformation. Business rules live in code that is versioned, reviewed and automatically tested, not in manual processing or scattered queries. This is the condition for a change to be traceable and reversible.

Exposure. Tables intended for reporting tools, interfaces for applications, datasets for data science teams: each access mode has its permissions, its documentation and its freshness commitments.

Our approach by use slices

We advise against delivering a complete foundation before the first use. Our consultants help your teams build a first vertical slice: one or two sources, the complete chain through to exposure, a real use and its users. This slice tests the architecture, sets the standards and serves as a model for the following ones.

Structuring decisions are taken early and recorded in writing: architecture model, environment layout, permissions management, modelling conventions. Each one mentions the options set aside. This record avoids reopening the same debates every time someone joins the team.

Automation is present from the first slice. Environments are described as infrastructure as code, transformations go through a continuous integration and deployment pipeline, tests run on every change. These are the practices of DevOps and platform engineering, applied to data.

Security and access management are designed together with the platform, based on the rules set by data governance: classification, role-based permissions, masking of sensitive data, access logging.

The mistakes we help you avoid

  • Loading everything before serving a use. An exhaustive inventory of sources delays the first value and produces tables that nobody asked for.
  • Choosing real time by default. A continuous stream is harder to build and operate than batch loading. It must meet a demonstrated need.
  • Reproducing the old warehouse identically. A migration is an opportunity to remove what is no longer used. Taking over all existing processing carries the debt into the new foundation.
  • Adopting data mesh without domain teams. Decentralising responsibility to business teams that have neither the time nor the skills moves the bottleneck instead of removing it.
  • Discovering the cost after the fact. Without allocating spend by use, the bill grows with no possible trade-off. Our consultants help put this tracking in place from the start, following the principles of FinOps.

The signs of a platform that works

At the framing stage, we agree with you on a few observable indicators, tracked throughout the engagement.

  • Usage: number of users and processes that are genuinely active, share of exposed tables that are actually consulted.
  • Reliability: share of loads completed within the agreed time, time to detect and resolve a data incident.
  • Delivery speed: time between the request for a new source or a new dataset and its availability.
  • Cost clarity: share of spend attached to an identified use or team.
  • Autonomy: the ability of your teams to add a source and fix an incident without our help.

The last indicator is the one that matters most at the end of the engagement. A platform that only the team that built it can evolve remains a project. It becomes a service when other teams use it, understand it and evolve it.

Typical work

What our consultants contribute to

Architecture file

Component diagram, flows, security rules and a record of architecture decisions with their justifications.

Foundation described as code

Environments, permissions and components described as infrastructure as code, rebuildable and auditable.

Development standards

Modelling conventions, pipeline templates, testing and review rules applicable by all teams.

Operations file

Monitoring, incident recovery and cost tracking procedures, maintained by the team that operates the platform.

Management Square in figures

300+
projects completed
43+
clients supported
Since2015
serving our clients
  • Allianz
  • Euler Hermes
  • Groupe BPCE
  • ODDO BHF
  • Candriam
  • Ledger
  • Saur
  • Veolia
  • Voltalia
  • emeis
  • Bonpoint
  • Legal Suite

FAQ

Frequently asked questions

Another question? Ask us

What is the difference between a data warehouse, a data lake and a lakehouse?

The warehouse stores structured, modelled data, queried in SQL for management reporting. The lake keeps raw files of any format, useful for exploration and machine learning. The lakehouse adds open table formats to the lake's storage, bringing transactions and schema management, so as to serve analysis and AI on the same data.

Does data mesh replace the data platform?

No. Data mesh is an organisational model, not a technology: it gives each business domain responsibility for publishing its data as products. On the contrary, it requires a solid self-service platform on which the domains rely, and governance shared by all.

Should the entire existing platform be migrated in one go?

Rarely. A complete switchover concentrates the risks and ties up the teams without delivering anything new. We prefer a migration by domain or by use, with a period of parallel running during which the results of the two systems are compared before the old one is decommissioned.

Batch or real-time loading: how do you choose?

By starting from the decision the data must inform. A report consulted every morning does not need a continuous stream, whereas fraud detection or equipment monitoring does. Continuous streaming costs more to build and to operate: it is justified use by use, not on principle.

How do you control the cost of a data platform?

By making the cost visible before trying to reduce it: tagging resources, allocating compute and storage spend to each team or use, alerts on drift. Then come the technical levers, such as stopping unused processing, sizing compute capacity and data retention rules.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.