Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Artificial intelligence

Generative AI in the enterprise: reliable business assistants, all the way to production

A prototype that impresses in a meeting is not a service. Our consultants help your teams build business assistants whose quality is measured, whose answers cite their sources and whose cost is known.

Our principles

  • A test set built before the system
  • Sourced answers and deliberate refusals
  • Operations planned from the design stage
In brief

Generative AI in the enterprise mainly covers systems based on a large language model (LLM) that draft, summarise, extract or answer questions from internal documents. Management Square consultants help your teams design these business assistants, evaluate them on a test set built with domain experts and put them into production while keeping security, cost and operations under control.

Scope

How we contribute

  1. Framing the assistant

    Definition of the task, the audience, the corpus and the limits: what the assistant must do, what it must refuse and the criterion that will say whether it is ready to be deployed.

  2. Preparing the document corpus

    Selection of sources, handling of duplicates and outdated versions, extraction of tables and diagrams, metadata and carrying over of access rights.

  3. Architecture design

    Choice between prompting alone, retrieval-augmented generation and fine-tuning the model, then selection of the language model based on your tasks and your confidentiality constraints.

  4. Evaluation framework

    A set of reference questions built with your experts, measurement of retrieval and of answer faithfulness, regression tests rerun at every change.

  5. Security

    Enforcement of access rights in the index, protection against prompt injection, filtering of inputs and outputs, logging of exchanges.

  6. Industrialisation and operations

    Integration into work tools, tracking of cost and quality, management of model version changes, a user feedback loop.

Page contents

From a convincing prototype to a disappointing service

Generative AI in the enterprise almost always begins with a successful demonstration. A prototype answers questions about a set of documents, and the audience is convinced. The difficulties come later, when real users ask real questions.

The prototype answered well because the questions were chosen by those who had built it. In real use, requests are ambiguous, incomplete, sometimes out of scope. The system answers anyway, with the same confidence, including when it is wrong. A user who spots two serious errors stops trusting the tool, then stops using it.

Then there are the questions the demonstration had left aside. Does the assistant respect access rights to documents? What becomes of an answer when the source document changes? What does a request cost, and what will a whole department’s daily use cost? Who is alerted if quality deteriorates after a change of model version? These are engineering questions, not model questions.

LLM, RAG, fine-tuning: the options and what they involve

A large language model, or LLM, produces text from the patterns learnt during its training. It does not know your internal documents, it is unaware of facts after that training, and it can produce a plausible but false statement. The architecture of a business assistant consists of compensating for these three limitations.

Approach Principle Suitable for Limitation
Prompting alone The model receives an instruction and, if needed, the text to process Rewording, summarising, extraction from a supplied document No knowledge specific to the organisation
Retrieval-augmented generation (RAG) The system retrieves the relevant passages from your document base and passes them to the model Questions about an internal corpus, sourced answers The answer is no better than the passages retrieved
Fine-tuning the model The model is retrained on your examples Highly specialised format, vocabulary or task Maintenance burden, work to redo when the base model evolves

RAG is often the starting point for a document assistant, because it makes it possible to cite sources and update knowledge without touching the model. Its quality is decided before generation: document chunking, metadata, version management, combining keyword search with semantic search, reranking of results.

The choice of model comes afterwards. It is made on your tasks, not on a general ranking: quality measured on your own questions, place of data processing, cost per request, response time, the possibility of changing provider. A model operated by a provider and an open model you host yourselves entail neither the same confidentiality constraints nor the same operating burden.

Our approach to generative AI in the enterprise: starting from evaluation

Our consultants work within your teams; decisions remain yours.

Defining the task and its limits. An assistant ‘that answers everything’ cannot be evaluated. We start from a precise task, an audience and a bounded corpus. We also write down what the assistant must refuse: an out-of-scope question, a request for which no source exists.

Building the reference set before the system. With the domain experts, we put together a set of real questions and expected answers, trick cases included. This set serves as the acceptance criterion. It separates two measurements: does the system retrieve the right passage, and is the answer faithful to that passage?

Preparing the corpus. Duplicates, outdated versions, contradictory documents, poorly extracted tables and diagrams: document preparation often weighs more on the result than the choice of model. It also raises the question of who owns each source and how often it is updated.

Treating security as a functional requirement. Document access rights must be carried through into the index: a user must not obtain through the assistant what they are not entitled to read. Indexed content may also contain malicious instructions (prompt injection), which makes it necessary to filter inputs and to limit what the system can do.

Industrialising. Logging of exchanges, tracking of cost and quality, regression tests rerun at every change of prompt, index or model, collection of user feedback. These practices extend those of MLOps, adapted to language models.

When the need goes beyond answering a question and involves chaining actions in your applications, the subject becomes that of AI agents, with additional control requirements.

Common mistakes in generative AI projects

  • Choosing the model before defining the task. Comparing models only makes sense on your questions. Without a reference set, it boils down to an impression.
  • Indexing the entire document estate at once. A broad, unsorted corpus lowers the precision of answers. A narrow, clean scope gives better results, and is then widened.
  • Evaluating by demonstration. A few questions asked in a meeting say nothing about behaviour across all real requests. Only a reference set rerun systematically makes it possible to compare two versions.
  • Not providing for refusal. An assistant that cannot say ‘I cannot find this information’ invents. A reasoned refusal is a feature.
  • Ignoring cost at scale. The length of the context passed to the model and the number of calls per request determine the bill. They can be measured from the pilot onwards.
  • Delivering without an operator. An assistant whose corpus, evaluations and changes no one maintains deteriorates without anyone noticing.

How to measure the success of a business assistant

We track four families of indicators, set at framing and recorded both before and after going into service.

  • Quality. Share of answers judged correct and faithful to the sources on the reference set, tracked version after version. Justified refusals and wrongful refusals.
  • Usage. Share of target users who come back once the discovery phase is over, frequency of rephrasing, negative feedback reported.
  • Operations. Cost per request, response time, confidentiality incidents, time taken to reflect an updated document.
  • Business effect. The indicator of the process concerned, chosen with its owner: time to process a case, rework rate, workload of second-level support.

A heavily used assistant whose business indicator does not move raises a question of process design, not of technology. It is at this stage that work with teams on their practices takes over from engineering.

Typical work

What our consultants contribute to

Architecture file

Components selected, data flows, the chosen model and the reasons for the choice, reversibility conditions.

Reference test set

Real questions, expected answers and trick cases, with the results measured for each version of the system.

Assistant brought into service

An application integrated into users' working environment, with its safeguards and logging.

Operations file

Indicators tracked, procedure for updating the corpus, procedure for changing model, roles of the team responsible.

Management Square in figures

300+
projects completed
43+
clients supported
Since2015
serving our clients
  • Allianz
  • Euler Hermes
  • Groupe BPCE
  • ODDO BHF
  • Candriam
  • Ledger
  • Saur
  • Veolia
  • Voltalia
  • emeis
  • Bonpoint
  • Legal Suite

FAQ

Frequently asked questions

Another question? Ask us

What is the difference between an LLM and a RAG system?

An LLM is a language model that generates text based on what it learnt during training. A RAG system (retrieval-augmented generation) surrounds this model with a document search engine: it retrieves the relevant passages from a document base and passes them to the model so that it answers from them. The LLM is a component; RAG is an architecture.

Do you need to train your own model to deploy generative AI in the enterprise?

Rarely. Most document-related needs can be handled with an existing model that is given the right passages at the time of the question. Fine-tuning a model on examples specific to the organisation is justified for a highly specialised task or a strict output format, and it creates a maintenance burden at every change of the base model.

How do you limit the hallucinations of a generative AI assistant?

You do not eliminate them; you reduce them and make them detectable. The levers are well known: give the model relevant source passages, require it to cite these sources, provide for an explicit refusal when information is missing, and measure answer faithfulness on a test set rerun at every version. For sensitive uses, human review remains necessary.

Does our data remain confidential with an external language model?

It depends on the deployment mode and the contract. The points to check are any reuse of data by the provider, the place of processing, the retention period and the logs. Depending on the sensitivity of the information, the answer may be a service with contractual commitments, hosting in a cloud that meets your sovereignty requirements or an open model operated by your teams.

What determines the running cost of an assistant?

The cost depends mainly on the volume of text sent to and received from the model with each request, the number of calls needed to produce an answer, the chosen model and the number of requests. Added to this are the document search infrastructure, updating the corpus and the work of continuous evaluation. These parameters can be measured from the pilot onwards, which makes it possible to estimate the cost before rollout.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.