When a pilot fails to reach production, the post-mortem almost always shows the same thing: technically it all worked. What was missing was a person who decides what counts as good quality, and a person whose own numbers improve when it launches. Three engineers without those two roles build an interesting prototype and then stop.

The minimum team

  • A process owner on the business side. Accountable for the task being needed and for the way of working changing after launch.
  • A domain expert. Labels reference answers and settles disputed cases. Not a one-off effort — several hours a week.
  • An AI engineer. Retrieval, prompts, tools, quality evaluation.
  • An integration developer. Data, access, embedding into the working system.
  • An analyst or product manager. Owns the metrics and translates them into business language.

On a small project roles combine: one person can cover engineering and integration. Only one combination is forbidden — the expert and the developer cannot be the same person, or quality evaluation turns into marking your own homework.

The three roles teams forget

The first is a data owner. Someone must have the authority to say “this version of the policy is current, delete those three”. Without them the team argues for months about why the system answers from old documents.

The second is an owner of quality. Not a UI tester, but someone who maintains a set of reference questions and runs it before every change. Without that set, any prompt improvement is a lottery: something got better, something quietly broke.

The third is security and legal, involved at the start rather than a week before launch. Questions about where data lives and what is logged are answered by design. Answered retroactively, they are answered by moving the deadline.

If the expert cannot find time for labelling, the task is not a priority. It is the most honest test of whether the project is needed.

How to work: rhythm beats methodology

The working rhythm for a pilot is weekly iterations with one rule: every week you show a result on real data, not a slide about work done. The process owner and the expert attend the demo. Every two weeks you run the full set of reference questions and record the results in a table where the trend is visible.

Handover to operations

Production starts not with a deploy but with answers to three questions: who fixes it when the system answers wrongly, who updates the knowledge base, and who decides when the model should be replaced. Until those names exist, the project is still a pilot — even if real users depend on it.