A pilot is run on a curated dataset, with a friendly set of test questions, by people who want it to succeed. Production is run on the real estate, with the questions people actually ask, by people who will stop using it the first time it is confidently wrong. The gap between those two conditions is where most enterprise AI initiatives end.
In our experience the failure is rarely the model. It is almost always one of four things underneath it.
The content was never governed
Retrieval surfaces whatever is in the source. If the policy library contains three versions of a procedure and nobody knows which is current, the assistant will answer from whichever one scores best on similarity. It will do so fluently, with a citation, and it will be wrong.
This is not a retrieval problem to be tuned away. It is a content lifecycle problem, and it has to be fixed in the source systems before the AI layer is built on top of them.
Entitlements were applied after retrieval, not during it
Filtering a model output after generation is not access control. By that point the restricted content has already entered the context window, and a sufficiently determined prompt will extract it.
The governing principle is simple to state and awkward to retrofit: a user must never be able to obtain through AI something they could not access directly. That means the retrieval layer evaluates the requesting user against document-level metadata before any content reaches the model, which in turn means that metadata has to exist.
There was no evaluation, only a demonstration
A pilot is usually judged on whether it impressed people in a room. Production needs a dataset of representative questions with known-good answers, run on a schedule, measuring accuracy, groundedness, refusal behavior, latency and cost.
Without that, nobody can tell whether a change to a prompt, a model version or the underlying content made the system better or worse. Providers update models underneath you; a system with no regression suite silently changes behavior and nobody notices until a user complains.
Nobody was accountable for a wrong answer
Every production AI system needs a named person who owns its output, an escalation path when it is wrong, and oversight proportionate to the consequence. Assistance with drafting needs light oversight. Anything that changes enterprise state, or materially affects a customer, needs an approval gate and an audit trail.
Organizations that skip this do not avoid the question. They simply answer it during an incident instead of during design.
What this implies for sequencing
The uncomfortable conclusion is that the work which determines whether AI reaches production is mostly not AI work. It is content lifecycle, access metadata, evaluation practice and governance.
That is also the reassuring conclusion, because all four are tractable, and all four have value independent of any AI initiative.
- Pick a use case where being wrong is cheap and being right is measurable.
- Fix the content and the access metadata before building retrieval over them.
- Build the evaluation set before the interface.
- Name the accountable owner before go-live, not after the first incident.
Discuss an AI initiative
If this describes something you are working through, we can talk it through against your specific environment.