Three things I keep coming back to

AI pilots almost never fail on the model. They fail in the gap between the team that bought the system and the people who have to run it every day.

A model that scores well in a notebook is not a system that survives a Monday morning. The failure shows up where the tool meets the operators: the caseworker who does not trust the output, the clinician who has no time to check it, the process that was never redesigned around it. None of that is a modelling problem, so none of it is fixed by a better model. It is an ownership problem, and ownership is decided before procurement, not after.

The fix is usually organisational, not technical, which is exactly why it is so easy to miss.

The distance between a working demo and a production system is the entire job. It is also the part nobody budgets for, and the reason timelines double.

A demo has to work once, in front of a friendly audience, on inputs you chose. Production has to work every day, on inputs you did not choose, when nobody is watching. Everything in between, the evaluation, the monitoring, the failure modes nobody demoed, the edge cases that are two percent of traffic and ninety percent of the incidents, is the actual work. It is invisible in the pitch, which is why the pitch is always optimistic and the plan is always short.

Measured properly it is fairly predictable. Most teams just end up finding out the expensive way.

Most of what your consultants pitched you is trend-following dressed as strategy: confident, generic, and untestable.

Trendslop is advice that sounds like insight but survives no matter what happens. It is the deck that would read the same for your competitor, the framework with no failure condition, the recommendation that cannot be wrong because it cannot be checked. It thrives in AI because the field moves fast enough that nobody is held to last year's confident claim. The antidote is not more caution; it is a single demand for a testable prediction.

Confident nonsense does well for a while. Asking for evidence is how you stop paying for it.