Shipping is where the work starts

Almost everyone treats the day an AI system goes live as the finish line. The pilot worked, procurement signed, the thing is in production, and the team moves on to the next project. That instinct is exactly backwards. Go-live is not the finish line, it is the starting line, and the race that starts there is the one most projects lose.

The other essays on this site are about why AI fails before it ships: the management gap where ownership never lands, and the demo-to-ship delta where the invisible work goes unbudgeted. This one is about what happens after it ships, because a system that clears both of those and goes live can still die, and the way it dies is quiet. Nobody notices for months. This essay is about what running an AI system actually involves, day to day, and why the running is the job.

Three things that are true of every live system

A demo has none of these problems, which is precisely why the demo tells you nothing about the life ahead. The moment a system is live, three things become true of it that were never true on the bench.

It drifts. The world it learned from keeps moving, and the model does not move with it. It fails silently. When it gets worse, it does not crash, it just gets worse, and nothing tells you. And it has no natural owner of its day. The team that built it has moved on, and the people who use it were never given the job of watching it.

Take those one at a time, because each one is a way the pilot dies, and each one has a specific answer.

It drifts, and drift is not a bug

The single most important fact about a machine learning system in production is that its accuracy is not a fixed property. It is a snapshot that starts going stale the moment you take it.

The technical name for this is concept drift, and it has been studied for over a decade. The canonical survey defines it as what happens when the relationship between the inputs and the thing you are predicting changes over time (Gama et al., 2014). The model learned a relationship that was true when it was trained. The world then carries on changing, and the relationship quietly stops being true. Customer behaviour shifts, a supplier changes a form, a new kind of case starts appearing that was rare in the training data. None of it is a failure of the model. It is the model doing exactly what it was told, in a world that no longer matches the instructions.

This compounds with the ageing effect covered in the demo-to-ship delta: a peer-reviewed study across healthcare, finance, transport, and weather found that more than 90 percent of models degraded as time passed since their last training. Put drift and ageing together and you get the uncomfortable rule: a system’s performance on launch day is the best it will ever be, unless someone actively keeps it up. Left alone, it only goes one direction.

So “it worked when we launched” is not a durable claim. It is a claim with a shelf life, and nobody tells you the expiry date. Finding it is the job.

It fails silently, which is the dangerous part

Here is the difference between an AI system and almost any other piece of software you run.

When a web server breaks, it breaks loudly. Requests fail, errors spike, a dashboard turns red, someone gets paged within minutes. The failure announces itself. An AI system that is drifting does not do any of that. It keeps returning answers, on time, in the right format, with the same confidence it always had. The answers are just getting worse. There is no error, no crash, no red light. A model that has quietly lost fifteen points of accuracy looks, from the outside, identical to one that is working perfectly.

This is the trap that kills pilots. The system that crashes gets fixed, because someone finds out. The system that silently degrades runs for months returning worse and worse answers, and the first sign of trouble is a human noticing a pattern of bad outcomes long after the damage is done: the misfiled cases, the wrong flags, the decisions that quietly went the wrong way. By the time a person catches it by hand, you are not looking at an incident, you are looking at a backlog.

The whole point of monitoring an AI system is to make silent failure loud. That is not a nice-to-have you add later. It is the thing that decides whether you find out from a graph in hours or from a complaint in months.

What monitoring actually means here

“Monitor the system” is easy to say and easy to wave at. It is worth being concrete, because monitoring an AI system is not the same as monitoring ordinary software, and the part that matters is the part teams skip.

The most useful reference I know is a Google paper that turned production readiness into a checklist: 28 concrete tests, one whole section of which is devoted to what you have to monitor once the system is live (Breck et al., 2017). Strip it down and there are three things to watch, in rising order of difficulty.

The inputs. Has the data coming in started to look different from the data the model was trained on? A shift in the inputs is the earliest warning of drift, and it is the cheapest thing to watch, because you do not need to know whether any answer was right. You just need to notice that the questions changed.

The outputs. Has the distribution of the model’s own answers moved? If a system that used to flag two percent of cases is suddenly flagging nine, something changed, in the world or in the model, and either way you want to know before the ninth percent becomes a problem.

The outcomes. Were the answers actually right? This is the one that matters most and the one almost nobody does well, because it is genuinely hard.

The ground-truth problem, said honestly

Watching inputs and outputs is cheap. Knowing whether the model was right is not, because being right is measured against reality, and reality arrives late.

When your system predicts that a customer will churn, or that a case is low-risk, or that a document belongs in a certain pile, you do not find out the truth at the moment of prediction. You find out weeks later, if at all, when the customer either churns or does not, when the low-risk case turns out to have been high-risk, when someone downstream corrects the filing. The label that would tell you whether the model was right is delayed, and sometimes it never comes, because nobody records the correction in a way the system can learn from.

This is the honest hard part of running AI, and it is why so many systems drift undetected: the signal that would reveal the drift is exactly the signal you do not have in real time. The answer is not a clever algorithm. It is a deliberate feedback loop: decide, before launch, how you will find out whether the system was right, sample real cases for human review even when nothing looks wrong, and build the correction back into your records so that ground truth accumulates instead of evaporating. In regulated work this is not optional anyway, and it is where a small, funded human-review sample earns its cost many times over. It is the only thing standing between you and a system that has been wrong for a quarter.

Retraining is a decision, not a schedule

When monitoring does catch drift, the reflex is to retrain the model on fresh data. That reflex is right, but two things about it are commonly got wrong.

First, a drift alert is an investigation trigger, not a retrain button. Before you conclude the model has gone stale, check the boring explanations, because most alerts are one of them: a data pipeline broke, an upstream system changed a field, a label definition shifted, an instrumentation bug is feeding you noise. Retraining a healthy model on broken data does not fix anything, it launders the bug into the model. Look at the plumbing before you blame the model.

Second, when you do retrain, the new model has to earn its place exactly the way the first one should have. A retrained model is a new model. It gets the full demo-to-ship treatment again: tested on messy, real, unchosen inputs, judged against a threshold you wrote down beforehand, checked on the worst segments and not just the average. “The new version scored higher on our test set” is how a regression ships to production wearing the costume of an upgrade. The discipline that got you to launch is not a one-time gate. It runs every time the system changes, which in a healthy system is often.

What happens when it is wrong on a Tuesday

Sooner or later the system will be wrong in a way that matters, on an ordinary day, when no one is expecting it. What happens next is an operational question you want to have answered before it happens, not during.

Who gets told? Something has to connect a monitoring alert to a specific human who will look at it, not a shared inbox nobody reads. Who is allowed to pull it back? There has to be a person with the authority to take the system offline or revert to the previous version, and they have to be able to do it in minutes, not after a meeting. And what do you fall back to? A rollback plan, decided in advance, so that “turn it off and go back to the old way for a day” is a button someone can press, not a crisis you improvise.

None of this is exotic. It is ordinary operational hygiene, the same kind any serious system has. The reason AI systems so often lack it is that they were run as projects with an end date, not as services that have to be operated. A project ends at launch. A service is only just beginning. The pilots that survive are run as services.

The person who is supposed to be watching

Every part of this comes back to a human, and it is the same human the management gap is about: someone on the floor, close to the work, whose actual job includes watching this system.

If that job does not exist, none of the monitoring matters, because a dashboard nobody is paid to read is not monitoring, it is decoration. And if the job exists but is unfunded, piled on top of everything the person already did, it decays into alert fatigue: the flags get dismissed, the review sample gets skipped when the week is busy, and within a few months you are back to finding out from a complaint. Watching a live system is real work with a real cost, and the organisations that keep systems alive are the ones that treated that cost as part of the price of the system, not as an afterthought.

In the EU this is now partly a legal duty rather than a matter of good practice. The AI Act requires post-market monitoring for high-risk systems and puts the oversight duty on a named, competent human at the organisation that runs the system, not the one that built it (the EU AI Act, in plain terms). The law, read plainly, is describing exactly this: that a live AI system is a thing you operate and watch, not a thing you approve and forget.

What I have seen

I build and ship AI in settings where being wrong is expensive, and the pattern after launch is as consistent as the pattern before it. The systems that survived were not the ones with the best models on launch day. They were the ones where someone was watching, where a drop in quality reached a human quickly instead of a customer slowly, and where “who runs this now” had a real answer. The ones that died rarely died of a bad model. They died of neglect that looked like success, because for months the system kept answering and nobody knew the answers had gone bad.

What to do about it

Before you launch, decide three things in writing, because after you launch you will not have the calm to decide them well. Decide what you will monitor and what number, crossed, sends an alert to a named person. Decide how you will learn whether the system was actually right, and fund the human review that makes ground truth accumulate. And decide, in advance, who can pull the system back and what you fall back to when they do.

Then treat go-live as day one of operations, not the closing of the project, and budget the running the way you would budget the running of anything else that matters. The work does not end at launch. At launch, the real work is only starting, and the teams that understand that are the ones whose pilots are still alive a year later while everyone else is quietly wondering where the value went.

A pilot that dies in production almost never dies of a bad model. It dies because the day it went live, it stopped being anyone’s job.

---

Sources

• João Gama, Indre Zliobaite, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia, A Survey on Concept Drift Adaptation, ACM Computing Surveys (2014).
• Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley, The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, IEEE Big Data (2017).
• Vela, D. et al., Temporal quality degradation in AI models, Scientific Reports (2022).
• Regulation (EU) 2024/1689 (the EU AI Act), on post-market monitoring and human oversight.