Shipping AI
What actually breaks in a production ML system
It is almost never the model weights. Four failure modes we keep meeting, and the monitoring that catches each one.
5 min readByteWeave Studio
Teams shipping their first machine learning feature usually prepare for the model to degrade. They set up accuracy tracking, plan a retraining schedule, and wait for the slow decay they have read about.
The decay is real but slow. What takes systems down in the first year is more mundane, and mostly happens outside the model entirely.
The input pipeline changes without telling you
Someone upgrades the scanner and the new default is 200 DPI instead of 300. A vendor switches invoicing software. An upstream service starts sending dates in a different format. The model is unchanged and its accuracy falls off a cliff, and because nothing was deployed, nobody thinks to look at a deploy.
The monitoring that catches this is not accuracy monitoring, which is lagging and often needs labels you do not have in real time. Track the inputs: resolution distribution, file type mix, field presence rates, average text length. A step change in any of those is an alert, and it fires before the output quality problem reaches anyone.
Confidence drifts before accuracy does
A useful early signal is the distribution of the model’s own confidence scores. If the share of predictions above your threshold moves several points in a week, something upstream has changed, whatever the accuracy figure says.
This is cheap to track because it needs no ground truth. Plot the histogram weekly. In our experience it moves days before anyone notices a quality problem, which is usually enough time to find the cause before it becomes a complaint.
The queue nobody is draining
Any human-in-the-loop system has a review queue, and queues fail quietly. The person who used to clear it changes role, volume rises after a new client is onboarded, or someone goes on leave. Documents pile up, the pipeline looks healthy on every technical dashboard, and the business outcome you promised silently stops happening.
Monitor queue depth and the age of the oldest item, and alert on both. This is ordinary operational hygiene and it is the failure we have seen most often, because it does not look like a machine learning problem to anyone watching the machine learning.
Nobody can explain a specific decision
Six months in, a client asks why one particular invoice was posted with the wrong total. If you cannot answer, the conversation shifts from a bug to a question about whether the system can be trusted at all, and that is a much harder position to recover from.
Log the inputs, the model version, the confidence scores and the routing decision for every item, with a retention period you have agreed. Storage is cheap and the ability to reconstruct one decision from four months ago is worth more than most of the accuracy work you could do with the same effort.
- MLOps
- Monitoring
- Production
- Reliability
More reading
Shipping AI
How to price an AI project when you can’t promise accuracy
Fixed-price assumes a known scope. Machine learning does not give you one up front. A structure that protects both sides without pretending.
5 min read
Shipping AI
Human-in-the-loop design: deciding when to ask a person
Route too much to review and you have rebuilt the manual process. Route too little and people stop trusting the output. Where the line goes.
5 min read
Document AI
Why your invoice OCR works in testing and fails in production
The test folder is clean exports. The real intake is phone photographs. What changes between the two, and how to find out before a client does.
5 min read
Have a problem
worth solving?
Tell us what you're building. We'll help you figure out what's possible — and say so if we're not the right people for it.