At a financing bank, 25 machine learning models were running in production. Nobody was watching them.
I am not saying this to point the finger at the team. They had done the hard part: putting those models into service, which many organizations never manage to do. Once the models were deployed, though, nothing measured their performance anymore. And a model you stop measuring always ends up being wrong without anyone noticing.
In the carousel
The LinkedIn carousel slides are below.
A model portfolio with no dashboard
The bank had 3,000 employees. Its 25 models covered sensitive ground: credit scoring, pricing, fraud and anti-money laundering. In other words, decisions that touch risk, revenue and compliance directly.
None of these models had monitoring after deployment. There was no automatic alert either. The MLOps team found the problems after the fact, once they had already had an effect.
The only real checkpoint was the annual review. At the last one, we found 4 models that had degraded by more than 15%.
Why the annual review is not enough
A year is a long time for a model that is getting it wrong.
On credit scoring, the drift cost an extra €2 million in losses before anyone spotted it. The model kept returning scores, decisions kept being made, and nothing in day-to-day operations signaled that something was off.
The annual review had another problem too: its cost. It took 3 person-months per model. With 25 models, and a portfolio that kept growing, this approach could not keep up. The more models the bank put into production, the more exposed it became, and the more expensive the control got.
Then there is the regulator. Credit or anti-money-laundering models that are not monitored continuously are a regulatory risk. You have to be able to show that the models are being watched, and how.
A model does not break. It slips.
That is what makes drift so treacherous. When a server goes down, everyone sees it within a minute. A model that drifts keeps answering, with the same confidence, only less and less accurately.
We usually distinguish three forms of drift, and we monitored all three:
- data drift: the data arriving in production no longer looks like the data the model learned from. The customer base shifts, a product changes, an upstream source is modified;
- concept drift: the relationship between the data and what you are trying to predict changes. Fraud behavior is the typical example, because fraudsters adapt;
- performance drift: in the end, the model is wrong more often than it was at validation.
None of these kinds of drift arrives all at once. They settle in over weeks or months. That is exactly why a check once a year comes too late.
What we put in place
The project lasted 3 months. The goal was simple to state: know, at any moment, what state each of the 25 models is in.
First we deployed a continuous monitoring platform, built with MLflow and Evidently, on Databricks, with Grafana for visualization. It watches all three forms of drift across the whole portfolio, in real time.
Next we set an alert threshold. As soon as a model drifts by more than 5%, the team is notified. That is a long way from the 15% discovered at the annual review: the problem surfaces while it is still small.
The third piece is retraining. When drift is detected, retraining starts automatically in under 48 hours. The fix no longer depends on someone being available to take care of it.
Finally, a centralized dashboard shows the health of each model. The team looks at it every day, and a regulator can read it too. The monitoring is documented and auditable, which completely changes the conversation when the controls happen.
After go-live
I think this is the most important point of the project. The monitoring system was not delivered and then left to itself. It is operated over time, under contractual SLAs.
That fits what we argue for at Bomzai. A Data and AI system does not stop the day it goes into production. The data changes, and the models eventually drift. If nobody is responsible for watching them, you end up back where you started, sooner or later: models in service that nobody looks at.
What I take from this engagement is that monitoring is not a luxury you add once everything else is done. It is what keeps a model's value once it is in production. Without it, every deployed model is a risk you are not measuring.
A few questions to ask yourself
If you have models in production, these are the questions I would ask first:
- how many models are running today, and who keeps an up-to-date list?
- for each one, do you know how it is behaving this week, and not only at the time it was validated?
- who gets told, and how quickly, when one of them starts to drift?
- how long does it take to retrain it and put it back into service?
- can you show all of this to an auditor or to the regulator without spending weeks on it?
If several answers are fuzzy, you are not alone. It is a situation I come across often, including in teams that are very strong technically.
Where to start
Nobody is asked to rebuild everything. Usually you start with an audit or a training course, to inventory the models in production and spot the ones that carry the most risk. Then comes a first use case: monitoring a first batch of models, put into production and then operated in RUN, under an SLA. The program, which extends the setup to the whole portfolio, comes once that first batch has proved itself.
Shall we talk?
If you have models in production whose performance nobody really follows, or if your only control is still a review once a year, I would like to talk it through with you.
Write "DIAGNOSTIC" to me on LinkedIn or via the contact page. We take 30 minutes to look at your models and what should be watched first.
Jean-Emmanuel Orfèvre, founder of Bomzai
