Stack
MLflow + FastAPI
Monitoring
Prometheus + Grafana
Infra
Docker
Most ML tutorials end at model training. But training is 10% of the work — the remaining 90% is keeping the model alive, accurate, and observable in production. This is where most ML projects fail. I built an end-to-end MLOps pipeline for Telco Churn Prediction to demonstrate how to cross that gap.
The Full Stack
code┌─────────────────────────────────────────────┐
│ REQUEST │
└──────────────────────┬──────────────────────┘
↓
┌────────────────────────┐
│ FastAPI REST Server │
│ /predict endpoint │
└────────────┬───────────┘
↓
┌────────────────────────┐
│ MLflow Model Store │
│ Registered + Versioned│
└────────────┬───────────┘
↓
┌────────────────────────┐
│ Prometheus Scraper │
│ (every 60 seconds) │
└────────────┬───────────┘
↓
┌────────────────────────┐
│ Grafana Dashboard │
│ + Drift Alerts │
└────────────────────────┘Why Drift Detection Matters
The most important metric to track isn't accuracy on your test set. It's prediction drift — the divergence between your training data distribution and the live data your model is seeing in production.
For the Telco Churn project, I tracked two signals: the distribution of predicted probabilities (confidence drift) and the feature value distributions (data drift). When either deviated beyond a threshold, Grafana fired a Slack alert automatically.
The Setup in Three Steps
Instrument FastAPI
Expose a /metrics endpoint using the prometheus-fastapi-instrumentator library. This auto-tracks request latency, prediction count, and model confidence.
codefrom prometheus_fastapi_instrumentator import Instrumentator
Instrumentator().instrument(app).expose(app)Configure Prometheus
Point Prometheus at your FastAPI /metrics endpoint with a 60-second scrape interval.
codescrape_configs:
- job_name: 'churn_model'
scrape_interval: 60s
static_configs:
- targets: ['fastapi:8000']Build the Grafana Dashboard
Import a standard FastAPI dashboard JSON, then add a custom panel for prediction confidence percentiles. Set an alert when p10 confidence drops below 0.55.
This single addition — real-time drift monitoring — is the difference between a toy project and a production-grade ML system. Every model you ship deserves an observable lifecycle, not just a one-time deployment.