ML Model Monitoring & Alerts

MONITORING & ALERTS

• Monitor! Stop being a blind DS

• Monitor your dependencies! Stop being a blind DS

• Data science observability for executives

• Production Machine Learning Monitoring: Outliers, Drift, Explainers & Statistical Performance, youtube, uses alibi-explain (see compendium) and Ali-detect (see compendium)

• Mlflow, Hyperparameterhunter,hyperopt, concept drift, unit tests.

• meta anomaly over multiple models, aggregate.

• Vidhya on monitoring data & models

• Monitor ML features using Amazon SageMaker Feature Store and AWS Glue DataBrew

Drift

1. Data & concept drifts, 2

2. (good) Inferring Concept Drift Without Labeled Data. also talks about stream-based drift by Cloudera - fast forward labs.

3. Arize.ai

   1. Data, concept, feature drifts - various comparisons between train/prod/validation time windows, diff models, a/b testing etc.., and how to measure drifts

   2. Model store, Feature store, evaluation store

   3. Monitor model performance in production - real- time, biased, delayed, and no ground truth.

   4. use cases - i.e., how to use statistical differences/distances

4. Some advice on medium, relabel using latest model (can we even trust it?) retrain after.

5. Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber - Previous research on concept drift mostly proposed model retraining after observing performance decreases. However, this approach is suboptimal because the system fixes the problem only after suffering from poor performance on new data. Here, we introduce an adversarial validation approach to concept drift problems in user targeting automation systems. With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.

6. Drift estimator between data sets using random forest, the formula is in the medium article above, code here at mlBOX

7. Alibi-detect - is an open-source Python library focused on outlier, adversarial, and drift detection, by Seldon.

8. What is concept drift and why does it go undetected Breaking down concept drit and explaining the best methods to avoid it

9. **How does data drift hamper AI performance ** Understand how data drift affect peak AI performance and how you can detect it

Alibi Detection Drift Features

Tool Comparisons

1. State of MLOps (by me), medium article, open-source AirTable.
2. MLOps.toys - A curated list of MLOps projects by Aporia
3. Neptune.AI MLOPS tools landscape
4. Twimlai ML AI solutions
5. Ambiata how to choose the best MLOps tools
6. Lakefs on the state of data engineering - has monitoring and observability inside
7. The NLP Pandec - MLOps for NLP
8. ml-ops.org
9. Awesome production ML

Awesome production ML