Cosmos-ML-Flow – Production-Grade MLOps Accelerator for AWS

Overview: Cosmos-ML-Flow is an end-to-end MLOps pipeline accelerator built on Databricks that automates the complete ML lifecycle for enterprise use cases. It delivers a config-driven, reusable framework for deploying and governing ML models at scale with Unity Catalog. The reference implementation demonstrates real-time transaction fraud detection ingesting from Kafka, performing feature engineering, training and validating models, scoring at scale, and monitoring model health. Part of Coforge Data Cosmos™ - the innovation backbone comprising of platforms, agents, and services that accelerates execution across every phase of the data lifecycle

Key Features:

1. Automated Feature Engineering: 10+ behavioral features: temporal patterns, user velocity (rolling averages, z-scores), transaction sequencing, and currency encoding.

2. Dual Model Training: LightGBM (Champion) + XGBoost (Challenger) with Hyperopt tuning (50 trials, TPE algorithm) optimizing F1 score. Best model auto-promoted.

3. MLflow Experiment Tracking: Full lineage: parameters, metrics, artifacts, classification reports, feature importance — all versioned and reproducible.

4. Automated Quality Gates: 5-point validation (accuracy, F1, precision, recall, AUC-ROC) before any model reaches production. Models failing any gate are blocked.

5. Unity Catalog Model Registry: Champion/Archived alias management with full versioning, governance, access controls, and audit trails.

6. Batch Inference at Scale: Spark-distributed scoring writing predictions to Delta tables. Handles millions of transactions per batch cycle.

7. Continuous Monitoring: PSI-based distribution drift + KS statistical tests across 8 numeric features with automated alerting. Triggers retraining before degradation.

8. Interactive Dashboard: 5-page Streamlit UI: Executive Summary (KPIs, fraud distribution), Fraud Transactions (filterable with CSV export), Model Performance (live accuracy/F1 with pass/fail gates), Data Drift (PSI per feature), Real-Time Scoring (interactive with risk factors).

9. Config-Driven Architecture: YAML-based pipeline and model configuration. No code changes needed to adjust thresholds, schedules, or parameters.

Three Automated Pipelines: • Training — Feature engineering → Model training (tuned) → Validation → Champion promotion • Inference — Load Champion model → Score all transactions → Write predictions to Delta • Monitoring — Compute metrics → Detect drift (PSI/KS) → Evaluate alerts

Industry Applications: • Banking — Transaction fraud detection with dual model approach. Unity Catalog ensures model lineage for regulatory compliance. • Insurance — Claims fraud scoring with quality gates ensuring actuarial-grade precision. • Travel — Revenue optimization and dynamic pricing with continuous drift monitoring. • Healthcare — Clinical risk scoring with HIPAA-compliant governance through Unity Catalog.

Technology Stack: AWS, Databricks (Unity Catalog, Delta Lake), Kafka (Structured Streaming), PySpark, LightGBM + XGBoost, MLflow, Hyperopt (TPE), SciPy (KS test) + custom PSI.

Business Benefits: • Reduced fraud losses: automated detection flags suspicious transactions • Operational efficiency: fully automated pipelines eliminate manual model management • Governance & compliance: Unity Catalog provides full lineage and audit trails • Continuous improvement: drift detection triggers retraining before degradation • Business visibility: non-technical stakeholders get real-time dashboards • Scalability: Spark-distributed inference handles millions of transactions • Rapid iteration: config-driven changes without code deploys

Cloud-Native Deployment on AWS: Databricks on AWS. Amazon S3 as data lake (Delta Lake). Apache Kafka on Amazon MSK for streaming. Unity Catalog for model governance. Amazon CloudWatch for infrastructure monitoring.