agents/workflows/ml-pipeline.md

model

model
sonnet

Machine Learning Pipeline

Design and implement a complete ML pipeline for: $ARGUMENTS

Create a production-ready pipeline including:

1. Data Ingestion:

   • Multiple data source connectors
   • Schema validation with Pydantic
   • Data versioning strategy
   • Incremental loading capabilities

2. Feature Engineering:

   • Feature transformation pipeline
   • Feature store integration
   • Statistical validation
   • Handling missing data and outliers

3. Model Training:

   • Experiment tracking (MLflow/W&B)
   • Hyperparameter optimization
   • Cross-validation strategy
   • Model versioning

4. Model Evaluation:

   • Comprehensive metrics
   • A/B testing framework
   • Bias detection
   • Performance monitoring

5. Deployment:

   • Model serving API
   • Batch/stream prediction
   • Model registry
   • Rollback capabilities

6. Monitoring:

   • Data drift detection
   • Model performance tracking
   • Alert system
   • Retraining triggers

Include error handling, logging, and make it cloud-agnostic. Use modern tools like DVC, MLflow, or similar. Ensure reproducibility and scalability.