ML CI/CD and Model Deployment Pipelines

A Worked Example: The Retrained Fraud Model

Meridian Retail processes about 4 million orders a month across 12 marketplace regions. Two models matter here. A fraud model scores every card-not-present order at checkout, inside an 80 millisecond latency budget. A demand forecast runs weekly in batch and feeds regional replenishment.

A drift alert triggered a retrain of the fraud model. Version 14 beat version 13 on the offline holdout, overall and in every region. It was still not safe to release.

1. Offline gate Passed. The holdout predated a new digital wallet launched three weeks earlier, so the wallet barely appeared in evaluation data.
2. Shadow deployment Version 14 scored live traffic for seven days without affecting decisions. It would have declined wallet orders at more than twice the current rate.
3. Root cause A refactored encoder mapped the unseen wallet category to an unknown bucket. In training data, unknown values correlated with fraud.
4. Fix A fixture test for unseen categories was added to CI, the encoder was corrected, and the model was retrained and re-shadowed.
5. Canary Five percent of traffic for 48 hours, with guardrails on p99 latency and on decline rate within half a percentage point of baseline.
6. Promotion A blue-green switch moved all traffic. Version 13 stayed deployed and warm for seven days as the rollback target.

The offline gate was not wrong. It measured what it was given. The defect only existed in live traffic, which is exactly what shadow deployment is designed to expose before customers see it.

Every stage caught something the previous one structurally could not see.

Why ML Needs Two Delivery Pipelines

A conventional pipeline has one input: source code. An ML system has two moving parts that change on different schedules. Keeping them separate is what makes each one testable.

1. The code pipeline Triggered by a commit. It builds and tests training code, feature logic, serving code and the pipeline definition itself, then deploys the training pipeline.
2. The model pipeline Triggered by a schedule, new data or a monitoring signal. It runs the deployed training pipeline, evaluates the result and, if gates pass, releases a model.

Conflating them causes two failure modes. Teams retrain by hand every time code changes, which hides data effects behind code effects. Or they redeploy code to change a model, which makes rollback of one without the other impossible.

Separation also clarifies ownership. Engineers own the code pipeline and its tests. Model owners own the gates that decide whether a trained model is good enough to release.

Code changes deploy a new training pipeline. Data changes produce a new model. Keep the two paths distinct.

The Training Pipeline

The training pipeline is the thing you actually deliver. A trained model is its output. If the pipeline cannot be rerun to produce an equivalent model, nothing downstream is reproducible.

A production training pipeline runs as an orchestrated graph rather than a notebook. The same dependency and retry discipline described in data orchestration for modern data platforms applies. Tasks run when their inputs are ready. A failed step is retried or stopped, never silently skipped.

• Extract a versioned training snapshot, with the query and time window recorded.
• Validate the snapshot's schema, volume and value ranges before training starts.
• Build features with the same code that serving will use.
• Train with pinned library versions, seeds and hyperparameters.
• Evaluate against a holdout and against the current production model.
• Write the model, its metrics and its metadata as one package.

Feature logic deserves particular care. If training computes a feature one way and serving computes it another, the model is evaluated on data it will never see. Sharing one implementation between both paths removes that class of defect.

Validation Gates

A gate is an automated check that a model must pass to move forward. The useful question for each gate is what it can detect and what it structurally cannot.

Gates in the order they usually run, with the failure each one exists to catch.

The champion comparison is the easiest gate to omit. A fixed threshold only asks whether a model is acceptable. Comparing against production asks whether it is better than what customers already have.

Offline gates reject bad models reliably. They cannot prove a model is good on traffic they have not seen.

Model Packaging

A model file on its own is not deployable. The package that moves through environments needs everything required to reproduce a prediction.

• The trained weights or model object.
• The preprocessing and feature code, pinned to the version used in training.
• An input and output signature, so serving can reject malformed requests.
• The runtime environment, usually as a container image with pinned dependencies.
• A reference to the training snapshot, code commit and evaluation results.

Build the package once and promote the same artifact through staging and production. Rebuilding per environment reintroduces the drift that packaging is meant to remove. A content hash on the package makes it easy to prove that what reached production is what passed the gates.

The serving platform shapes the package format. Managed endpoints, a shared inference cluster and embedded models in an application each impose different constraints. That choice is part of a wider enterprise AI platform strategy, and it is worth settling before the first production model rather than after the third.

CI for ML

Continuous integration for ML keeps the conventional checks and adds tests for data and model behaviour. Each runs on every commit to training, feature or serving code.

1. Unit tests Feature functions, encoders and transformations, including unseen categories, nulls and boundary values.
2. Schema contract tests Training and serving agree on feature names, types and order.
3. Training smoke test The pipeline trains end to end on a small fixed sample, so a broken step fails in minutes rather than hours.
4. Reproducibility check The same sample and seed produce the same metrics within a tolerance.
5. Serving test The packaged model loads, accepts a valid request and rejects an invalid one.

The Meridian defect would have been caught here once the unseen-category fixture existed. CI cannot anticipate every category, but it can guarantee that the encoder handles the unknown case deliberately rather than by accident.

Keep CI fast enough that engineers wait for it. Full training runs belong in the model pipeline, not on every commit. A small fixed sample lets the smoke test exercise every step in minutes. The expensive run then happens only after the code has passed.

Continuous Training vs Continuous Deployment

The terms get used interchangeably, but they automate different things. Continuous training retrains models automatically when a trigger fires. Continuous deployment releases models automatically once they pass gates.

Google Cloud's MLOps architecture guidance describes this as a maturity progression. At the first level, the training pipeline itself is automated and retrains on fresh data. At the next, CI/CD automates how new pipeline implementations are tested and deployed.

1. Continuous training only Models retrain automatically, and a person approves each release. A sensible starting point for high-risk decisions.
2. Continuous training and deployment Models retrain and release automatically when every gate passes. Suitable where rollback is fast and the cost of a bad release is bounded.

Automating deployment before gates are trustworthy just ships mistakes faster. A safer sequence is to automate training first, observe gate decisions for a few cycles, and automate release only once those decisions match what a reviewer would have chosen.

Batch vs Online Model Serving

The serving pattern determines what deployment and rollback actually mean. Meridian runs both: the forecast in batch, the fraud model online.

How the serving pattern changes deployment mechanics.

Batch deployments are easier to make safe. The new model can score the full population into a separate table, and promotion becomes a pointer change once outputs have been compared. Online serving has no equivalent pause, which is why it needs traffic-based release strategies.

Online serving adds a second dependency: the features themselves must be available at request time, with the freshness the model was trained to expect. A fraud model trained on transaction counts from the last hour will degrade quietly if the serving path only has counts from yesterday. Test the online feature path as part of deployment, not just the model.

Shadow, Canary and Blue-Green Deployment

Each strategy limits a different risk. Choosing one per team rather than per model leaves some models over-protected and others exposed.

Release strategies for online models.

The strategies combine well. Meridian shadows first, canaries second and uses a blue-green switch for the final move, so the previous model remains a one-step rollback.

Managed platforms implement these patterns directly. Amazon SageMaker's deployment guardrails shift traffic to a new fleet all at once, as a canary, or in linear steps. Alarms are watched during a baking period, and the deployment rolls back automatically if one fires.

Shadow mode is also the safest bridge from experiment to production for a first release. The full sequence, from readiness gates to controlled rollout, is described in how to move AI pilots into production. For later releases, the shadow period can shorten as confidence in the gates grows.

Shadow tests behaviour, canary limits exposure, blue-green makes rollback instant.

Approval Gates and Promotion

Not every promotion needs a person. The level of human approval should follow the risk of the decision the model makes, not a single organisational rule.

1. Automated promotion Low-impact models with fast rollback, such as ranking or internal forecasts, promote when every automated gate passes.
2. Reviewed promotion A named owner reviews gate results, slice metrics and shadow comparisons before release.
3. Governed promotion Credit, pricing, fraud and other regulated decisions add documented sign-off, and the evidence is retained for audit.

Whatever the tier, the approval should be recorded against the exact package hash. An approval that refers to a model name rather than a specific artifact cannot prove what was approved.

Separate the person who trains a model from the person who approves its release where the decision is regulated. The pipeline can enforce this by requiring an approver identity that differs from the author of the training run.

Rollback

Rolling back code means redeploying a previous build. Rolling back a model has more dependencies, and each one can quietly make rollback impossible.

• The previous model package is still stored and loadable.
• Its serving environment, or a compatible one, is still available.
• The features it expects are still computed, with the same definitions.
• Downstream consumers still accept its output schema.
• For online models, it stays deployed and warm through the bake period.

Feature changes are the easiest dependency to break. If a new model ships alongside a changed feature definition, the old model is now receiving inputs it was never trained on. Version features alongside models, and keep the previous feature version available until the new model is confirmed.

Rehearse rollback. A rollback path that has never been exercised in staging should be treated as unverified.

A rollback target is only real if its features, schema and runtime still exist.

Retraining Triggers

Retraining should start for a reason that can be written down. The common triggers each suit different models.

1. Scheduled A fixed cadence, such as Meridian's weekly forecast. Predictable and simple, but it retrains whether or not anything changed.
2. New data volume Retrain once enough new labelled data has accumulated to matter.
3. Performance degradation Retrain when measured accuracy against delayed ground truth falls below a threshold.
4. Drift Retrain when input or prediction distributions move beyond an agreed tolerance.

Detection belongs to monitoring. The thresholds, baselines and alert handling behind drift and degradation signals are set out in AI model monitoring in production. The delivery pipeline's job is to respond to those signals safely.

Guard against retrain loops. If drift is caused by a broken upstream feed, retraining on the broken data teaches the model the defect. Run data validation before training, and route drift that coincides with a data quality alert to investigation rather than to retraining.

Deployment Metadata and Reproducibility

Every deployment should answer one question months later: exactly what was running, and why was it allowed to run? That requires recording metadata at release time, not reconstructing it during an incident.

• Package hash, model version and serving image.
• Training snapshot reference, code commit and feature versions.
• Gate results, including slice metrics and the champion comparison.
• Release strategy, traffic percentages and bake duration.
• Approver, approval time and the evidence reviewed.
• The rollback target at the moment of release.

Reproducibility follows from the same records. With the snapshot, commit, feature versions and seed, the pipeline can be rerun to produce an equivalent model. Without them, an investigation can describe what the model did but not why.

This metadata is what a model registry and experiment tracking system formalise. The delivery pipeline writes to them; it should not be the only place the history lives.

Production Verification

A release is not complete when traffic reaches the new model. It is complete when verification confirms the model behaves as the gates predicted.

1. Smoke test Synthetic requests immediately after deployment confirm the endpoint loads, responds and returns the expected signature.
2. Prediction distribution Compare live output distributions with the shadow and offline results. A sudden shift points to a serving or feature defect.
3. Operational guardrails Latency, error rate and resource use stay inside their budgets.
4. Business guardrails Decision rates, such as approvals or declines, stay within an agreed band of baseline.
5. Bake and close After the bake period, the release is marked verified and the previous model can be retired on schedule.

Verification is bounded in time. Once a release is confirmed, ongoing health becomes a monitoring responsibility. The distinction matters for LLM applications too, where evaluation and guardrails replace some of these checks, as set out in MLOps vs LLMOps.

Give verification a named owner and a written exit condition. Without one, bake periods get shortened under delivery pressure, and the previous model is retired before anyone confirmed the new one was safe to depend on.

Gates predict behaviour. Verification confirms it on live traffic before the fallback is removed.

Frequently Asked Questions