Experiment tracking records how every model candidate was produced. A model registry records which of those candidates the organisation trusts, where each one runs, and who approved it.
Training a model is not the hard part of running machine learning in production. Answering questions about it months later is. Which version scored this transaction? What data trained it? Which experiment produced it, and what did it beat? Who signed off, and what replaced it?
Without a system of record, those answers live in notebooks, chat threads and the memory of whoever ran the training job. Experiment tracking and a model registry replace that memory with metadata captured at the moment work happens, so the history of every production model can be read rather than reconstructed.
- Log every training run automatically, including failures, with parameters, metrics, code and data references.
- Reference datasets by immutable snapshot or version, never by a table name that can change.
- Register only candidates worth reviewing; the registry is a shortlist, not a run log.
- Make lifecycle state explicit: candidate, approved, production and archived each need entry criteria.
- Keep deployment and rollback history on the registry record, so the question of what ran when has one answer.
A Worked Example: Two Questions at Meridian Retail
Meridian Retail is the illustrative retailer used across this series: about 4 million orders a month in 12 regions, an online fraud model with an 80 millisecond budget, and a weekly demand forecast. Two requests reached the ML team in the same quarter.
The first came from the payments team. Decline rates in one region had risen for ten days, and they needed to know which fraud model scored those orders, what data it was trained on and who approved it. Because version 14 had been registered from a tracked run, the answer took minutes. The registry record linked to the training run, the dataset snapshot, the corrected encoder, the shadow and canary evaluation reports, the approval and the deployment dates. It showed that version 14 had been live for the whole period, which pointed the investigation at the region's traffic mix rather than a release.
The second came from supply-chain planning, who wanted to reproduce the forecast that had performed best during the previous peak season. That model had been trained in a notebook. Its parameters were in a slide, and its training data was recorded as a table name. The table had been overwritten since. The model file existed, but nobody could say exactly how it was made.
The fraud model had a history. The forecast only had an artifact. Tracking and registration are what turn one into the other.
Experiment Tracking vs Model Registry
The two systems are often sold together and share a user interface, but they answer different questions for different people. Treating them as one leads either to a registry cluttered with thousands of runs, or to experiment logs that someone has to promote to production by hand.
The tracker records exploration. The registry records decisions.
| Aspect | Experiment tracking | Model registry |
|---|---|---|
| Unit of record | A run: one training or evaluation execution | A model version: a candidate someone chose to keep |
| Volume | Many runs per model, including failures | Few versions per model |
| Primary users | Data scientists and ML engineers | Reviewers, release owners, risk and audit |
| Key question | What did we try, and what happened? | What do we trust, and where is it running? |
| Mutability | Append-only history | Versions immutable; state and aliases change |
| Retention | Can be pruned after a period | Kept for the life of the system and beyond |
The link between them is the important part. Every registered version should point back to the run that produced it, so the decision record and the exploration record never drift apart.
Experiment Runs, Parameters and Metrics
A run is the smallest unit worth tracking. It should be created by the training code itself, not by someone remembering to log results afterwards. Logging that depends on discipline stops happening under deadline pressure, and it is usually the failed runs that go unrecorded.
- Identity: run ID, experiment name, author, start and end time, and the code commit that ran.
- Parameters: hyperparameters, feature list, sampling and split settings, random seeds.
- Metrics: training and validation curves, holdout results, and metrics per important segment.
- Environment: library versions, container image, hardware type.
- Inputs: dataset versions and the feature definitions they were built with.
- Outcome: completed, failed or abandoned, with a short note on why.
Segment metrics deserve particular attention. A fraud model can improve overall while getting worse on one payment method, and a single aggregate number hides that. After the wallet incident, Meridian made payment type a logged segment on every fraud run.
Name experiments by the question being asked, such as a new feature set or a different label window, rather than by person or date. Runs can then be compared across months by anyone.
If a run is not logged automatically, the comparison that matters later will be missing its baseline.
Artifacts and Dataset Version References
Artifacts are the files a run produces: the serialised model, preprocessing objects such as encoders and scalers, evaluation reports, plots and sample predictions. Store them in managed object storage attached to the run, with a content hash, so a version can be verified as unchanged later.
Datasets are the harder part. Copying training data into every run is expensive and often not allowed. Recording a table name is cheap and useless, because the table will change. The workable middle ground is a reference to an immutable version: a snapshot ID in a lakehouse table format, a partition range with a data version, or a dataset registry entry with a hash.
What a dataset reference should include
The source and version identifier, the time window and filters applied, the label definition and its version, row counts per split, and a schema fingerprint. That is enough to rebuild the training set or to prove that two runs used the same one.
The Meridian forecast failed on exactly this point. The model artifact survived. The data reference did not, because it pointed at something mutable.
What Reproducibility Actually Requires
Reproducibility in production ML rarely means bit-identical retraining. Hardware and non-deterministic operations make that difficult. It means being able to rebuild a model whose behaviour matches the original within agreed tolerance, and to explain any difference.
- Pin the code Record the exact commit, and refuse to register models trained from uncommitted changes.
- Pin the data Record immutable dataset and label versions, as described above.
- Pin the environment Record the container image or a locked dependency file, not just a list of top-level packages.
- Record randomness Log seeds, and note operations that remain non-deterministic on the chosen hardware.
- Test it Periodically retrain a registered version from its record and compare metrics with the original.
The last step is the one that proves the others. A reproducibility check that has never been run is an assumption, and it tends to fail the first time an auditor asks for it.
The Model Registry: Registered Models and Versions
A registry organises models in two levels. A registered model represents a purpose, such as checkout fraud scoring. Under it sit versions, each an immutable package produced by a specific run. Version numbers increase as candidates are added, and a version is never edited after registration. If something needs to change, a new version is registered.
Registration should be a deliberate step taken when a candidate is worth reviewing. The fraud team at Meridian registers a run only after it passes offline gates. Dozens of runs fed version 14; only three became versions.
What belongs on a version record
A link to the source run, the artifact location and hash, the input and output signature, the serving runtime it expects, the feature definitions it depends on, evaluation reports, a model card, lifecycle state and the history of every state change.
Registered models name a job. Versions are immutable candidates for doing it.
The Model Lifecycle: Experiment to Archive
The lifecycle only helps if each state has an entry criterion and a named owner. Otherwise states become labels that people set to whatever unblocks them.
Five states, each with evidence attached.
| State | Entry criterion | Who moves it | What is recorded |
|---|---|---|---|
| Experiment | A tracked run exists | Training pipeline | Parameters, metrics, artifacts, data references |
| Candidate | Offline gates passed; version registered | ML engineer or pipeline | Evaluation report, signature, model card draft |
| Approved | Review complete for the model's risk tier | Designated reviewer | Approver, date, conditions, evidence links |
| Production | Deployed and verified in an environment | Release pipeline | Environment, endpoint, deployment and verification dates |
| Archived | Replaced or retired; no longer a rollback target | Model owner | Reason, successor version, retention date |
Tools express state differently. MLflow's documentation marks its older fixed stages as deprecated and recommends aliases and tags instead: an alias such as champion points to one version, and tags record status. That is more flexible than fixed stages, because a version can be approved for one region while serving in another. Whatever the mechanism, the five states above are the questions it needs to answer.
Approval and Promotion
Approval is a decision about a specific version, made on recorded evidence. The registry is where that decision and its evidence live together.
- Assemble evidence The candidate's record gathers evaluation results, segment metrics, comparison with the current production version and the model card.
- Review by risk tier Low-risk models may be approved by a second engineer. High-impact decisions, such as fraud declines, need a named business owner.
- Record the decision Approver, time, conditions and any exceptions are written to the version, not to an email.
- Promote Approval changes state or moves an alias. The release pipeline reacts to that change.
- Verify and confirm Production state is set only after verification passes in the target environment.
The registry decides what may be deployed. How it is deployed, including shadow, canary and blue-green release, belongs to the delivery pipeline described in ML CI/CD and model deployment pipelines. Keeping those concerns apart lets release strategy change without rewriting approval rules.
Approval should be a recorded event on a version, and deployment should be triggered by it, not the other way round.
Lineage: From Source Data to Deployment
Lineage connects the version to everything upstream and downstream of it. Upstream: the run, code commit, dataset versions, label definitions and feature definitions. Downstream: the environments and endpoints it served, and the systems that consumed its predictions.
Both directions are needed. Upstream lineage answers what went into a model when a data defect is discovered: every version trained on the affected snapshot can be listed at once. Downstream lineage answers what a model affected, which matters when a version is found faulty after it has been serving.
Lineage records where a model came from. Proving that the artifact has not been tampered with since, through signing and provenance attestations, is a related security control explained in AI supply chain security.
Upstream lineage tells you which models a data defect touched. Downstream lineage tells you which decisions a faulty model touched.
Model Cards and Version Metadata
A model card is a short, structured description of a model written for people who did not build it. The format comes from research on model reporting and is now supported directly in several registries, including SageMaker, which can attach model card information to registered models.
- Intended use: the decision it supports and uses it is not approved for.
- Training data: sources, time window and known gaps.
- Performance: headline and segment metrics, with the evaluation dataset named.
- Limitations: conditions where behaviour is known to degrade.
- Ownership: model owner, business owner and review date.
Generate as much of the card as possible from tracked metadata, so the numbers cannot disagree with the run. Leave intended use and limitations to people, because those are judgements. At Meridian, version 14's card records that the model has limited evidence for payment methods introduced after its training window, which is the lesson of the wallet incident written where the next reviewer will see it.
Environment Association and Rollback History
A version can be approved without running anywhere, and it can run in several places at once. The registry should record, per environment, which version is serving, since when, and which version it replaced. That history turns rollback from a search into a lookup.
Environment association
Record staging, shadow, canary and production placements separately, with region where it matters. Aliases per environment, or per region, keep the mapping explicit.
Rollback history
Each promotion should name its predecessor. A rollback is then a promotion of the predecessor, recorded with a reason, rather than an undocumented redeploy. Keep a version marked as the rollback target, and do not archive it until its successor has passed its bake period.
When Meridian's payments team asked which model scored the disputed orders, this history is what answered: version 14 in that region from its promotion date onward, with version 13 still marked as the rollback target at the time.
MLflow, SageMaker and Vertex AI Registry Concepts
The major registries implement the same ideas under different names. Knowing the mapping helps when writing governance rules that should survive a platform change.
Terminology differs; the underlying model is similar.
| Concept | MLflow | SageMaker Model Registry | Vertex AI Model Registry |
|---|---|---|---|
| Model grouping | Registered model | Model group (model package group) | Model |
| Version | Model version, auto-incremented | Model version (model package) | Model version |
| Named pointer | Alias, such as champion | No alias object; approval status and version number are used | Alias, including a default alias |
| Status and approval | Tags; older stages deprecated | Approval status: PendingManualApproval, Approved, Rejected | Labels and aliases |
| Lineage | Version linked to its source run | Lineage tracking for traceability | Links to training and metadata services |
| Grouping at scale | Naming conventions and tags | Model registry collections | Labels |
Details are in the MLflow Model Registry documentation, the SageMaker Model Registry documentation and Google Cloud's Model Registry introduction. Vertex AI is now documented as part of Google Cloud's Gemini Enterprise Agent Platform. In SageMaker, a change to approval status can drive CI/CD deployment when project templates are used. In MLflow, moving an alias is the equivalent signal.
Registry Governance
A registry anyone can write to is a shared folder with a better interface. Governance makes its records trustworthy enough to rely on in a dispute or an audit.
- Separation of duties: the person who trained a high-risk model does not approve it.
- Controlled transitions: only pipelines and designated roles can move production aliases or states.
- Immutable versions: artifacts and core metadata cannot be edited after registration.
- Complete audit trail: every state change, alias move and approval is logged with actor and time.
- Retention rules: archived versions and their data references are kept as long as their decisions may be questioned.
- Naming standards: registered model names describe the business purpose and owner.
The registry supplies evidence. Deciding how much validation a model needs, and who can accept residual risk, is a policy question addressed in model risk management for enterprise AI.
Where the Registry Hands Off
The registry sits at the centre of the ML lifecycle, so it is tempting to make it do everything. It works best as the system of record that other systems read from and write to.
- Deployment: release pipelines read approved versions and write deployment events back. See ML CI/CD.
- Drift and quality: monitoring tracks how a production version performs over time. See AI model monitoring in production.
- System telemetry: traces, latency and dependency health across the application. See AI observability for enterprise systems.
- Features: the registry records which feature definitions a version expects. Serving those features consistently is the job of a feature store.
- LLM applications: prompts, retrieval configuration and model providers add lifecycle objects a classic registry does not model. See MLOps vs LLMOps.
Monitoring findings should flow back to the registry as annotations on the affected version. That keeps the version's record complete: how it was built, why it was approved, where it ran and how it behaved.
The registry owns the record of each version. Pipelines, monitors and feature stores own the work around it.
Frequently Asked Questions
What is the difference between experiment tracking and a model registry?
Experiment tracking records every training run with its parameters, metrics, artifacts and data references. A model registry holds the smaller set of versions chosen for review or production, with their approval, lifecycle state and deployment history. Each registered version should link back to the run that produced it.
What should be logged for every experiment run?
The code commit, parameters, random seeds, dataset and label versions, environment details, metrics overall and per important segment, output artifacts with hashes, and the run's outcome, including failures. Logging should happen automatically from the training code.
How do you make ML models reproducible?
Pin the code commit, reference immutable dataset versions, capture the full runtime environment and record seeds. Then test it: retrain a registered version from its record and compare the result with the original within an agreed tolerance.
Are MLflow model stages still recommended?
MLflow's documentation marks model stages as deprecated and recommends model version aliases and tags instead. An alias such as champion points to a specific version, and tags can record status such as validation results.
Who should approve a model for production?
It depends on risk. Low-impact models can be approved by a second engineer. Models that drive consequential decisions should be approved by a named business or risk owner who did not build them, with the decision and evidence recorded on the version.
How does a model registry support rollback?
By recording which version served in each environment, when, and which version it replaced. The previous version stays marked as the rollback target until its successor is verified, so rolling back is a recorded promotion rather than an ad hoc redeploy.
