ML Experimentation

MicrosoftOfficialPublished by Microsoft

Skill
Guidelines
Data and AI
CC-BY-4.0
No issues found

From the author

Machine learning experimentation reference for model-experimentation conventions, experiment tracking and reproducibility, dataset and model abstractions, ML engagement fundamentals, and model-production readiness. Use when standing up ML experimentation infrastructure or assessing whether a trained model is ready for production.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured reference for machine learning experimentation, focusing on reproducibility, environment management, and production readiness. It helps teams establish consistent conventions for repository structure, dataset and model abstractions, and evaluation flows. By grounding ML projects in established engineering playbooks, it ensures that experiments are comparable and that models are assessed against clear criteria before moving to production.

What it does

Provides conventions for virtual environment management and repository structure.
Outlines requirements for experiment tracking and reproducibility.
Defines dataset and model abstractions to ensure consistency.
Offers a checklist for assessing ML engagement fundamentals.
Provides a structured readiness review for moving models to production.
Clarifies ownership boundaries between experimentation, data operations, and other domains.

When to use it

When setting up infrastructure for a new machine learning project.
When establishing repository and environment conventions for an ML team.
When assessing whether a trained model meets the criteria for production deployment.
When needing to ensure experiments are repeatable and comparable across a team.
When evaluating the maturity of an ML engagement.

Advantages

Promotes consistent engineering practices across ML projects.
Improves reproducibility by standardizing environment and data tracking.
Provides a clear, structured path for production readiness assessments.
Helps teams balance rapid research exploration with operational requirements.
Reduces ambiguity in team roles and project lifecycle stages.

Usage examples

“How should we structure our ML repository to ensure reproducibility?”: The agent will provide guidance on folder structures, virtual environment configuration, and the necessity of tracking datasets, parameters, and code versions.
“Is our model ready for production?”: The agent will guide you through the ML Model Production Checklist, helping you evaluate readiness domains and make informed decisions based on your specific use case.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/ml-experimentation/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip ml-experimentation.zip -d .agents/skills

Documentation by Microsoft

ML Experimentation Reference Pack

Goal

Ground machine learning experimentation in the Microsoft CSE engineering playbook so that environment setup, repository structure, experiment tracking, dataset and model abstractions, evaluation flow, and production-readiness review are applied consistently and attributed accurately.

This pack is machine learning specific. It assumes a model is being trained, tracked, evaluated, or assessed for production. General experiment framing, hypothesis formation, and vetting belong to experiment-design.

Inputs

The ML experimentation setup under discussion: environments, repository layout, tracking framework, or evaluation flow
The model under assessment and its training and evaluation history, when readiness is the question
Existing dataset versioning, parameter tracking, and environment capture practice
The engagement stage, since the production checklist has a lifecycle precondition

Reference index

Read only the reference that matches the active concern.

Reference	Read this when
model-experimentation.md	Standing up virtual environments, repository and notebook structure, experiment tracking and reproducibility, dataset and model abstractions, or evaluation flow
ml-checklists.md	Checking ML engagement fundamentals or assessing whether a trained model is ready to move toward production
provenance.md	Confirming what is upstream guidance, what is HVE Core derivation or repository convention, and where upstream is silent

Success criteria

Reproducibility keeps all four elements: dataset names and versions, parameters, code, and environment. Tracking a dataset name without its version is a labelling practice, not reproducibility.
Datasets and evaluation are defined consistently enough that experiments are comparable. A tracking framework alone does not deliver comparability.
Checklist section headings and readiness domains are preserved, and applicability caveats travel with them.
Experiment reproducibility stays distinct from pipeline replayability, which belongs to dataops, the DataOps skill for tier behavior, pipeline invariants, validation placement, tests, drift, and operational signals.

Constraints

Reproduce only the minimum text necessary for a specific technical point, and paraphrase everything else. Attribute every reference and describe accurately what each reference reproduces.
Preserve checklist section headings as structural identifiers and summarize what each section covers. Do not restore the upstream item lists or reconstruct either checklist locally; link readers to the upstream page for item wording.
Label repository conventions as substitutions rather than upstream recommendations. The uv environment convention is a repository substitution, not a playbook recommendation.
Do not convert a checklist into an unconditional gate. Upstream scopes the production checklist to teams that have already trained a model and permits use-case-specific decisions.

Ownership boundaries

Concern	Owner
Experiment framing, hypothesis formation, vetting criteria, and red flags	`experiment-design`
Pipeline mechanics, data tiering, replay semantics, and DS/MLOps test technique	`dataops`
Data validation, drift detection, and their asymmetric remediation	`dataops`
Ethical and Responsible AI review	`rai-planner`
Telemetry naming and data sensitivity classification	`telemetry-foundations` and `privacy-standards`

Stop rules

Stop and route to experiment-design, the general experiment selection, hypothesis, vetting, scope, and evaluation skill, when the request is about whether an experiment is worth running rather than how to run it.
Stop and state the lifecycle precondition when the production checklist is invoked for a model that has not been built or trained.
Stop and state the gap when upstream does not cover the request, such as a universal framework, tool, or metric choice made without project context.

Attribution

This pack declares CC-BY-4.0.

Both references derive from Microsoft CSE Code With Engineering Playbook documentation pages, which are licensed CC BY 4.0. Upstream guidance is summarized rather than reproduced; section headings and tool and file names are carried across as identifiers. The upstream project applies MIT through a separate LICENSE-CODE file to code samples only, which this pack does not reproduce. Each reference cites its own upstream URL, states that changes were made, and describes what it reproduces.

Explanatory framing, the lifecycle caveat, the readiness-domain grouping, the routing table, and the uv substitution are repository-original.

See provenance.md for the consolidated source map and derivation labels.

Files

ATTRIBUTION.mdAdded by AutoGineer987 B
LICENSEAdded by AutoGineer16.6 KiB
SKILL.md6.7 KiB
references/ml-checklists.md6.2 KiB
references/model-experimentation.md8.2 KiB
references/provenance.md6.1 KiB

references/ml-checklists.md

title: ML checklists description: Topic-level summaries of the ML Fundamentals Checklist and ML Model Production Checklist, with their lifecycle scope and applicability caveats

Sources

Microsoft CSE Code-with-Engineering-Playbook, ML Fundamentals Checklist, documentation licensed CC BY 4.0.
Microsoft CSE Code-with-Engineering-Playbook, ML Model Production Checklist, documentation licensed CC BY 4.0.

Section headings are preserved as structural identifiers. Item content is not reproduced: each section carries a repository-original summary of what it covers and when it applies, and readers follow the upstream links for the item wording. Both pages are licensed CC BY 4.0; see THIRD-PARTY-NOTICES for the recorded attribution and usage scope. Explanatory framing, the lifecycle caveat, the readiness-domain grouping, and the routing table are repository-original.

ML Fundamentals Checklist

Six sections covering engagement fundamentals. Each summary below states what the section is for and when it applies. Read the upstream page for the item wording: https://microsoft.github.io/code-with-engineering-playbook/ml-and-ai-projects/ml-fundamentals-checklist/

Data Quality and Governance

Establishes that the data underpinning the engagement is obtainable, understood, and lawful to use. Work here before modelling begins, because an unresolved access, labelling, provenance, or compliance gap invalidates everything downstream.

Feasibility Study

Establishes whether the data can support the proposed task at all, and whether the expected return justifies the risk. Its conclusion is a documented recommendation on how to proceed. feasibility owns this workflow.

Evaluation and Metrics

Establishes how performance is measured, that the measure traces to agreed success criteria, and that one evaluation flow spans every model version. Treat evaluation code as production code: tested and peer-reviewed, and structured to support later results and error analysis.

Model Baseline

Establishes a specified, measured baseline so later models have something to be judged against. Without it, a performance claim has no reference point.

Experimentation setup

Establishes that experiments are comparable and repeatable: specified datasets, recorded hypotheses and results, systematic hyperparameter tuning, and consistent metrics across candidates. See model-experimentation.md for the tracking and reproducibility conventions.

Production

Establishes readiness to operate: reviewed models, a tested inferencing pipeline, agreed SLAs, monitoring of both data feeds and model output, consistent schemas across pipeline components, and a completed Responsible AI review. The production checklist below covers this ground in more depth.

ML Model Production Checklist

Lifecycle scope and caveat

Read this before applying the checklist. Upstream scopes it to teams that have already built or trained a model and are now considering putting it into production. Its stated purposes are confirming the model is ready for production before moving to scoring, and preparing a production plan.

Upstream also states that there may be scenarios where it is not possible to check every item, and advises going through all items and making informed decisions based on the specific use case.

Treat it as a structured readiness review with a lifecycle precondition, not as an unconditional gate.

Readiness domains

The upstream checklist works through the question of whether a model will behave in production as it did in training. Its items group into five domains. Read the upstream page for the item wording: https://microsoft.github.io/code-with-engineering-playbook/ml-and-ai-projects/ml-model-checklist/

Baseline and metric definition. Whether a specified baseline exists, whether the model beats it, whether performance metrics are settled for training and scoring, and whether the model has been measured against a benchmark.

Offline-to-online representativeness. Whether ground truth will be available or derivable once the model is live, and whether the distribution of data across training, testing, and validation sets has been examined. This is where a model that looks strong offline turns out not to be.

Integration and operating budgets. How the model joins the systems around it and what that integration affects, plus the targets and hard limits set for performance, prediction speed, and cost that give trade-offs something to be judged against.

Monitoring. How incoming data quality, shifts in data characteristics, and model performance are each monitored. dataops owns these.

Ethical review. Whether ethical implications have been considered. rai-planner owns this.

Routing

Some items are owned elsewhere.

Items	Route to
Incoming data quality monitoring, drift monitoring, and performance monitoring	`dataops`, which carries the validation-versus-drift distinction and its asymmetric remediation
Ethical concerns	`rai-planner`
Experiment setup, tracking, and evaluation flow	model-experimentation.md

A distinction worth preserving

The ML Fundamentals Checklist names reproducible, logged experiments. That is experiment reproducibility, which is not the same as pipeline replayability. Neither checklist states a pipeline-replayability requirement, and neither names a data-tiering model. Collapsing the two loses technical precision; pipeline replay semantics belong to dataops.

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/hve-core · .github/skills/data-science-engineering/ml-experimentation · commit f425e94

Licence

Creative Commons Attribution 4.0 International (redistribution allowed)

LICENSE (written by AutoGineer)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download ml-experimentation.zip