Dataops

MicrosoftOfficialPublished by Microsoft

Skill
Guidelines
Software development
CC-BY-4.0
Low risk
Has scripts

From the author

DataOps and DS/MLOps testing reference for data tiering, Bronze-to-Silver validation placement, pipeline invariants, pytest categories, and validation-versus-drift. Use when designing, reviewing, or generating data pipelines, transformation code, data validation, or data-science test suites.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The dataops skill provides a structured reference for designing and reviewing data pipelines and machine learning workflows. It synthesizes engineering best practices regarding data quality tiers (Bronze, Silver, Gold), the strategic placement of validation logic, and the importance of maintaining faithful source copies for pipeline replayability. The skill helps teams distinguish between data validation and drift detection, ensuring that data-heavy systems are robust, testable, and maintainable.

What it does

Defines data tier semantics for Bronze, Silver, and Gold storage layers.
Provides rules for placing validation at the Bronze-to-Silver boundary.
Explains the rationale for maintaining immutable, faithful source copies in the Bronze tier.
Offers guidance on distinguishing between data validation and drift detection.
Provides a framework for testing data science and MLOps code.
Includes a contract for synthetic data operations to ensure safe preflight validation.

When to use it

When designing the architecture of a new data pipeline or lakehouse.
When reviewing data transformation code or pipeline test suites.
When deciding where to implement data validation checks.
When troubleshooting pipeline failures that require data replay or recovery.
When establishing testing strategies for machine learning models and data loading processes.

Advantages

Ensures consistent application of data quality and pipeline invariants.
Promotes robust recovery strategies through faithful data ingestion.
Clarifies the distinction between validation and drift for better observability.
Provides clear, attributed guidance based on established engineering playbooks.
Prevents common pitfalls like premature validation before data landing.

Usage examples

“Where should I place validation logic in my ETL pipeline?”: The skill advises placing validation at the Bronze-to-Silver boundary, explaining that Bronze should remain a faithful, immutable copy of the source to support replayability and recovery.
“What is the difference between data validation and drift detection?”: The skill provides a breakdown distinguishing data validation (schema and invariant checks) from drift detection (monitoring changes in data or model signals over time).

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/dataops/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip dataops.zip -d .agents/skills
```

Documentation by Microsoft

DataOps Reference Pack

Goal

Ground pipeline and test generation in the Microsoft CSE engineering playbook so that data tier semantics, validation placement, recovery invariants, and DS/MLOps test technique are applied consistently and attributed accurately.

Inputs

The pipeline, transformation, validation, or test work under discussion
The tier of each dataset involved, when the consuming workflow records one
Existing test layout, package structure, and data-access boundaries
A data classification produced elsewhere, when sensitivity matters

Reference index

Read only the reference that matches the active concern.

Reference	Read this when
data-tiers-and-pipeline-invariants.md	Assigning tier meaning, placing validation, routing malformed records, or asserting replay, idempotency, testability, source-control, and configuration invariants
testing-data-science-and-mlops.md	Writing or reviewing tests for data loading, transformation, model load or predict, data validation, or model robustness
validation-drift-and-observability.md	Distinguishing data validation from drift detection, choosing remediation, or deciding which data and model signals matter
synthetic-data-operation-contract.md	Validating synthetic-data preflight authority, field lineage, conditional subgroup evidence, result linkage, or guarded local replacement
provenance.md	Confirming what is upstream guidance, what is HVE Core derivation, and where upstream is silent

Success criteria

Tier language distinguishes the three upstream quality tiers from the additional storage areas.
Validation is placed at the Bronze-to-Silver boundary, and the faithful-copy rationale with both replay purposes travels with that placement whether or not the request challenges it.
Test guidance names the operation category, its technique, and where mocking stops.
Validation and drift keep their distinct definitions and their distinct remediation paths.
Every claim traces to an attributed upstream source or is labelled as HVE Core guidance.
Synthetic-data generation stops before source access or writes unless its applicable SYNTHETIC_DATA_OPERATION_V1 preflight passes deterministic validation.

Constraints

This skill generates code, assertions, and review guidance. It does not execute pipelines, transformation engines, or telemetry backends.
Reproduce only the minimum text necessary for a specific technical point, and paraphrase everything else. Attribute every reference and describe accurately what each reference reproduces.
Label HVE Core derivations as such. Do not present a derived consequence or a repository convention as upstream guidance.
Where upstream is silent, say so rather than inventing an upstream-sounding rule.

Ownership boundaries

This skill decides which data and model signals matter. It does not own the vocabulary, the classification, or the surrounding workflow.

Concern	Owner
Metric names, instrument types, units, cardinality discipline, and the PII emission denylist	`telemetry-foundations`
Data sensitivity classification and DPIA thresholds	`privacy-standards`
Entity semantics, relationships, and which tier a dataset is recorded as	The calling workflow
Feasibility assessment and go/no-go recommendation	The calling workflow

No repository artifact currently owns the last two rows. When the caller supplies neither, state the gap rather than deciding tier assignment or feasibility here.

This skill never decides what is sensitive. It reads a classification produced elsewhere.

For synthetic-data operations, read synthetic-data-operation-contract.md and validate records against synthetic-data-operation-v1.schema.json. The contract records qualified decisions by immutable reference; it does not store source values or replace the authority of data, privacy, Responsible AI, fairness, or domain owners.

Stop rules

Stop and route metric naming, units, and cardinality to telemetry-foundations, the OpenTelemetry-aligned vocabulary and instrumentation skill. Route data sensitivity to privacy-standards, the privacy classification and DPIA-threshold reference.
Stop and state the gap when the request depends on guidance the playbook does not provide, such as a drift threshold or an alerting policy.
Stop and offer the correct placement when asked to validate before Bronze landing, rather than complying or refusing without an alternative.

Attribution

This pack declares CC-BY-4.0 because both bodies of content it holds carry that license.

Source pages are Microsoft CSE Code With Engineering Playbook documentation licensed CC BY 4.0. The upstream project applies MIT through a separate LICENSE-CODE file to code samples only, which this pack does not reproduce. The references derive from those documentation pages and have been changed: upstream guidance is paraphrased, and only identifiers and structural names are carried across as facts. THIRD-PARTY-NOTICES carries the attribution CC BY 4.0 requires and states that the content has been changed. Each reference cites its own upstream URL and states what it reproduces.

Content labelled as HVE Core derivation is repository-original material under CC BY 4.0. See provenance.md for the consolidated source map and derivation labels.

Files

ATTRIBUTION.mdAdded by AutoGineer965 B
LICENSEAdded by AutoGineer16.6 KiB
SKILL.md7.6 KiB
assets/synthetic-data-operation-v1.schema.json9.2 KiB
pyproject.toml681 B
references/data-tiers-and-pipeline-invariants.md10.2 KiB
references/provenance.md10.8 KiB
references/synthetic-data-operation-contract.md6.4 KiB
references/testing-data-science-and-mlops.md8.8 KiB
references/validation-drift-and-observability.md7.2 KiB
scripts/synthetic_data_operation.pyScript21.2 KiB
tests/corpus/0_empty0 B
tests/corpus/1_empty_object2 B
tests/corpus/2_non_object_json2 B
tests/corpus/3_minimal_preflight27 B
tests/fuzz_harness.pyScript1.2 KiB
tests/test_synthetic_data_operation.pyScript23.9 KiB
uv.lock101.9 KiB

references/synthetic-data-operation-contract.md

title: Synthetic Data Operation Contract description: Versioned preflight, lineage, subgroup-evidence, result, and guarded local replacement contract for synthetic-data generation author: Microsoft ms.date: 2026-09-09 ms.topic: reference

Contract Boundary

SYNTHETIC_DATA_OPERATION_V1 is the deterministic authorization and evidence boundary for synthetic-data generation. It uses an immutable preflight record before any project write, package installation, notebook creation, source access, or generation. A separate result record reports observed behavior and links to the exact preflight revision.

The contract does not classify data, select protected attributes or affected groups, set quality thresholds, determine fairness, or authorize replacement. Qualified owners make those decisions in durable source records. This contract stores stable references, immutable revisions, owner roles, provenance references, artifact digests, review state, applicability, and current status so deterministic validation can reject missing or stale authority.

Validate records with synthetic-data-operation-v1.schema.json and the validate command in scripts/synthetic_data_operation.py.

Preflight Record

A preflight record declares one operation before execution:

Stable operation and immutable record revision identifiers
Intended purpose and new-output or replace-local mode
A non-secret source reference, expected SHA-256 digest when available, and caller-owned tier
Strict decision references for classification, protected-attribute applicability, subgroup evaluation, and replacement authority
Planned field lineage for every output field
Qualified activated subgroup references
A passed or blocked gate with stable reason categories

Decision records use these qualified roles: data-owner, privacy-owner, rai-fairness-owner, and domain-owner. Each decision reference may name more than one role, but roles and provenance references are unique.

Qualified Branch Rules

The schema validates structure. The validator applies these branch rules:

Source access requires an approved, current, applicable classification decision with data-owner authority.
Protected-attribute generation is active when its decision is applicable. It requires approved and current privacy-owner, rai-fairness-owner, and domain-owner authority.
Subgroup evaluation is active when its decision is applicable. It requires approved and current rai-fairness-owner and domain-owner authority plus at least one qualified activated subgroup reference.
replace-local requires approved, current, applicable data-owner replacement authority and an expected source digest.
A current qualified not-applicable decision is valid only for an optional inactive branch. It never satisfies an activated branch.

Any missing, pending, rejected, superseded, withdrawn, or incorrectly inapplicable required decision blocks the operation. A non-approved review state carries at least one stable review-reason category.

Field Lineage

Field lineage identifies each output field exactly once and records one disposition:

copied: value comes directly from a source field
transformed: source value is changed by a named transformation
generated: value comes from a named generator without a source value
derived: value is computed from one or more source or generated values
dropped: planned source field is deliberately absent from output

transformed and derived entries require a transformation reference. Other dispositions omit it unless a reviewed contract revision adds a new semantic rule. Records carry references and identities, not raw field values.

Result Record

A result record references the exact preflight revision and records:

Observed source and output SHA-256 digests
Actual field lineage
One explicit result for every activated subgroup
Validation checks using passed, failed, insufficient, not-measured, or not-applicable
Commit state and rollback-available, unchanged-original, committed-new-output, external-change-preserved, or not-applicable evidence

For replace-local, the caller supplies a not-attempted result revision and a contained --final-result destination. A successful commit publishes a distinct immutable successor result at that destination. Its previous_result_revision identifies the supplied result, while preflight_revision retains the exact authorizing preflight. The five-field command summary references, but does not replace, this full result.

Missing subgroup evidence is never success. Every activated subgroup has an explicit passed, failed, insufficient, or not-measured result. not-applicable is valid only when the qualified preflight did not activate subgroup evaluation.

Local Replacement State Model

New versioned output is the default. replace-local supports one existing regular file under a caller-approved local root. Before replacement, the commit adapter:

Revalidates the result and matching preflight.
Rechecks the target SHA-256 digest against the approved precondition.
Creates and synchronizes a recoverable predecessor.
Stages and synchronizes candidate bytes beside the target.
Validates commit readiness, then performs one local replacement.

Every failure before the replacement leaves the entry target bytes unchanged and records unchanged-original evidence, except an observed external target change, which records external-change-preserved and does not overwrite those newer bytes. The adapter rechecks the target and predecessor digests immediately before replacement. This narrows but does not eliminate the user-space check-to-replace race; the local filesystem offers no compare-and-swap guarantee here.

A committed replacement records the target digest and recoverable predecessor evidence. The target replacement is the canonical commit. Successor-result publication follows it and is not atomic with it; publication failure reports target and predecessor digests and retains the predecessor for recovery. Directories, links, remote stores, network shares, databases, APIs, and multi-target publication are unsupported.

Evolution

Unknown properties are rejected. Adding an optional property requires a reviewed schema revision. Changing semantics, required fields, or enumerated states requires a new contract version.

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter, Needs Python

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/hve-core · .github/skills/data-science-engineering/dataops · commit f425e94

Licence

Creative Commons Attribution 4.0 International (redistribution allowed)

LICENSE (written by AutoGineer)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download dataops.zip