Data Science Engineering Foundation
MicrosoftOfficialPublished by Microsoft
- Skill
- Meta (authoring, orchestration, planning, memory)
- Agent tooling
- CC-BY-4.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The Data Science and Engineering Foundation serves as the core orchestration layer for the Data Science and Engineering Coach. It provides a centralized set of protocols for managing session state, job lifecycles, and flow-state mechanics. This foundation ensures consistency across different coaching sessions and job transitions by defining how the coach should initialize, resume, and navigate between various tasks. It acts as an internal reference index rather than a user-facing tool, enforcing rules for when to interrupt, how to handle durable writes, and how to maintain session integrity.
What it does
- Defines protocols for session initialization, state reconstruction, and resumption.
- Provides a registry for routing tasks to specific skills and specialists.
- Implements a flow-state protocol to manage interruptions and user gates.
- Enforces a durable-write gate to scan for sensitive content before creating artifacts.
- Maintains clear boundaries between general orchestration and job-specific logic.
- Standardizes the lifecycle management of coaching jobs.
When to use it
- When the coach needs to initialize or resume a session.
- When transitioning between different data science or engineering jobs.
- Before performing a durable write of a customer artifact.
- When validating the state of a coaching session.
- When determining the appropriate skill or specialist to handle a specific task.
Advantages
- Ensures consistent behavior across different coaching sessions.
- Provides a structured approach to handling sensitive data during artifact creation.
- Decouples orchestration logic from job-specific implementation details.
- Improves reliability by enforcing clear state and lifecycle rules.
- Facilitates modularity by routing tasks to specialized skills.
Usage examples
- “Initialize the coaching session for a new data engineering project.”
- The coach loads the session-state protocol, validates the project identity, and prepares the environment for the selected job.
- “I want to save this new data catalog document.”
- The coach triggers the flow-state protocol, performs a sensitive content scan on the document, and proceeds with the write only if the scan passes.
- “Switch from the current feasibility study to a new pipeline design task.”
- The coach follows the transition protocol to pause the current job, confirm the switch with the user, and route to the dataops skill.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/data-science-engineering-foundation/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip data-science-engineering-foundation.zip -d .agents/skills
Documentation by Microsoft
Data Science and Engineering Foundation
Goal
Keep Data Science and Engineering Coach orchestration consistent across job changes and sessions without duplicating job-specific guidance. The coach loads this index at initialization and resume, then reads the reference for the current coaching moment.
Reference index
| Reference | When to read |
|---|---|
| job-registry.md | Before offering or selecting a job, routing to a skill or specialist, or naming a durable output |
| lifecycle-classes.md | When starting, pausing, resuming, completing, or re-invoking a job |
| transition-protocol.md | When a topic shift, explicit request, or completion suggests moving between jobs |
| session-state.md | Before initialization, validation, mutation, resume, recovery, or reconstruction of coaching state |
| flow-state.md | Before interrupting work, crossing a gate, writing a durable artifact, or offering post-job choices |
Success criteria
Each orchestration rule has one owner in this package.
State and lifecycle mechanics remain independent of job-specific methods.
The coach reads the applicable reference before acting on its contract.
Constraints
Keep catalog guidance in
data-catalog, the durable data-catalog workflow for entities, declared relationships, lineage, and ERD-ready model semantics.Keep feasibility guidance in
feasibility, the evidence-led data and ML feasibility-study workflow with lifecycle and interchange traceability.Keep pipeline and testing guidance in
dataops, the DataOps reference for tier behavior, pipeline invariants, validation placement, tests, drift, and operational signals.Keep general experiment guidance in
experiment-design, the reusable workflow for candidate selection, hypotheses, vetting, minimum scope, and result evaluation.Keep ML-specific experiment guidance in
ml-experimentation, the reference for ML environments, reproducibility, tracking, evaluation, abstractions, and production readiness.Keep notebook and dashboard guidance in
analysis-authoring, the reference for EDA notebook and analytical dashboard composition, visualization selection, and dashboard validation.Keep AI-system evaluation guidance in
evaluation-design, the reference for evaluation dataset design, difficulty balance, metric selection, and tooling fit.Treat this package as internal foundation knowledge, not a user-selectable workflow.
Preserve the skill authority boundaries and cross-cutting concerns defined in the job registry.
Stop rules
Stop before a state mutation when the session-state protocol has not been loaded or project identity is uncertain.
Stop before a job switch when the user has not confirmed the transition.
Stop and report the missing owner when requested behavior belongs to neither this foundation nor a registered job capability.
Attribution
This package is repository-original orchestration guidance licensed under CC BY 4.0. It does not reproduce or summarize an external standard.
Files
- ATTRIBUTION.mdAdded by AutoGineer1021 B
- LICENSEAdded by AutoGineer16.6 KiB
- SKILL.md3.9 KiB
- references/flow-state.md5.3 KiB
- references/job-registry.md17.8 KiB
- references/lifecycle-classes.md4.0 KiB
- references/session-state.md6.9 KiB
- references/transition-protocol.md3.0 KiB
references/job-registry.md
title: Data Science and Engineering Job Registry description: Authoritative job names, lifecycle classes, skill and specialist routes, and durable outputs for Data Science and Engineering Coach sessions
Data Science and Engineering Job Registry
Purpose
Use this registry before offering or selecting work. A job is a user-confirmed unit of focus inside one continuing coaching session. The registry routes work; it does not duplicate the methods owned by a skill.
Registry
| Job | Class | Primary route | Optional supporting route | Durable output or completion evidence |
|---|---|---|---|---|
catalog | continuous | data-catalog: durable catalog entities, declared relationships, lineage, and model semantics | analysis-authoring when a presented dataset profile is the requested output | Caller-approved data catalog |
model-diagram | episodic | data-catalog: authoritative declared entities and relationships | architecture-diagrams: render the declared model without becoming semantic authority | Mermaid or ASCII diagram |
problem-framing | episodic | experiment-design: candidate data-science problem classes for a stated business outcome, with entry-test reasoning | None | Candidate framing set with reasoning, unknowns, and gaps |
feasibility | bounded | feasibility: evidence-led feasibility studies with durable traceability | None | Caller-approved feasibility study |
pipeline | episodic | dataops: tier behavior, pipeline invariants, validation, tests, and drift | None | Transformation, validation, or pipeline code |
analysis | episodic | analysis-authoring: EDA notebook and analytical dashboard authoring and validation | data-catalog for column semantics; dataops for derived-dataset persistence | Notebook, dashboard, or analysis deliverable |
evaluation | episodic | evaluation-design: AI-system evaluation dataset design, metrics, and tooling | rpi-research only for a demonstrated live evaluator-catalog fact gap | Evaluation dataset with curation, metric, and tooling documents |
experiment | episodic | experiment-design: experiment selection, hypotheses, vetting, scope, and evaluation | ml-experimentation: ML reproducibility, tracking, evaluation, abstractions, and production readiness | Hypothesis, experiment assets, and result disposition |
testing | episodic | dataops: DataOps and DS/MLOps test techniques | analysis-authoring for dashboard validation technique | Test code and assertions |
observability | episodic | dataops: data/model signals and validation-versus-drift guidance | None | Instrumentation code and signal recommendations |
Optional supporting routes in the registry are job-specific skill support and do not limit the separate segment-level Research eligibility defined by the RPI depth matrix.
RPI depth matrix
RPI is a user-directed depth route inside the confirmed job, not a job or lifecycle. Keep simple or adequately evidenced work on the primary route. No RPI segment auto-activates. Before each eligible segment, state the demonstrated need, purpose, expected artifact, expected interaction cost, limits, and direct path available if the user skips it.
| Job | Research posture | Execution-loop eligibility | RPI does not own |
|---|---|---|---|
catalog | None by default | None by default | Catalog identity, semantics, enrichment, or continuous lifecycle |
model-diagram | None by default | None by default | Declared relationships, diagram format, or episodic completion |
problem-framing | analysis mode for a named current-fact or prior-art gap; preserve alternatives and unknowns | None by default | Selecting, ranking, or scoring a problem class |
feasibility | Bounded current-fact evidence that the feasibility study cannot establish locally | None by default | Feasibility verdict, study lifecycle, or recommendation authority |
pipeline | Current platform, compatibility, standard, or prior-art facts identified by dataops | Full Plan, Implement, and Review for substantial delivery | Tier semantics, pipeline invariants, validation placement, or scan authority |
analysis | Current library, platform, accessibility, or analytical-method facts identified by analysis-authoring | Full Plan, Implement, and Review for substantial delivery | Analytical intent, visualization choice, dashboard validation, or scan authority |
evaluation | Current evaluator names, availability, preview state, compatibility, and prerequisites identified by evaluation-design | Full Plan, Implement, and Review for substantial delivery | Metric selection, dataset design, tooling fit, or scan authority |
experiment | Inherit the Experiment Designer convergence Research contract | Inherit the Experiment Designer post-design execution loop | Hypotheses, criteria, experiment outcome, or partner commitments |
testing | Current tool, framework, compatibility, or prior-art facts identified by the owning skill | Full Plan, Implement, and Review for substantial delivery | Test intent, assertion authority, or scan authority |
observability | Current platform, telemetry, compatibility, or prior-art facts identified by the owning skill | Full Plan, Implement, and Review for substantial delivery | Signal-selection intent, telemetry vocabulary, drift interpretation, or scan authority |
Substantial delivery means multi-step code or artifact production whose dependencies, validation, or interruption risk make direct episodic execution unreliable. The owning skill first accepts the domain design. RPI then plans and executes delivery and reviews conformance without changing the job, class, owning skill, coach state authority, output root, or durable-write gate.
RPI Researcher may run only as an internal helper to an active rpi-research phase for one bounded source-gathering lane. Its return is not an independent job, invocation, state record, evidence authority, or recommendation. The active Research phase verifies selected sources and exclusively owns canonical evidence IDs, findings, recommendations, and the primary artifact.
The cross-cutting concerns below are evaluated in addition to the routes in this table. They are not listed per job, because they apply by trigger rather than by job name.
Cross-cutting concerns
Privacy, Responsible AI, and telemetry are evaluated across every job rather than being optional per-job routes. Evaluate each concern whenever its trigger is present, name the concern to the user, and route to its owning skill for the contribution that skill owns.
| Concern | Trigger to evaluate | Owning skill | Contribution |
|---|---|---|---|
| Privacy | Personal, sensitive, or regulated data is catalogued, moved, transformed, sampled, analyzed, presented, or persisted | privacy-standards | Sensitivity classification, citation fields, DPIA thresholds, and data-flow reasoning |
| Responsible AI | An AI or ML system is designed, evaluated, or experimented on | rai-standards, then rai-planner when a risk surfaces | Risk framing and standards mapping, plus a scoped assessment when a surfaced risk warrants one |
| Telemetry | Signals, metrics, traces, or logs are emitted, named, or instrumented, including drift and pipeline health signals | telemetry-foundations | Metric names, instruments, units, cardinality limits, and PII-safe telemetry conventions |
A cross-cutting concern always surfaces an observation and offers concrete choices. It never blocks a durable write, never edits an artifact silently, and never decides for the user. The durable-write scan gate remains the only condition that blocks a write.
When a concern triggers, state which concern applies and why, name its owning skill, offer the contribution that skill can make, and let the user choose whether to take it. When the owning skill is unavailable in the installed collection, say so and continue without improvising its authority.
Skill authority boundaries
The Data Science and Engineering Coach routes to the data science and engineering skills below, plus the cross-cutting skills above. Keep their authority separate even when one job loads more than one skill.
| Exact skill name | Capability description | Does not own |
|---|---|---|
data-catalog | Durable catalog entities, declared relationships, identity, coverage, lineage, and attached dataset profiles | Pipeline tier behavior, inferred diagram relationships, or feasibility verdicts |
dataops | DataOps tier semantics, pipeline invariants, validation, tests, drift, signals, and derived-dataset persistence | Catalog meaning, experiment selection, ML tracking setup, or sensitivity classification |
feasibility | Evidence-led feasibility studies, lifecycle, recommendations, and interchange traceability | Functional requirement numbering, downstream writeback, catalog semantics, or implementation |
analysis-authoring | EDA notebook and analytical dashboard composition, visualization selection, and dashboard validation | Column semantics, persistence format, accessibility conformance, or evaluation dataset design |
evaluation-design | AI-system evaluation dataset design, difficulty balance, metric selection, and tooling fit | Trained-model evaluation, Responsible AI approval, or session and durable-write mechanics |
experiment-design | Problem-class framing for a stated business outcome, plus experiment selection, hypotheses, vetting, minimum scope, and result evaluation | ML infrastructure, production implementation, or pipeline mechanics |
ml-experimentation | ML environments, reproducibility, tracking, abstractions, evaluation, and readiness | General hypothesis framing, pipeline replay, or Responsible AI approval |
privacy-standards | Sensitivity classification, data-flow reasoning, standards mapping, and DPIA thresholds | Catalog semantics, pipeline mechanics, or durable-write authority |
rai-standards | Responsible AI standards, risk framing, and standards mapping | Evaluation dataset design, model performance, or approval authority |
rai-planner | Scoped Responsible AI assessment when a surfaced risk warrants one | Data science method authority, session mechanics, or approval authority |
telemetry-foundations | Metric names, instruments, units, cardinality, and PII-safe telemetry conventions | Signal selection intent, pipeline behavior, or drift interpretation |
The two Responsible AI authorities are sequential, not alternatives.
rai-standards provides risk framing and standards alignment and is evaluated
first whenever an AI or ML system is in scope. rai-planner provides a scoped
assessment of one surfaced risk and is reached only after that risk is
identified.
problem-framing and experiment both route to experiment-design and stay
distinct. problem-framing asks what kind of data-science problem a stated
business outcome is, and completes without choosing one. experiment assumes a
direction and selects what to test.
Problem-framing completion evidence
This job has two completion shapes and neither is an abandonment.
Normal completion records the candidate problem classes, the grounded-versus- original reasoning behind each, unresolved discriminating questions marked unknown, and gaps carrying an answer holder and a coordinator. No candidate is selected, ranked, or scored.
An unstated business outcome completes with a single outcome-definition gap, its answer holder, its coordinator, and an explicit statement that no candidate analysis was performed. Record it as completed work, not
discarded-cleanly.
Completion returns control to the coach. A feasibility or experiment job starts only after a separate user-confirmed transition.
The two evaluation authorities are adjacent and must not be blended.
evaluation-design covers systems whose output is a response, such as
assistants and agents. ml-experimentation covers a trained model's predictive
performance.
Selection rules
Present the relevant jobs with their class and expected output.
Ask the user to choose; do not infer or activate a job silently.
Record the confirmed job and its class using the session-state protocol.
Load the primary skill before job-specific guidance.
Load a supporting route only for the concern it owns, then reconcile the result into the session artifact list without transferring authority to it.
When a capability is unavailable in the installed collection, state the gap and offer an available route. Do not improvise a missing capability.
Routing boundary
A supporting route contributes within its own authority. The coach retains session identity, job selection, transition decisions, state mutation, pause and resume, durable-write gates, and completion choices. Loading a supporting skill never changes the active job without user confirmation.
The RPI depth matrix is the only job-specific Research and execution-routing authority. Every route remains subordinate to the active job's primary skill, lifecycle class, coach state, transition protocol, and durable-write gate.
Provenance
This registry is repository-original guidance licensed under CC BY 4.0. It does not reproduce or summarize an external standard.
Requirements and compatibility
Written for: GitHub Copilot, Claude Code
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Claude Code frontmatter
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Microsoft
- Publisher
- Microsoft (Official)
- Source
- microsoft/hve-core · .github/skills/data-science-engineering/data-science-engineering-foundation · commit f425e94
- Licence
- Creative Commons Attribution 4.0 International (redistribution allowed)
- LICENSE (written by AutoGineer)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download data-science-engineering-foundation.zip