AutoGineer
Generated banner for AWS AI ML
Generated logo for AWS AI ML

AWS AI ML

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts
Download ZIP

From the author

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.

What it does

  • Guides users through the end-to-end lifecycle of SageMaker model customization.
  • Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
  • Provides workflows for validating and transforming training and evaluation datasets.
  • Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
  • Facilitates the deployment of models to SageMaker endpoints or Bedrock.
  • Diagnoses endpoint issues including latency, errors, and container logs.

When to use it

  • When planning a new model customization or deployment project on SageMaker.
  • When you need to validate or transform datasets for training or evaluation.
  • When selecting a base model or choosing a fine-tuning technique.
  • When you need to configure training, processing, or evaluation jobs.
  • When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
  • When setting up or managing a SageMaker Managed MLflow environment.

Advantages

  • Provides structured, step-by-step workflows for complex SageMaker tasks.
  • Ensures dataset formatting meets specific SageMaker requirements.
  • Offers clear routing to relevant documentation based on user intent.
  • Helps resolve configuration issues for IAM, S3, and SDK settings.
  • Simplifies the diagnosis of endpoint performance and failure logs.

Usage examples

“I need to fine-tune a model on SageMaker. Where do I start?”
The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
“My SageMaker endpoint is returning errors. Can you help?”
The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
“How do I convert my CSV training data for SageMaker?”
The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
“Which base model should I use for a text summarization task?”
The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/aws/aws-ai-ml/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills

Documentation by AWS

AWS AI/ML Model Customization

Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.

Routing

Match the user's intent to the appropriate reference folder and load only that content.

User intentReferenceWhen to use
Plan a model customization project, discover scope of work, resume or modify a planreferences/planning/User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference.
Define the business problem, success criteria, or use case specreferences/use-case-specification/User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines.
Select or change a base modelreferences/model-selection/User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation.
Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF)references/finetuning-technique/User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first.
Validate dataset quality and formatreferences/dataset-evaluation/User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job.
Transform or convert a dataset between formatsreferences/dataset-transformation/User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code.
Generate fine-tuning code and start trainingreferences/finetuning/User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers.
Train a model with a SageMaker Training jobreferences/training-jobs/Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis.
Process data with a SageMaker Processing jobreferences/processing-jobs/Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis.
Evaluate or benchmark a trained modelreferences/model-evaluation/User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer.
Deploy, benchmark, or optimize a model on an endpoint or Bedrockreferences/model-deployment/User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways.
Set up IAM roles, S3 buckets, SDK configurationreferences/sdk-getting-started/User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment.
Manage project directory and artifactsreferences/directory-management/Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory.
Set up, update, or delete a SageMaker Managed MLflow appreferences/manage-mlflow/User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected.
Diagnose a failing or unhealthy SageMaker endpointreferences/endpoint-diagnostics/User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments.

Rules

  • Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.

  • Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.

  • Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.

  • Usage attribution. Before running any AWS CLI command or packaged script, set export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.

Files

references/planning/overview.md

Principles

  • One question at a time. Each question should resolve a branching decision in the plan. Avoid generic or out-of-domain questions.

  • Surface constraints early. If a user decision would constrain downstream options, flag it before the plan is finalized.

  • Keep plans short. Only include tasks that are necessary for the user's stated goal.

  • Don't ask what you already know. Check conversation history and project files before asking the user.

Phase 1: Brainstorming

Goal: Understand what the user wants to accomplish and identify which references belong in the plan.

Read references/input-output-contracts.md, references/model-customization-plan.md, references/evaluate-first-plan.md, and references/deploy-base-model-plan.md to:

  • Determine whether the user's request aligns with an existing plan template. If exactly one matches, offer it. If multiple could apply, present the relevant options with brief pros/cons and ask the user to choose. If no template matches but the request involves multiple steps, propose a custom plan adapted to the user's needs. If the user declines a plan, proceed directly with their request using available references and general knowledge.

  • Identify which references could be relevant to the user's stated goal.

  • Check whether the user has the necessary input artifacts for each reference. If not, find the references that generate those inputs and add them first.

  • Order references to allow a smooth transition from one to the next and avoid dead ends.

  • Check if a recommended workflow matches the user's needs. If not, assess what modifications are needed and verify they are possible against the contracts table.

  • Decide which references in a matching workflow can be skipped.

  • Surface limitations early — if a user decision (model choice, region, evaluation method) would constrain downstream options, mention it proactively, get user feedback, and adapt the plan accordingly.

During brainstorming:

  • Serverless customization region gate (check FIRST). If the user's request involves fine-tuning or model customization, AND their region is already known (stated in their message, or stored in conversation context from a prior sdk-getting-started run), check it against the supported regions in references/region-availability.md. If the region is NOT supported, STOP immediately. Do NOT ask about evaluate-first vs direct fine-tuning. Do NOT generate any fine-tuning plan. Instead, tell the user the blocking message from references/region-availability.md (substituting their region).

    If the region is not yet known, continue to the workflow choice gate — the pre-execution region check below will enforce this constraint before any fine-tuning task runs.

  • Workflow choice gate: Before generating any plan, determine which workflow the user needs. There are three paths:

    1. Deploy a base model — the user wants to select and deploy a model from the catalog without fine-tuning.

    2. Evaluate-first — the user wants to evaluate a base model before deciding whether to fine-tune. Always present this option alongside direct fine-tuning, even if the user says "I want to fine-tune" — the user may not know evaluate-first exists.

    3. Direct fine-tuning — the user is committed to fine-tuning a model.

    Disambiguation for "deploy": If the user says "I want to deploy a model" (or similar), first determine whether they want to deploy a fine-tuned model they already have, or deploy a base model as-is:

    • If they already have a training job or fine-tuned model → this is NOT the deploy-base-model path. Proceed with the existing workflow and the model-deployment reference will handle it as a later step.

    • If they want to select and deploy a base model without fine-tuning → this IS the deploy-base-model path. Read references/deploy-base-model-plan.md.

    • If unclear and no prior context exists, ask: "Do you have a model you've already fine-tuned, or would you like to select and deploy a base model from the catalog?"

    If the user has explicitly chosen a path (e.g., "evaluate first", "skip evaluation", "deploy a base model", "I already fine-tuned"), proceed with their choice. Otherwise, present the relevant options with brief pros/cons and ask the user to choose. Saying "fine-tune" or naming a technique alone is NOT an explicit choice to skip evaluation — the user may not know evaluate-first is an option. Do NOT present a plan until the user has chosen a path. After they choose, read ONLY the corresponding reference plan.

  • Use the Restrictions column of the contracts table to flag constraints as soon as the relevant decision is made. Examples (non-comprehensive list, check contracts table for the full picture):

    • User picks a Nova model → alert that deployment regions are limited.

    • User picks a region → alert if it conflicts with model availability.

  • If a restriction applies, check whether it requires changes to other steps in the plan.

  • Do NOT ask the user about base model selection or preferences. Model selection is handled exclusively by the model-selection reference.

  • Move to Phase 2 as soon as you can determine which references and tools the plan needs.

Phase 2: Plan Generation

Goal: Propose a structured plan for the user to review.

Generate a plan as a numbered list of tasks. Each task has:

  • A short name

  • A one-sentence description of what happens

  • Which reference handles it (if applicable)

Format:

Based on what you've described, here's what I propose:

1. ⬜ **[Task Name]** — [What happens]. *(Reference: [reference-name])*
2. ⬜ **[Task Name]** — [What happens]. *(Reference: [reference-name])*
3. ⬜ **[Task Name]** — [What happens]. *(Reference: [reference-name])*

Does this plan look right, or would you like to change anything?

Rules for plan generation:

  • Infer ordering from the Prerequisites column in the contracts table — a skill cannot appear before its prerequisites. If unsure, consult references/skill-routing-constraints.md.

  • Prefer existing references when they cover the user's need. When no reference matches part of the request, proceed with general knowledge while respecting skill boundaries.

  • Tailor the plan to the user's actual intent. Not every plan needs every reference.

  • If the user already has input artifacts (e.g., a trained model), skip the steps that produce them.

When the user approves the plan, write it to PLAN.md and save it under the project directory structure defined by the directory-management reference.

# Plan

1. ⬜ **[Task Name]** — [Description]. _(Reference: [reference-name])_
2. ⬜ **[Task Name]** — [Description]. _(Reference: [reference-name])_
3. ⬜ **[Task Name]** — [Description]. _(Reference: [reference-name])_

Status indicators:

  • ⬜ Not Started

  • 🔄 In Progress

  • ✅ Completed

Update PLAN.md whenever a task's status changes.

Phase 3: Plan Iteration

Goal: Refine the plan until the user approves it.

  • If the user suggests changes, regenerate the plan incorporating their feedback.

  • If the user approves, begin execution by handing off to the first task's reference.

Execution

Once the plan is approved:

  1. Pre-execution region check (MANDATORY for fine-tuning plans). If the plan includes any fine-tuning step, immediately check the user's region by running:

    python -c "import boto3; print(boto3.session.Session().region_name)"

    If the result is None, skip this check and continue — the sdk-getting-started skill will prompt the user to set a region later. Otherwise, check the region against the supported regions in references/region-availability.md. If it is NOT supported → STOP. Do NOT start executing the plan. Tell the user the blocking message from references/region-availability.md (substituting their region). If it IS supported → briefly confirm to the user that serverless model customization is available in their region (e.g., "Your region (<region>) supports serverless model customization ✅") and continue.

  2. Before starting a task, update its status in PLAN.md to 🔄 (In Progress).

  3. If the task maps to a reference, load that reference's overview.md before doing any work. Do not attempt the task from general knowledge — always defer to the reference's instructions.

  4. Execute the task by following the loaded reference's workflow.

  5. When the task completes:

    • Update its status in PLAN.md to ✅ (Completed). If the task generated output files (scripts, notebooks, manifests), record the file paths under the completed task:

      - [x] Fine-tune model
  - Output: `scripts/01_sft_finetuning.py`
  - Output: `manifests/sft-llama-20260515.json`

    • Briefly confirm completion and move to the next task.

  6. If the user interrupts with a new request mid-execution:

    • Completed tasks are immutable — do NOT modify them.

    • Regenerate the remaining tasks to incorporate the user's new input.

    • Present the updated remainder for approval before continuing.

Plan Completion

When all tasks in the plan are done: Present to the user:

"We've completed everything in the plan. What would you like to do next?"

This re-enters Phase 1 (Brainstorming) for a new goal. There is no terminal state — the conversation continues as long as the user wants.

References

Load the reference plan that matches the customer's intent, then adjust based on their needs.

  • references/evaluate-first-plan.md — The evaluate-first workflow: evaluate a base model before deciding whether to fine-tune.

  • references/model-customization-plan.md — The direct fine-tuning plan. Use when the user has explicitly committed to fine-tuning.

  • references/deploy-base-model-plan.md — The deploy-base-model workflow: select and deploy a base model without fine-tuning.

  • references/input-output-contracts.md - A table showing all references, required inputs, produced outputs, prerequisites, and constraints.

  • references/skill-routing-constraints.md — Optional supplemental resource about Mandatory inclusion rules, ordering constraints, and skill boundary rules.

  • references/region-availability.md — Single source of truth for the regions where serverless model customization is available, plus the blocking message to use when a region is unsupported.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip