Generated banner for AWS AI ML
Generated logo for AWS AI ML

AWS AI ML

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts

From the author

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.

What it does

  • Guides users through the end-to-end lifecycle of SageMaker model customization.
  • Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
  • Provides workflows for validating and transforming training and evaluation datasets.
  • Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
  • Facilitates the deployment of models to SageMaker endpoints or Bedrock.
  • Diagnoses endpoint issues including latency, errors, and container logs.

When to use it

  • When planning a new model customization or deployment project on SageMaker.
  • When you need to validate or transform datasets for training or evaluation.
  • When selecting a base model or choosing a fine-tuning technique.
  • When you need to configure training, processing, or evaluation jobs.
  • When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
  • When setting up or managing a SageMaker Managed MLflow environment.

Advantages

  • Provides structured, step-by-step workflows for complex SageMaker tasks.
  • Ensures dataset formatting meets specific SageMaker requirements.
  • Offers clear routing to relevant documentation based on user intent.
  • Helps resolve configuration issues for IAM, S3, and SDK settings.
  • Simplifies the diagnosis of endpoint performance and failure logs.

Usage examples

“I need to fine-tune a model on SageMaker. Where do I start?”
The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
“My SageMaker endpoint is returning errors. Can you help?”
The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
“How do I convert my CSV training data for SageMaker?”
The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
“Which base model should I use for a text summarization task?”
The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/aws/aws-ai-ml/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills

Documentation by AWS

AWS AI/ML Model Customization

Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.

Routing

Match the user's intent to the appropriate reference folder and load only that content.

User intentReferenceWhen to use
Plan a model customization project, discover scope of work, resume or modify a planreferences/planning/User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference.
Define the business problem, success criteria, or use case specreferences/use-case-specification/User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines.
Select or change a base modelreferences/model-selection/User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation.
Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF)references/finetuning-technique/User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first.
Validate dataset quality and formatreferences/dataset-evaluation/User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job.
Transform or convert a dataset between formatsreferences/dataset-transformation/User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code.
Generate fine-tuning code and start trainingreferences/finetuning/User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers.
Train a model with a SageMaker Training jobreferences/training-jobs/Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis.
Process data with a SageMaker Processing jobreferences/processing-jobs/Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis.
Evaluate or benchmark a trained modelreferences/model-evaluation/User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer.
Deploy, benchmark, or optimize a model on an endpoint or Bedrockreferences/model-deployment/User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways.
Set up IAM roles, S3 buckets, SDK configurationreferences/sdk-getting-started/User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment.
Manage project directory and artifactsreferences/directory-management/Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory.
Set up, update, or delete a SageMaker Managed MLflow appreferences/manage-mlflow/User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected.
Diagnose a failing or unhealthy SageMaker endpointreferences/endpoint-diagnostics/User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments.

Rules

  • Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.

  • Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.

  • Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.

  • Usage attribution. Before running any AWS CLI command or packaged script, set export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.

Files

references/model-selection/overview.md

Model Selection

Guides the user through selecting a base model based on their use case.

When to Use

  • User asks which model to use

  • User wants to select or change their base model

  • User mentions a model name or family (e.g., "Llama", "Mistral", "Nova") — the exact Hub model ID still needs to be resolved

  • User wants to evaluate a base model before deciding whether to finetune

Prerequisites

None — this skill can be entered directly.


Routing Rules
  1. Ambiguity gate: If it is unclear whether the user wants to fine-tune a model or deploy a base model as-is, you MUST ask before proceeding. Do not assume either path. Do NOT recommend, list, or name any models until this is resolved.

  2. Recommend use-case-spec first: If no *_use_case_spec.md file exists for this use case, recommend creating one via references/use-case-specification/overview.md before filtering. This produces better results because the spec captures constraints that map to concrete filter values. If the user declines or provides a specific model name/ID, proceed without it. After making this recommendation, stop and wait for the user's response. Do not continue with model selection in the same turn.

  3. Base model filtering MUST use select-for-deployment: When making a final model selection or recommendation for deployment, you MUST follow references/select-for-deployment.md and use its scripts. SageMakerPublicHub uses get_deployable_models.py for a selection-session catalog snapshot and search_deployable_models.py for confirmed candidates; private Hubs keep get_deployable_models.py plus filter_deployable_models.py. Do not make a final model selection or recommendation for deployment ad hoc from your own knowledge.

  4. Offer to create a use-case spec after filtering: If filtering was done without a spec, and no *_use_case_spec.md file exists for this use case, offer to create one to refine the criteria for future iterations.

  5. No premature recommendations: You may provide an initial quick-filtered list of models based on what you know, but do NOT make a recommendation or suggest a specific model as "the best choice" until the full selection workflow (select-for-deployment scripts or Step 3A) has been followed. If you have NOT run get_deployable_models.py or get_model_names.py in this session, you MUST prefix any model list with:

⚠️ This is a preliminary list based on general knowledge. I have not yet queried all available models in your Hub. There may be better matches once we run the full selection workflow.

NEVER present a model list without stating whether it came from the Hub scripts or from static reference data.


Workflow

Step 1: Check Region

Run the same boto3 region check used by planning and SDK setup:

python -c "import boto3; print(boto3.session.Session().region_name)"
  • Result is None → STOP. Tell user: "Set your region via export AWS_DEFAULT_REGION=us-west-2 or aws configure."

  • Region set → store REGION in context, continue.

Step 2: Discover Hub
  1. List all available SageMaker Hubs in the user's region:

    aws sagemaker list-hubs --region <region>
  2. From the results, filter out any hub whose HubDescription contains "AI Registry" — these do not contain JumpStart models.

  3. The remaining hubs are eligible (e.g., SageMakerPublicHub and any private hubs).

  4. If exactly one eligible hub exists, use it automatically — do not ask the user.

  5. If multiple eligible hubs exist, present them to the user and ask which one to use. Example:

    I found the following model hubs:
    - SageMakerPublicHub — SageMaker Public Hub
    - Private-Hub-XYZ — Private Hub models
    Which hub would you like to use?
    
  6. Store the selected hub name for use in subsequent steps.

Step 3: Select Base Model

Determine the user's goal from the plan, conversation context, or use_case_spec.md (check the ## Intent section):

  • If the goal is fine-tuning / model customization (Intent says "Fine-tune") → follow Step 3A: Select for Fine-tuning below.

  • If the goal is base model deployment (Intent says "Deploy base model") → read references/select-for-deployment.md and follow it through all steps (including hosting configuration selection). Then proceed to Step 4 below.

If the intent is ambiguous, ask:

"Would you like to fine-tune a model, or deploy a base model as-is?"

Step 3A: Select for Fine-tuning

First, retrieve all available SageMaker Hub model names by running: python model-selection/scripts/get_model_names.py <hub-name>. Note how many models the script returns.

Present all available models to the user with their licenses before making any recommendations. Cross-reference the model list with references/model-licenses.md and display each as <model name> - [<license>](<url>). For example: "Qwen3-4B - Apache 2.0"

Display every model — completeness is required:

  • get_model_names.py returns the models in a fixed, deterministic order: alphabetical by name (A→Z, case-insensitive). Present them in exactly that order — do not re-sort or reorder the list. (This applies only to this raw availability list; the benchmark ranking tables in references/model-selection.md keep their performance order and must not be alphabetized.)

  • List each model returned by get_model_names.py as its own separate line with its own license URL. The number of models you display MUST equal the number the script returned.

  • Do not omit, drop, or skip any model — even when the list is long.

  • Do not separate models into Text vs VLM categories. Do not exclude VLMs from text-only use cases. Present all models in a single unified list regardless of modality.

  • Do not merge or group models under a shared license, and do not collapse similar-looking models into one entry. Models that look like variants of each other are still distinct and each needs its own line. For example:

    • Text vs. vision variants of the same family (e.g., Qwen3-32B text models vs. Qwen3-VL-27B vision models) are different models.

    • Version variants (e.g., Amazon Nova Lite vs. Amazon Nova Lite v2) are different models.

    • Models that share an identical license URL still each get their own line.

  • After presenting the list, verify your output: confirm the count of displayed models matches the count returned by the script. If any are missing, add them before continuing.

If you already know the model the user wants to use (from conversation context or planning files), confirm that it's in the list, display its license, and move on. Otherwise, help the user pick a model following the instructions in references/model-selection.md. Important: Make sure to remember this list of available models when helping with model selection. Don't recommend a model that's not available to the user.

EULA disclaimer (REQUIRED — you MUST include this whenever you present this model list):

After presenting the model list, always append this disclaimer:

"ℹ️ Note: Some models in this list require accepting an End-User License Agreement (EULA) before deployment. This is a SageMaker JumpStart requirement that is independent of the model's open-source license. Once you choose a model, I'll tell you whether it requires EULA acceptance."

Step 4: Confirm Selection
4a. Check EULA gating status

After the user selects a model — on both the base model deployment path and the fine-tuning path — check whether that model requires a JumpStart hub EULA acknowledgment. If the HubContentDocument was already retrieved earlier in this session, reuse that response instead of calling the API again. Otherwise, call describe-hub-content and check the GatedBucket field in the HubContentDocument JSON:

aws sagemaker describe-hub-content \
    --region <region> \
    --hub-name <hub-name> \
    --hub-content-type Model \
    --hub-content-name <selected-model-id>

Where:

  • <region> — the REGION resolved in Step 1

  • <hub-name> — the hub name selected in Step 2

  • <selected-model-id> — the HubContentName of the model the user just chose

From the response, parse the HubContentDocument JSON string and read GatedBucket:

  • If GatedBucket is true: the model is gated — tell the user this model requires accepting an EULA at deployment time.

  • If GatedBucket is false or absent: the model is not gated — tell the user this model does not require accepting an EULA.

If the describe-hub-content call fails (permissions, model not found, region mismatch), do not block selection — tell the user the gating status could not be determined and recommend they verify the model's license terms independently.

Important: The JumpStart hub EULA gate is independent of the upstream model license. A model can be Apache 2.0 licensed and still be gated (requiring EULA acceptance), or vice versa. The upstream license governs legal usage rights; the hub gate is a separate acknowledgment for models whose artifacts are stored in a gated S3 bucket.

4b. Present confirmation summary

Present a summary to the user:

Here's what we've selected:
- Base model: <model name>
- EULA required at deploy: Yes / No

If the model is gated, add:

"⚠️ This model requires accepting an End-User License Agreement (EULA) at deployment time."

For the base model deployment path, also carry forward the three fields resolved in select-for-deployment.md Step 6:

- Model ID (Hub ID): [model_id]
- Instance type: [instance_type]
- Hosting configuration: [inference_config_name, or "none (base config)"]

model_id is the JumpStart Hub ID of the base model selected above (the (Hub ID) label is a reminder to use that exact identifier, not a friendly name). model_id and instance_type are both required; inference_config_name may be null when the model has no labeled configs. Carry all three forward — do not drop any of them.

Ask if they'd like to proceed with this model.

References

  • references/model-selection.md — Fine-tuning model selection instructions and benchmark descriptions

  • references/select-for-deployment.md — Base model deployment selection sub-workflow

  • references/model-licenses.md — Model license information for display during model selection

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip