AutoGineer
Generated banner for AWS AI ML
Generated logo for AWS AI ML

AWS AI ML

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts
Download ZIP

From the author

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.

What it does

  • Guides users through the end-to-end lifecycle of SageMaker model customization.
  • Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
  • Provides workflows for validating and transforming training and evaluation datasets.
  • Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
  • Facilitates the deployment of models to SageMaker endpoints or Bedrock.
  • Diagnoses endpoint issues including latency, errors, and container logs.

When to use it

  • When planning a new model customization or deployment project on SageMaker.
  • When you need to validate or transform datasets for training or evaluation.
  • When selecting a base model or choosing a fine-tuning technique.
  • When you need to configure training, processing, or evaluation jobs.
  • When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
  • When setting up or managing a SageMaker Managed MLflow environment.

Advantages

  • Provides structured, step-by-step workflows for complex SageMaker tasks.
  • Ensures dataset formatting meets specific SageMaker requirements.
  • Offers clear routing to relevant documentation based on user intent.
  • Helps resolve configuration issues for IAM, S3, and SDK settings.
  • Simplifies the diagnosis of endpoint performance and failure logs.

Usage examples

“I need to fine-tune a model on SageMaker. Where do I start?”
The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
“My SageMaker endpoint is returning errors. Can you help?”
The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
“How do I convert my CSV training data for SageMaker?”
The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
“Which base model should I use for a text summarization task?”
The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/aws/aws-ai-ml/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills

Documentation by AWS

AWS AI/ML Model Customization

Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.

Routing

Match the user's intent to the appropriate reference folder and load only that content.

User intentReferenceWhen to use
Plan a model customization project, discover scope of work, resume or modify a planreferences/planning/User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference.
Define the business problem, success criteria, or use case specreferences/use-case-specification/User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines.
Select or change a base modelreferences/model-selection/User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation.
Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF)references/finetuning-technique/User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first.
Validate dataset quality and formatreferences/dataset-evaluation/User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job.
Transform or convert a dataset between formatsreferences/dataset-transformation/User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code.
Generate fine-tuning code and start trainingreferences/finetuning/User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers.
Train a model with a SageMaker Training jobreferences/training-jobs/Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis.
Process data with a SageMaker Processing jobreferences/processing-jobs/Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis.
Evaluate or benchmark a trained modelreferences/model-evaluation/User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer.
Deploy, benchmark, or optimize a model on an endpoint or Bedrockreferences/model-deployment/User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways.
Set up IAM roles, S3 buckets, SDK configurationreferences/sdk-getting-started/User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment.
Manage project directory and artifactsreferences/directory-management/Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory.
Set up, update, or delete a SageMaker Managed MLflow appreferences/manage-mlflow/User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected.
Diagnose a failing or unhealthy SageMaker endpointreferences/endpoint-diagnostics/User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments.

Rules

  • Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.

  • Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.

  • Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.

  • Usage attribution. Before running any AWS CLI command or packaged script, set export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.

Files

references/model-evaluation/references/custom-scorer-evaluation.md

Custom Scorer Evaluation

Guide the user through the process for evaluating a model with Custom Scorer (built-in Prime Math/Code or custom Lambda).

Workflow

Step 0: Consider prior context

Before proceeding, silently think about the context you have about the user's project, including conversation history and file reads. You should use that knowledge, and avoid asking questions you already know the answer to.

Step 1: Validate Custom Scorer compatibility

Before proceeding, confirm one thing:

  1. Does the user have an evaluation dataset?

If the check fails (the user has no eval dataset), tell the user and offer to help them pick an alternative:

"Custom Scorer evaluation requires an evaluation dataset. Would you like help choosing a different evaluation type?"

If they want help choosing a different evaluation type → break this workflow and read references/evaluation-type-guide.md.

If the check passes, proceed.

Step 2: Understand the task

For this step, you need: to understand the task the model is trained to do. If you know this already, skip this step. If not, ask the user:

"What task is this model trained to do?"

Step 3: Get evaluation dataset

For this step, you need: the evaluation dataset S3 path. If you know this already, skip this step. If not, ask the user:

"Where's your evaluation dataset stored in S3?"

Step 4: Choose scorer type

For this step, you need to know which scorer to use.

If the model is Nova: built-in scorers (Prime Math, Prime Code) are not supported for Nova models. Inform the user and proceed with Custom Lambda:

"For Nova models, only Custom Lambda scoring is supported. Built-in scorers (Prime Math, Prime Code) won't produce results. Let's set up a Custom Lambda scorer."

Then read references/custom-lambda-scorer.md and follow its instructions. Return here and proceed to Step 5.

If the model is OSS: ask if you don't already know from context:

"Which type of scorer would you like to use?

  1. Prime Math — built-in scorer for math problems (checks answer correctness)

  2. Prime Code — built-in scorer for coding problems (executes code against test cases)

  3. Custom Lambda — your own scoring logic as a Lambda function. You can use an existing registered evaluator or create a new one.

Which would you prefer?"

  • If built-in (Prime Math or Prime Code) → note the choice and proceed to Step 5.

  • If custom Lambda → read references/custom-lambda-scorer.md and follow its instructions to resolve the evaluator. Then return here and proceed to Step 5. You MUST follow these instructions before moving on.

Step 5: Validate dataset format

IMPORTANT: you MUST validate that dataset, to ensure that it is the correct format. Note that there are precise requirements based on model and evaluation type, so you cannot skip this step.

Reference the dataset-evaluation reference to perform this validation.

Step 6: Determine evaluation scope

For this step, you need to know which model(s) to evaluate.

If you already know from context, confirm and move on. Otherwise, ask:

"Would you like to evaluate:

  1. Just your fine-tuned model

  2. Just a base model

  3. Both, with a comparison

⏸ Wait for user approval.

Step 7: Resolve Model Package ARN

This step only applies if the evaluation scope includes the fine-tuned model (option 1 or 3 from Step 6). If the user chose base model only, skip to Step 8.

For this step, you need: the Model Package ARN of the fine-tuned model.

If you already have it from prior context, confirm with the user and move on. Otherwise, ask:

"What's the Model Package ARN (or group name) of your fine-tuned model?"

If they provide a group name, resolve the ARN by running aws sagemaker list-model-packages --model-package-group-name <group-name> --sort-by CreationTime --sort-order Descending --max-results 1. Use the latest version's ModelPackageArn.

Validate the resolved ARN:

  • Must look like: arn:aws:sagemaker:REGION:ACCOUNT:model-package/NAME/VERSION

  • If it's a group ARN (:model-package-group/), resolve to a package ARN by running aws sagemaker list-model-packages --model-package-group-name <group-name> --sort-by CreationTime --sort-order Descending --max-results 1. Use the latest version's ModelPackageArn.

  • If it contains :model-package/ but does NOT end with a version number (e.g., /1), resolve it: run aws sagemaker list-model-packages --model-package-group-name <group-name> --sort-by CreationTime --sort-order Descending --max-results 1.

  • If it contains /DataSet/, /TrainingJob/, or other non-model-package resource types, flag it: "That looks like a [Dataset/TrainingJob] ARN, not a model package ARN. Could you double-check?"

  • Verify it exists by running aws sagemaker describe-model-package --model-package-name <arn>. If this fails, tell the user the ARN wasn't found and ask them to double-check.

Step 8: Resolve base model

This step only applies if the evaluation scope includes the base model (option 2 or 3 from Step 6). If the user chose fine-tuned only, skip to Step 9.

For comparison mode (option 3): the base model is resolved automatically from the fine-tuned model's lineage — no additional input needed.

For base model only (option 2): you need a JumpStart model ID (e.g., meta-textgeneration-llama-3-2-1b-instruct). Check if you already know it from context. If not, ask:

"What's the JumpStart model ID of the base model you'd like to evaluate?"

<!-- TODO: Add guidance for helping the user find their JumpStart model ID. -->

Step 9: Resolve IAM role ARN

For this step, you need: an IAM role ARN with permissions for evaluation.

If you already know the IAM role ARN from conversation context, confirm with the user and move on.

If not, but you know the training job name, look up the role from that training job, since it should be a suitable role:

  • Run aws sagemaker describe-training-job --training-job-name <name> and extract RoleArn.

  • Confirm with the user.

If you don't have either, ask:

"What IAM role ARN should I use for the evaluation job?"

Important: If the user chose a custom Lambda scorer, the IAM role must also have lambda:InvokeFunction permission for the scorer's Lambda function. Mention this to the user:

"Note: since you're using a custom Lambda scorer, this role needs lambda:InvokeFunction permission for your scorer function. Does your role have that?"

Step 10: Resolve AWS Region

For this step, you need: the AWS region for the evaluation job. If you already know it from context, confirm and move on. Otherwise, ask.

Step 11: Resolve S3 output path

For this step, you need: an S3 path for evaluation output. If you already know it from context, confirm and move on. Otherwise, ask.

Step 12: MLflow experiment name (optional)

For this step, you need to know: the MLflow experiment name. If you already know this from context, confirm and move on. If not, the name should follow this pattern: <project-name>-eval.

Perform this step quietly. The user does not need to know about this. This is to make sure their results are viewable alongside other experimental work they're doing.

Step 13: Final Checks

IMPORTANT: Before moving on, you MUST think through every step of this workflow. Did you complete it, or did you have a good reason for skipping? If not, please complete it now. In particular, make sure you have carefully validated the dataset and the lambda against the correct requirements.

Step 14: Confirm configuration

Summarize everything and ask for approval:

"Here's the evaluation setup:

  • Task: [task]

  • Dataset: [path]

  • Scorer: [Prime Math / Prime Code / Custom Lambda ARN]

  • Model: [Model Package ARN or JumpStart model ID]

  • Evaluation scope: [fine-tuned only / base only / both with comparison]

  • IAM role: [ARN]

  • Region: [region]

  • S3 output: [path]

  • MLflow experiment name: [MLflow experiment name]

Does this look right?"

⏸ Wait for user approval.

Step 15: Generate code

Read ../references/code_output_guide.md for output format rules.

If no project directory exists, load the directory-management reference to set one up.

Read code_templates/custom_scorer_evaluator.py, substitute the collected values into the placeholders, and write the cells. The template uses # Cell N: Label markers — each marker starts a new notebook cell, with everything between one marker and the next becoming that cell's content.

Step 16: Post-generation

Notebook mode:

To run:
1. Cell 1 — configuration and SDK install
2. Cell 2 — start evaluation
3. Cell 3 — polls status automatically (~25-60 min)
4. Cell 4 — show results

Script mode:

Evaluation can take hours depending on your dataset. Present the user with options:

"Would you like me to:

  1. Leave it to you — run with python scripts/[script_name]

  2. Run it and wait until it's done

  3. Start it but don't wait — we can check status later"

  • Option 1: Done. Wait for user to come back.

  • Option 2: Execute the script as-is. execution.wait() polls until complete. Report results.

  • Option 3: Remove the execution.wait() call, execute, report the evaluation ARN.

Note: evaluate() does not accept a wait parameter. It always returns immediately. Blocking is done via execution.wait(target_status="Succeeded").

Checking status:

  • describe-pipeline-execution --pipeline-execution-arn ARN → PipelineExecutionStatus

  • list-pipeline-execution-steps --pipeline-execution-arn ARN → per-step StepStatus, FailureReason

Showing results after completion:

  • Run: EvaluationPipelineExecution.get(arn=ARN).show_results()

FAQ

Q: What metrics do I get with Custom Scorer? A: Both built-in and custom scorers automatically produce standard NLP metrics (F1, ROUGE, BLEU) alongside your custom scores.

Q: Does my IAM role need special permissions for Custom Scorer? A: Yes — if using a custom Lambda scorer, the IAM role needs lambda:InvokeFunction permission for the scorer's Lambda function. Built-in scorers (Prime Math/Code) don't require additional permissions.

Q: Can I create a new reward function through this skill? A: Yes — if you choose Custom Lambda and don't have an existing evaluator, the agent will walk you through creating one from a template and registering it via Evaluator.create.

Nova Model Notes

Custom Scorer evaluation works with Nova models via Custom Lambda. Built-in scorers (Prime Math, Prime Code) are not supported — the pipeline will run without error, but the scorer will not execute.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip