AWS AI ML

AWSOfficialPublished by AWS

Skill
Tool integration
Cloud and DevOps
Apache-2.0
Low risk
Has scripts

From the author

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.

What it does

Guides users through the end-to-end lifecycle of SageMaker model customization.
Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
Provides workflows for validating and transforming training and evaluation datasets.
Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
Facilitates the deployment of models to SageMaker endpoints or Bedrock.
Diagnoses endpoint issues including latency, errors, and container logs.

When to use it

When planning a new model customization or deployment project on SageMaker.
When you need to validate or transform datasets for training or evaluation.
When selecting a base model or choosing a fine-tuning technique.
When you need to configure training, processing, or evaluation jobs.
When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
When setting up or managing a SageMaker Managed MLflow environment.

Advantages

Provides structured, step-by-step workflows for complex SageMaker tasks.
Ensures dataset formatting meets specific SageMaker requirements.
Offers clear routing to relevant documentation based on user intent.
Helps resolve configuration issues for IAM, S3, and SDK settings.
Simplifies the diagnosis of endpoint performance and failure logs.

Usage examples

“I need to fine-tune a model on SageMaker. Where do I start?”: The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
“My SageMaker endpoint is returning errors. Can you help?”: The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
“How do I convert my CSV training data for SageMaker?”: The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
“Which base model should I use for a text summarization task?”: The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/aws-ai-ml/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills
```

Documentation by AWS

AWS AI/ML Model Customization

Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.

Routing

Match the user's intent to the appropriate reference folder and load only that content.

User intent	Reference	When to use
Plan a model customization project, discover scope of work, resume or modify a plan	references/planning/	User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference.
Define the business problem, success criteria, or use case spec	references/use-case-specification/	User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines.
Select or change a base model	references/model-selection/	User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation.
Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF)	references/finetuning-technique/	User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first.
Validate dataset quality and format	references/dataset-evaluation/	User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job.
Transform or convert a dataset between formats	references/dataset-transformation/	User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code.
Generate fine-tuning code and start training	references/finetuning/	User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers.
Train a model with a SageMaker Training job	references/training-jobs/	Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis.
Process data with a SageMaker Processing job	references/processing-jobs/	Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis.
Evaluate or benchmark a trained model	references/model-evaluation/	User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer.
Deploy, benchmark, or optimize a model on an endpoint or Bedrock	references/model-deployment/	User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways.
Set up IAM roles, S3 buckets, SDK configuration	references/sdk-getting-started/	User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment.
Manage project directory and artifacts	references/directory-management/	Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory.
Set up, update, or delete a SageMaker Managed MLflow app	references/manage-mlflow/	User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected.
Diagnose a failing or unhealthy SageMaker endpoint	references/endpoint-diagnostics/	User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments.

Rules

Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.
Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.
Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.
Usage attribution. Before running any AWS CLI command or packaged script, set export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.

Files

ATTRIBUTION.mdAdded by AutoGineer883 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer67 B
SKILL.md8.1 KiB
references/dataset-evaluation/overview.md5.4 KiB
references/dataset-evaluation/references/custom-scorer-evaluation-dataset-formats.md3.6 KiB
references/dataset-evaluation/references/strategy_data_requirements.md4.8 KiB
references/dataset-evaluation/scripts/format_detector.pyScript26.7 KiB
references/dataset-transformation/code_templates/transformation.pyScript1.5 KiB
references/dataset-transformation/overview.md13.3 KiB
references/dataset-transformation/references/code_output_guide.md2.8 KiB
references/dataset-transformation/references/dataset_transformation_code.md5.2 KiB
references/dataset-transformation/references/sagemaker_dataset_formats.md2.9 KiB
references/dataset-transformation/scripts/transformation_tools.pyScript4.8 KiB
references/directory-management/overview.md1.9 KiB
references/endpoint-diagnostics/overview.md3.5 KiB
references/endpoint-diagnostics/references/minimum_iam_policy.md1.2 KiB
references/endpoint-diagnostics/scripts/collect_diagnostics.pyScript12.8 KiB
references/finetuning-technique/overview.md2.1 KiB
references/finetuning-technique/references/finetune_technique_selection_guide.md2.3 KiB
references/finetuning-technique/scripts/get_recipes.pyScript817 B
references/finetuning/code_templates/dpo.pyScript6.4 KiB
references/finetuning/code_templates/rlaif_builtin.pyScript5.8 KiB
references/finetuning/code_templates/rlaif_custom_prompt.pyScript6.3 KiB
references/finetuning/code_templates/rlvr.pyScript6.5 KiB
references/finetuning/code_templates/sft.pyScript6.1 KiB
references/finetuning/overview.md11.4 KiB
references/finetuning/references/code_output_guide.md3.2 KiB
references/finetuning/references/continuous_customization.md8.4 KiB
references/finetuning/references/eula_links.md7.3 KiB
references/finetuning/references/rlaif_guide.md3.8 KiB
references/finetuning/references/rlvr_reward_function.md10.0 KiB
references/finetuning/scripts/mlflow_reference.pyScript841 B
references/finetuning/templates/nova_rlvr_reward_function_source_template.pyScript12.8 KiB
references/finetuning/templates/rlvr_reward_function_source_template.pyScript9.1 KiB
references/manage-mlflow/overview.md4.8 KiB
references/manage-mlflow/references/app-creation-workflow.md3.2 KiB
references/manage-mlflow/references/app-deletion-workflow.md2.3 KiB
references/manage-mlflow/references/app-discovery-workflow.md1.5 KiB
references/manage-mlflow/references/app-update-workflow.md1.7 KiB
references/manage-mlflow/references/sm-mlflow-guide.md1.3 KiB
references/model-deployment/code_templates/deploy-jumpstart-sagemaker.pyScript7.5 KiB
references/model-deployment/code_templates/deploy-nova-bedrock.pyScript2.0 KiB
references/model-deployment/code_templates/deploy-nova-sagemaker.pyScript1.9 KiB
references/model-deployment/code_templates/deploy-oss-bedrock.pyScript4.0 KiB
references/model-deployment/code_templates/deploy-oss-sagemaker.pyScript2.6 KiB
references/model-deployment/code_templates/optimize-benchmark.pyScript4.6 KiB
references/model-deployment/code_templates/optimize-huggingface-to-s3.pyScript2.0 KiB
references/model-deployment/code_templates/optimize-recommendation.pyScript6.4 KiB
references/model-deployment/overview.md16.6 KiB
references/model-deployment/references/code_output_guide.md3.2 KiB
references/model-deployment/references/deploy-jumpstart-sagemaker.md14.3 KiB
references/model-deployment/references/deploy-nova-bedrock.md4.4 KiB
references/model-deployment/references/deploy-nova-sagemaker.md5.8 KiB
references/model-deployment/references/deploy-oss-bedrock.md5.2 KiB
references/model-deployment/references/deploy-oss-sagemaker.md12.9 KiB
references/model-deployment/references/model-licenses.md11.9 KiB
references/model-deployment/references/optimization/benchmark-workflow.md12.7 KiB
references/model-deployment/references/optimization/benchmarking-guidance.md8.9 KiB
references/model-deployment/references/optimization/huggingface-to-s3.md3.1 KiB
references/model-deployment/references/optimization/interpreting-results.md8.5 KiB
references/model-deployment/references/optimization/recommendation-deploy.md1.5 KiB
references/model-deployment/references/optimization/recommendation-options.md3.1 KiB
references/model-deployment/references/optimization/recommendation-workflow.md10.3 KiB
references/model-evaluation/code_templates/custom_scorer_evaluator.pyScript2.5 KiB
references/model-evaluation/code_templates/llmaaj_evaluator.pyScript2.8 KiB
references/model-evaluation/overview.md5.2 KiB
references/model-evaluation/references/code_output_guide.md3.3 KiB
references/model-evaluation/references/create-reward-function.md3.0 KiB
references/model-evaluation/references/custom-lambda-scorer.md4.4 KiB
references/model-evaluation/references/custom-scorer-evaluation.md10.5 KiB
references/model-evaluation/references/evaluation-type-guide.md10.1 KiB
references/model-evaluation/references/llmaaj-builtin-evaluation.md5.0 KiB
references/model-evaluation/references/llmaaj-custom-evaluation.md2.5 KiB
references/model-evaluation/references/llmaaj-evaluation.md14.6 KiB
references/model-evaluation/references/supported-judge-models.md2.9 KiB
references/model-evaluation/scripts/nova_reward_function_source_template.pyScript13.1 KiB
references/model-evaluation/scripts/reward_function_source_template.pyScript9.2 KiB
references/model-evaluation/scripts/validate_custom_metrics.pyScript3.5 KiB
references/model-selection/overview.md10.3 KiB
references/model-selection/references/benchmarks/agentic-index.md2.8 KiB
references/model-selection/references/benchmarks/coding-index.md2.8 KiB
references/model-selection/references/benchmarks/gpqa.md2.8 KiB
references/model-selection/references/benchmarks/hle.md2.8 KiB
references/model-selection/references/benchmarks/ifbench.md2.8 KiB
references/model-selection/references/benchmarks/intelligence-index.md2.8 KiB
references/model-selection/references/benchmarks/mmmu-pro.md3.0 KiB
references/model-selection/references/benchmarks/tau2.md2.8 KiB
references/model-selection/references/model-licenses.md7.3 KiB
references/model-selection/references/model-selection.md8.2 KiB
references/model-selection/references/select-for-deployment.md19.8 KiB
references/model-selection/scripts/filter_deployable_models.pyScript10.1 KiB
references/model-selection/scripts/get_deployable_models.pyScript21.7 KiB
references/model-selection/scripts/get_model_names.pyScript1.4 KiB
references/model-selection/scripts/search_deployable_models.pyScript34.9 KiB
references/planning/overview.md10.3 KiB
references/planning/references/deploy-base-model-plan.md699 B
references/planning/references/evaluate-first-plan.md1.9 KiB
references/planning/references/input-output-contracts.md3.1 KiB
references/planning/references/model-customization-plan.md1.5 KiB
references/planning/references/region-availability.md1.2 KiB
references/planning/references/skill-routing-constraints.md2.7 KiB
references/processing-jobs/code_templates/dataset_definition_processing.pyScript4.1 KiB
references/processing-jobs/code_templates/framework_processing.pyScript3.0 KiB
references/processing-jobs/code_templates/script_processing.pyScript2.9 KiB
references/processing-jobs/code_templates/spark_processing.pyScript3.7 KiB
references/processing-jobs/overview.md6.7 KiB
references/processing-jobs/references/configuration.md4.6 KiB
references/processing-jobs/references/troubleshooting.md2.4 KiB
references/sdk-getting-started/overview.md643 B
references/sdk-getting-started/references/execution-role-setup.md2.4 KiB
references/sdk-getting-started/references/sagemaker-python-sdk-setup.md2.1 KiB
references/training-jobs/code_templates/custom_training.pyScript6.2 KiB
references/training-jobs/code_templates/jumpstart_training.pyScript5.6 KiB
references/training-jobs/code_templates/recipe_training.pyScript5.7 KiB
references/training-jobs/overview.md9.1 KiB
references/training-jobs/references/configuration.md4.0 KiB
references/training-jobs/references/troubleshooting.md4.4 KiB
references/use-case-specification/overview.md2.0 KiB
references/use-case-specification/references/spec-for-deployment.md4.0 KiB
references/use-case-specification/references/spec-for-finetuning.md2.1 KiB

references/model-deployment/references/optimization/benchmark-workflow.md

Benchmark Workflow

Guide the user through creating and running an AI Benchmark Job to measure inference performance (latency, throughput) on an existing SageMaker endpoint.

Read benchmarking-guidance.md first for the know-how that shapes the choices below — when a benchmark stops (request_count, not a timer), how many requests a given percentile needs (e.g. p99 needs ≥ ~1000 requests), synthetic vs. dataset vs. real-traffic selection (and why speculative-decoding setups need real data), and comparability rules.

Applies to

Benchmarking scope (supported endpoint types, and the not-Bedrock/HyperPod rule incl. the don't-search directive): see model-deployment/overview.md → Optimization applicability (single source of truth). Custom (non-OpenAI) endpoint formats are handled via template mode (see Step 3).
Regions: see model-deployment/overview.md → Optimization applicability (single source of truth).

Step 1: Identify the Endpoint

You need the endpoint name and its region. Everything else (job name, workload-config name, S3 output location, tokenizer) the agent infers and confirms — do not make the user supply them.

If the user doesn't know their endpoint name:

"I can look up your SageMaker endpoints. Would you like me to list them?"

Run aws sagemaker list-endpoints --status-equals InService to list candidates. Run aws sagemaker describe-endpoint --endpoint-name <name> to confirm it exists, capture its region, and read the model behind it. If the endpoint uses inference components, run aws sagemaker list-inference-components --endpoint-name <name> to enumerate them. (The AWS MCP server is recommended but not required — the runtime routes these CLI calls through it automatically when available.)

Determine the request/response format yourself — don't ask the user. Read the container image/env vars via aws sagemaker describe-model --model-name <name> (LMI/TGI/vLLM → OpenAI ChatCompletions → synthetic mode). If still ambiguous, one probe (aws sagemaker-runtime invoke-endpoint with {"messages":[...], "max_tokens":8}) settles it — but a probe sends real inference, so gate it behind the Step 2 confirmation (don't send it before the user agrees). A non-OpenAI shape → template mode.

Region / Bedrock check: confirm the endpoint is in a supported region (see Applies-to). If the "endpoint" is actually Bedrock, this workflow can't benchmark it — tell the user, and offer to benchmark it if they deploy it to a SageMaker endpoint.

Step 2: In-Service Safety Check

Benchmarking (and the Step 1 format probe) sends real inference load that can add latency for live users. If the endpoint was just created in this conversation, proceed. Otherwise warn that it's InService, suggest a dedicated copy if it serves production traffic, and ⏸ wait for explicit confirmation before sending any load (including the probe).

Step 3: Choose the Workload Source

The workload defines the traffic the benchmark sends. Pick the source that matches the user's intent:

Intent	Source	SDK call
Standard perf check / "how much can it serve" (OpenAI ChatCompletions endpoint)	Synthetic	`Workload.synthetic(...)`
Load test — push high concurrency to find the ceiling	Synthetic, high `concurrency` + higher `request_count`	`Workload.synthetic(...)`
Endpoint speaks a custom (non-OpenAI) format	Template — a Jinja2 request template + JMESPath response field	`Workload.template(...)`
"How does it do on my dataset" — a curated file of prompts in S3	Dataset (S3 file)	`Workload.from_dataset(<s3-jsonl-prefix>, ...)`
"How does it do on my real traffic" — actual requests the endpoint served	Traffic capture (SageMaker Data Capture)	`Workload.from_dataset(<data-capture-prefix>, custom_dataset_type="sagemaker-datacapture", ...)`

Dataset vs. traffic capture are different sources — keep them distinct:

Dataset comes from S3: a JSONL file the user curated/exported (ShareGPT / OpenAI Chat / OpenAI Completions). It's whatever prompts they chose to put there.
Traffic capture comes from an endpoint: SageMaker Data Capture writes the requests/responses the endpoint actually served to an S3 prefix. It's real production traffic, not a curated file. Point from_dataset at that capture prefix with custom_dataset_type="sagemaker-datacapture". (The endpoint must have Data Capture enabled and use a supported request format.)

Tokenizer

All workload modes — synthetic, dataset, and template (custom-format) — take a tokenizer (HuggingFace tokenizer id, e.g. meta-llama/Llama-3.2-1B) to size prompts; pass tokenizer=TOKENIZER in every mode (it's required for Workload.synthetic). Infer it from the model behind the endpoint (aws sagemaker describe-endpoint --endpoint-name <name> → model → HuggingFace model id / model name). If you can't infer it confidently, suggest your best guess and ask the user to confirm rather than asking them to supply it cold.

Workload parameters

Infer input/output token sizes and concurrency from the endpoint's use case (e.g. chat = short-to-moderate; summarization = long input; a load test drives concurrency high); only ask if genuinely ambiguous. Defaults if nothing else is known: input 512, output 256, concurrency 1, request_count 100.

Custom-format endpoints (template mode)

For endpoints that don't speak OpenAI ChatCompletions, use Workload.template(...):

request_template — a Jinja2 template (local path or inline string) describing the endpoint's request payload; it's uploaded for you. (Or pass template_s3_uri if already in S3.)
response_field — a JMESPath query that extracts generated text from the response (e.g. generated_text or choices[0].message.content). Omit to let AIPerf auto-detect.

Dataset mode

For "benchmark on my dataset / real traffic", use Workload.from_dataset(s3_uri, tokenizer=..., custom_dataset_type=...). The user gives you an S3 location — you inspect it and figure out the format; don't ask them to name it.

Inspect first, then decide: read a couple of records from the S3 prefix (or the endpoint's SageMaker Data Capture prefix) and identify the schema.

custom_dataset_type is optional — a hint to AIPerf naming the schema so it parses records correctly; pass the matching value if you identify the schema, or omit it to let AIPerf auto-detect (omitting does not fail the job). See benchmarking-guidance.md → "Dataset schema hint (custom_dataset_type)" for the schema → value mapping.
If the data is in a format none of those cover, do NOT ask the user to reformat it by hand. Hand off to the dataset-transformation skill to convert it into a supported schema (e.g. OpenAI Chat JSONL), then come back and benchmark the converted data. That skill owns all format conversion.

Two dataset sources (keep them straight):

My dataset — a curated file in S3: the user points you at a JSONL prefix they exported/curated. Inspect it, detect the schema, benchmark it. This measures the endpoint against the prompts they chose.
My real traffic — SageMaker Data Capture: the user points you at (or asks you to find) the endpoint's Data Capture prefix. This is what the endpoint actually served. Use custom_dataset_type="sagemaker-datacapture". Prerequisites: the endpoint must have Data Capture enabled (check aws sagemaker describe-endpoint --endpoint-name <name> → DataCaptureConfig) and serve a supported request format. If Data Capture isn't enabled, tell the user it must be turned on first (it can't be applied retroactively to past traffic). This is the most production-faithful source, and the right choice for speculative-decoding comparisons (see benchmarking-guidance.md).

Step 4: Confirm the Auto-Generated Names

Do not ask the user to type job names, config names, or the S3 output path. Infer a short prefix from the endpoint/model and today's date, derive the names, and confirm once:

"I'll use these names — ok, or would you like a different prefix?
Benchmark job: bench-<prefix>
Workload config: bench-<prefix>-wl
Results output: s3://<default-bucket>/benchmarks/<prefix>/
(Default bucket is your account's SageMaker default bucket; tell me if you'd rather use a specific bucket.)"

If the user accepts, proceed. If they give a different prefix or bucket, use it. This is the only naming interaction.

Step 5: Generate Code

Generate a Jupyter notebook or a Python script — read ../code_output_guide.md for the mode choice and output format rules (the template's # NOTEBOOK_ONLY markers make it dual-mode). Each cell's content comes from ../../code_templates/optimize-benchmark.py, split on the # Cell N: comments. The template uses the SageMaker Python SDK (>=3.20.0; Cell 1 installs it in notebook mode), not boto3.

Cell 0 (markdown): Section header
Cell 1: Setup (%pip install — notebook only)
Cell 2: Configuration (imports, set_attribution, values)
Cell 3: Define the Workload — Workload.synthetic(...) by default; swap to Workload.template(...) (custom format) or Workload.from_dataset(...) (dataset/Data Capture) per Step 3
Cell 4: Run the Benchmark (start_benchmark(...), blocks until the job is terminal)
Cell 5: Display Results (job.show_result())
Cell 6: Save Manifest
Cell 7 (optional): Compare runs — compare_benchmarks(...). Include only when the user has (or wants) a second run to compare against; reload a prior run with BenchmarkJob.get("<name>").show_result().

Values

Cell 2:

[REGION] → endpoint's region (from Step 1)
[ENDPOINT_NAME] → endpoint name (from Step 1)
[ROLE_ARN] → execution role. Infer from the endpoint's model (aws sagemaker describe-model --model-name <name> → ExecutionRoleArn) or the SageMaker default execution role; only ask if none can be resolved.
[S3_OUTPUT_LOCATION], [BENCHMARK_JOB_NAME], [WORKLOAD_CONFIG_NAME] → the confirmed auto-generated names from Step 4
[TOKENIZER] → inferred tokenizer (Step 3)

Cell 3 (workload parameters): [INPUT_TOKENS_MEAN] (default 512), [INPUT_TOKENS_STDDEV] (50), [OUTPUT_TOKENS_MEAN] (256), [OUTPUT_TOKENS_STDDEV] (30), [CONCURRENCY] (1), [REQUEST_COUNT] (100).

Cell 3 (optional extra params): [EXTRA_PARAMS] → a dict of extra AIPerf workload parameters passed through **params, only when needed. Leave the line commented if not. Use it for ignore_eos: True (when the user specified an exact output length — see benchmarking-guidance.md), a low sampling temperature (for speculative-decoding runs), or a warmup control if the runtime's AIPerf workload exposes one. Example: {"ignore_eos": True, "temperature": 0.2}.

Cell 3 (dataset mode only): [DATASET_S3_URI] → S3 URI of the dataset/traffic-capture prefix to replay (e.g. s3://my-bucket/benchmark-data/ or the endpoint's SageMaker Data Capture prefix). Only used in the commented Workload.from_dataset(...) block; leave the synthetic workload in place if the user isn't benchmarking on their own data.

Cell 4 conditional: [INFERENCE_COMPONENT_NAME] → the inference-component name(s) to target, only if the endpoint uses inference components. Obtain via aws sagemaker list-inference-components --endpoint-name <name> (Step 1). If the endpoint has none, leave inference_components=[...] commented out.

Cell 6: [PROJECT_DIR] → absolute path to the user's project directory where the run manifest is written (e.g. /home/user/my-project); use the directory established by the directory-management reference, or the current working directory if none.

Cell 7 (only when comparing runs): [BASELINE_JOB_NAME] → the earlier benchmark job to compare against (reloaded via BenchmarkJob.get(...)); [BASELINE_LABEL] / [THIS_RUN_LABEL] → short human labels for the two runs (e.g. before-opt / after-opt, or concurrency-4 / concurrency-150). Omit this cell entirely for a single run.

Step 6: Present Results

When the job completes, read interpreting-results.md and present the metrics using the "Benchmark Job Results" headline format exactly (one consistent human-readable line first, second-order metrics after).

After presenting results, offer next steps:

"What would you like to do next?
Run another benchmark — different concurrency or token lengths (e.g. a higher-concurrency load test)
Compare against a previous run — line this run up against an earlier one (before/after optimization, synthetic vs. dataset, or a concurrency step) with a single Δ% table
Get deployment recommendations — find a cheaper/faster instance + serving config for this model
Done"

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

Also distributed in

aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip