AutoGineer
Generated banner for Model Evaluation
Generated logo for Model Evaluation

Model Evaluation

AWS LabsOfficialPublished by AWS Labs

  • Skill
  • Workflow
  • Software development
  • Apache-2.0
  • Low risk
  • Has scripts
Download ZIP

From the author

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for evaluating SageMaker models, supporting both open-source and Nova models. It guides users through selecting an evaluation strategy—either LLM-as-Judge for qualitative assessment or a Custom Scorer for programmatic validation—and generates the necessary Python code or Jupyter notebooks to execute these benchmarks within the SageMaker environment.

What it does

  • Determines the appropriate evaluation type based on model architecture and user requirements.
  • Validates compatibility between the chosen evaluation method and the target model.
  • Generates Python scripts or Jupyter notebooks for model evaluation.
  • Supports LLM-as-Judge for OSS models.
  • Facilitates the creation and registration of Lambda-based custom scorers.
  • Integrates with SageMaker SDK for streamlined model benchmarking.

When to use it

  • When you need to benchmark the performance of a fine-tuned or base SageMaker model.
  • When you want to use an LLM to grade model outputs.
  • When you require custom programmatic logic to score model responses.
  • When you need to compare different model versions or configurations.

Advantages

  • Ensures consistent evaluation workflows through guided steps.
  • Prevents incompatible configurations by validating model types early.
  • Automates the generation of boilerplate code for evaluation tasks.
  • Provides flexibility by supporting both automated LLM grading and custom logic.

Usage examples

“Evaluate my model.”
The agent will initiate the evaluation workflow, asking you to choose between LLM-as-Judge or Custom Scorer, and verify your model type.
“How did my model perform?”
The agent will guide you through setting up a benchmark or custom scorer to generate performance metrics.
“Run a benchmark on my fine-tuned Llama model.”
The agent will confirm the model type and guide you through the LLM-as-Judge or Custom Scorer setup process.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/awslabs/model-evaluation/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip model-evaluation.zip -d .agents/skills

Documentation by AWS Labs

Model Evaluation

Generate code that evaluates a SageMaker model.

Prerequisites

  • The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Principles

  1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.

  2. Confirm before proceeding. Wait for the user to agree before moving to the next step.

  3. Don't read files until you need them. Only read reference files when you've reached the step that requires them.

  4. Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.

  5. No narration. Share outcomes and ask questions. Keep responses short.

  6. No repetition. If you said something before a tool call, don't repeat it after.

Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

Tell the user when the skill is activated:

"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."

If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.

Evaluation Types

There are two evaluation types:

  • LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)

  • Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.

Workflow

Step 1: Determine evaluation type

Do you already know which evaluation type to use?

Check conversation history, plan.md, workflow_state.json, or anything else you've already read.

If yes: confirm with the user.

"It sounds like you want to run [evaluation type]. Is that right?"

⏸ Wait for confirmation. If confirmed → go to Step 2.

If no: ask.

"What kind of evaluation would you like to run? I support:

  1. LLM-as-Judge — an LLM grades your model's responses

  2. Custom Scorer — programmatic scoring (math, code, or your own logic)

Pick one, or say 'help me decide' if you're not sure."

⏸ Wait for user.

  • If user picks one → go to Step 2.

  • If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read references/evaluation-type-guide.md and follow its instructions. It will guide the user to a choice and then return here. You MUST NEVER make a recommendation to the user on eval type without reading references/evaluation-type-guide.md.

Step 2: Validate and hand off to evaluation workflow

Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to.

LLM-as-Judge validation

  1. What model type are we evaluating? LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it):

    • If you have the training job name or ARN, use the AWS MCP tool list-tags on the training job ARN and look for the sagemaker-studio:jumpstart-model-id tag. Contains "nova" → Nova. Anything else → OSS.

    • If you have a Model Package ARN, use the AWS MCP tool describe-model-package and check the model description or source tags.

    • If neither is available, ask the user.

  2. Does the user have an evaluation dataset? LLM-as-Judge requires one.

Custom Scorer validation

  1. Does the user have an evaluation dataset? Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.)

If validation fails, tell the user which requirement(s) aren't met and offer alternatives:

"[Evaluation type] won't work because [reason]."

If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user:

"Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation."

If the failure reason is something else, offer to help them pick a different evaluation type.

⏸ Wait for user.

If they say they do want help choosing a different eval type → read references/evaluation-type-guide.md.

If validation passes, read the corresponding reference file:

User choseRead
LLM-as-Judgereferences/llmaaj-evaluation.md
Custom Scorerreferences/custom-scorer-evaluation.md

Follow the reference file's instructions from the beginning.

Files

scripts/validate_custom_metrics.py

"""Validate custom metrics JSON against the Bedrock LLM-as-Judge format.

Usage:
    python validate_custom_metrics.py '<json_string>'
    python validate_custom_metrics.py path/to/custom_metrics.json
"""

import json
import sys
from typing import Optional, Union

from pydantic import BaseModel, field_validator, model_validator


class RatingValue(BaseModel):
    floatValue: Optional[float] = None
    stringValue: Optional[str] = None

    @model_validator(mode="after")
    def exactly_one_value(self):
        has_float = self.floatValue is not None
        has_string = self.stringValue is not None
        if has_float == has_string:  # both set or neither set
            raise ValueError("Exactly one of 'floatValue' or 'stringValue' must be set.")
        return self


class RatingScaleEntry(BaseModel):
    definition: str
    value: RatingValue

    @field_validator("definition")
    @classmethod
    def definition_length(cls, v):
        if len(v) > 100:
            raise ValueError(f"Definition exceeds 100 chars ({len(v)}).")
        return v


class CustomMetricDefinition(BaseModel):
    name: str
    instructions: str
    ratingScale: Optional[list[RatingScaleEntry]] = None

    @model_validator(mode="after")
    def check_instructions(self):
        if len(self.instructions) > 5000:
            raise ValueError(
                f"Instructions exceed 5000 char limit ({len(self.instructions)})."
            )
        if "{{prediction}}" not in self.instructions and "{{prompt}}" not in self.instructions:
            raise ValueError(
                "Instructions must contain at least {{prompt}} or {{prediction}}."
            )
        return self

    @model_validator(mode="after")
    def consistent_scale_types(self):
        if not self.ratingScale:
            return self
        types = set()
        for entry in self.ratingScale:
            if entry.value.floatValue is not None:
                types.add("float")
            if entry.value.stringValue is not None:
                types.add("string")
        if len(types) > 1:
            raise ValueError("ratingScale mixes float and string values. Use one type.")
        return self


class CustomMetric(BaseModel):
    customMetricDefinition: CustomMetricDefinition


def validate(raw: str) -> tuple[bool, list[str]]:
    """Validate a JSON string of custom metrics. Returns (ok, errors)."""
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as e:
        return False, [f"Invalid JSON: {e}"]

    if not isinstance(data, list):
        return False, ["Must be a JSON array of metric definitions."]
    if len(data) == 0:
        return False, ["Array is empty — need at least one metric."]
    if len(data) > 10:
        return False, [f"Too many metrics ({len(data)}). Maximum is 10."]

    errors = []
    for i, item in enumerate(data):
        try:
            CustomMetric.model_validate(item)
        except Exception as e:
            errors.append(f"Metric [{i}]: {e}")

    return len(errors) == 0, errors


def main():
    if len(sys.argv) < 2:
        print("Usage: python validate_custom_metrics.py '<json>' | file.json")
        sys.exit(1)

    arg = sys.argv[1]
    try:
        with open(arg, encoding="utf-8") as f:
            raw = f.read()
    except (FileNotFoundError, IsADirectoryError):
        raw = arg

    ok, errors = validate(raw)
    if ok:
        count = len(json.loads(raw))
        print(f"✅ Valid — {count} custom metric{'s' if count != 1 else ''} defined.")
    else:
        print("❌ Validation failed:")
        for err in errors:
            print(f"  - {err}")
        sys.exit(1)


if __name__ == "__main__":
    main()

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS Labs
Publisher
AWS Labs (Official)
Source
awslabs/agent-plugins · plugins/sagemaker-ai/skills/model-evaluation · commit fbefb37
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by AWS Labs or AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download model-evaluation.zip