Model Evaluation

Evaluation Type Guide

Help the user decide which evaluation type to use based on their goals and constraints.

Evaluation types at a glance

When to use which — in short:

• To assess subjective qualities like tone, helpfulness, coherence, or faithfulness → LLM-as-Judge

• When a programmatic approach can give a meaningful signal about output quality → Custom Scorer

Decision flow

Work through the steps below in order. For each, use what you already know from conversation history, plan.md, workflow_state.json, or other files you've read. Only ask the user if you genuinely don't know.

Step 1: Check for evaluation dataset

For this step, you need to know: whether the user has an evaluation dataset.

If you don't know from previous context, ask:

"Do you have an eval dataset?"

⏸ Wait for user.

If the user does not have one:

"All supported evaluation types require an evaluation dataset. Unfortunately, this skill can't help with model evaluation without one."

Stop here. Do not offer to help create or find a dataset, since our skills do not support this.

If the user has an evaluation dataset, continue.

Step 2: Check model compatibility

For this step, you need to know: what type of model is being evaluated (open source or Nova).

If you don't already know from conversation context, try to determine it:

1. If you have the training job name or ARN, use the AWS MCP tool list-tags on the training job ARN and look for the sagemaker-studio:jumpstart-model-id tag.

   • Contains "nova" (e.g., nova-micro, nova-lite, nova-pro) → Nova

   • Anything else (Llama, Mistral, Qwen, GPT-OSS, DeepSeek, etc.) → OSS

2. If you have a Model Package ARN, use the AWS MCP tool describe-model-package and check the model description or source tags for the same model ID.

3. If neither is available, ask the user:

   "What model are you evaluating — is it a Nova model or an open-source model (like Llama, Mistral, Qwen, etc.)?"

If the model is Nova, LLM-as-Judge is not supported. Tell the user:

"Unfortunately, LLM-as-Judge isn't available for Nova models. Can we use Custom Scorer instead?"

If Nova -> Skip to step 4.

Else -> Continue to Step 3 with the remaining options.

Step 3a: Understand the task and data

For this step, you need to understand: what the model does, what the evaluation data looks like, and what "success" means for this task.

If you don't already have a clear picture from conversation context, ask:

"Can you tell me about the task you're focused on? Please explain what you want your model to do and what your evaluation dataset looks like."

You need enough context to reason about Steps 3b and 3c. If the user's answer is vague, ask a follow-up before moving on.

⏸ Wait for user.

Step 3b: Assess Custom Scorer signal strength

Based on what you know about the task and data, think about: how strong of a signal a custom (programmatic) scorer could give us about task success

Rate the signal strength as strong, medium, or weak:

• Strong: A programmatic check can reliably tell you whether the output is correct. Examples include math problems with numerical answers, classification tasks with labels, or extraction tasks with exact ground truths.

• Medium: The task has reference answers, and programmatic comparison gives a useful but imperfect signal. Examples include summarization or Q&A where text comparison against reference answers captures something meaningful, or format compliance checks where you can verify structure even if you can't verify content quality.

• Weak: What matters about the output is hard to capture in code. There may be no reliable reference answer to compare against, or the reference doesn't capture what the user actually cares about.

Think broadly — even tasks that seem subjective may have a programmatic angle.

Step 3c: Assess LLM-as-Judge signal strength

Based on what you know about the task and data, think about: how strong of a signal an LLM judge could give us about task success

Rate the signal strength as strong, medium, or weak:

• Strong: Key model quality metrics are inherently subjective — helpfulness, coherence, tone, etc.

• Medium: The task has some subjective element, but also a clear factual or structural component that a programmatic approach could partially cover.

• Weak: The task has a single objectively correct answer, and a judge model carries the risk of hallucinating, while a programmatic check would be more reliable.

Think broadly — LLM-as-Judge can surface issues that are hard to anticipate with code, but it's not always the best tool for the job.

Step 3d: Check cost sensitivity

For this step, you need to know: how important keeping costs low is to the user.

LLM-as-Judge invokes a model to score each sample, which adds cost. Custom Scorer runs your own code, which is cheaper.

If you already know from context, skip to Step 4. If not, ask:

"On a scale of 1-5, how important is it to you to keep evaluation costs low, even if it means less nuanced results? 5 means prioritize budget above all else."

⏸ Wait for user.

Step 4: Recommend an evaluation type

Use the signal strength assessments and cost sensitivity to make a recommendation:

• Nova model → recommend Custom Scorer (only available option).

• Custom Scorer signal is strong → recommend Custom Scorer. It's deterministic, reproducible, and cost-effective. A programmatic approach gives you a reliable signal for this task.

• Cost sensitivity is very high (4-5) and Custom Scorer signal is weak but not totally absent → recommend Custom Scorer, but be upfront that the programmatic signal may be limited for this task. A partial signal at low cost may be preferable to a richer signal at higher cost.

• LLM-as-Judge signal is strong and Custom Scorer signal is weak → recommend LLM-as-Judge. The task needs the kind of nuanced judgment that only an LLM can provide.

• Both have medium or strong signal → Carefully weigh the customer's cost concerns with the benefits of each eval type. Recommend the one that you think fits all of their needs the best.

Present your recommendation with a brief reason:

"Based on what you've told me, I'd recommend [evaluation type] — [one sentence explaining why]. Want to go with that?"

⏸ Wait for user to confirm.

Once the user confirms, return to the main SKILL.md workflow (Step 2: Validate and hand off).

Custom Scorer: choosing the right scorer type

If the user chose Custom Scorer, use this logic to recommend the specific scorer:

Decision logic:

1. If task is math with verifiable answers → recommend Prime Math.

2. If task is code generation with testable I/O → recommend Prime Code.

3. Otherwise → recommend Custom Lambda.