AWS Observability

CloudWatch Omni: Agent Evaluation

Tool capability: the AWS CLI (aws___call_aws, or the CLI from a shell) for the AgentCore Evaluations control plane (bedrock-agentcore-control) and data plane (bedrock-agentcore), plus the CloudWatch Omni SQL query surface for the agent's own telemetry and the stored scores. SQL syntax (tables, the required `@timestamp` bound, bracket-notation field access, quoting, CAST, JOINs) lives in query/sql-logs-traces.md — this file adds only the evaluation-specific schema and patterns and does not restate it.

Use when the user asks to:

• "Score/evaluate these traces" / "How helpful was my agent on this trace/session?" (on-demand)

• "Which evaluator should I use?" / "What evaluators are available?"

• "Set up online / continuous evaluation" / "monitor my deployed agent's quality"

• "Create a dataset from traces" / "build a regression / golden set"

• "What did my evaluation find?" / "how are my agent's scores this week" / "which sessions scored low and why" (read back stored scores)

• "Is my agent's instrumentation healthy / are traces flowing end-to-end?" (audit)

Deploying an agent so that traces reach CloudWatch (AgentCore, ECS, Lambda, EC2 env vars, IAM, ADOT floors) is instrumentation, not evaluation — see setting-up-cloudwatch-observability → references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md ("Production deployment").

1. What agent evaluation is

Agent Observability quality evaluation scores an AI agent's OpenTelemetry traces with an evaluator — a judge (LLM-as-judge or code) that reads a trace's spans and returns a score (numeric value and/or categorical label) with an explanation. Three ways to run it, all writing their scores into the same telemetry store so they can be read back later:

Every online and batch score, and every numeric on-demand score, lands as a gen_ai.evaluation.result log record in logs.default (Section 7). The one exception: an on-demand categorical-only result (a label with no numeric value) lives only in the helper's receipt and is never written back (Section 4) — read it from the receipt, and do not report "no results" when logs.default has no row for it. The agent's own spans are in traces.default; the two are separate records joined by traceId.

Evaluators come from list-evaluators (Section 3): Builtin.<Name> (AWS), ThirdParty.<Provider>.<Name>, and any Custom/CustomCode evaluators created in the account. Each has a level — TRACE (whole request/response), TOOL_CALL (individual tool invocations; needs tool spans), SESSION (whole conversation) — and every evaluator in one scoring call shares one level.

Prerequisites (every path): the agent must be emitting telemetry with CloudWatch Transaction Search enabled, and there is a 2–5 min ingestion lag before a fresh session is evaluable. Custom evaluators that invoke Bedrock need bedrock:InvokeModel on the caller (on-demand) or on the execution role (online); built-in evaluators do not — the service invokes the judge model itself. When Omni itself runs the model (Agent Playground, or an evaluation started from Omni), the call goes through the Space's space access role, which needs the optional CloudWatchOmniModelInferencePolicy; a Bedrock AccessDeniedException there is that policy missing. Attaching it is setup: route to the setting-up-cloudwatch-observability skill. All of these resources are Region-scoped — they appear only in the Region they were created in.

Operating contract

Before any action that scores, creates, edits, or deletes, confirm the specifics with the user first. This contract governs every step below, even where a step does not restate it:

• The evaluator set is the user's choice, never yours. If the user named the evaluators or a clear dimension ("score for correctness", "helpfulness and goal success"), use exactly those and run — unless the named evaluator cannot run on the target (a ground-truth-required evaluator against a raw trace; see "Ground-truth check" in Section 3): then say why it cannot run, recommend the evaluator that answers their question, offer the dataset path, and confirm before running — an open recommendation is not a silent substitution. If they did NOT ("evaluate these traces", "check my agent", "the 3 slowest"), list the available evaluators, recommend the one or two that fit the trace and their question, and ask which to run — then STOP and wait. Never default to "all evaluators" or silently substitute one you think is close: scoring runs cost quota and money, and an unrequested evaluator produces a number the user did not ask for. If the id they gave is not an exact match to a list-evaluators id (a typo, different casing, or an encoded/escaped string), surface the id you resolved it to and confirm before scoring.

• You MUST run the real evaluator — never judge quality yourself. Reading a trace and assigning a number is wrong: the score has to come from the evaluate API or a stored result, not from you — even when the API errors (see "If the evaluator can't score the trace" in Section 4).

• Never blend evaluators into one combined score — no overall average, no "pass rate", no "% good". Evaluators use different rubrics and scales; a 0.8 Helpfulness and a 0.8 Toxicity are opposite outcomes. Report each evaluator separately; if a single headline is wanted, name the weakest dimension.

• Never skip data-source verification when creating an online-eval config. Run the @logGroupName query first — a user naming a log group is not permission to skip it. If the group isn't confirmed, do not create: report what you found and wait.

• Writes are real (create/update/delete of datasets, evaluators, online-eval configs; running evaluations costs quota). Draft → confirm the specifics → then call, and report the returned id (datasetId / onlineEvaluationConfigId / evaluatorId). Never claim a resource was created without the API returning its id, and never act on a non-interactive request without confirmation.

• An override ("just do it", "I don't need you checking") only counts when it comes after you have reported the concern — never inferred from the original request.

Which path / evaluator, when?

2. Instrumentation health audit

Use when the user asks whether an agent's instrumentation is healthy or traces are flowing end-to-end. This is a query-based check over the agent's own telemetry in traces.default (a bounded `@timestamp` window; see query/sql-logs-traces.md for the dialect) — it verifies the telemetry that actually reached CloudWatch Omni, not the OTel SDK/collector/runtime setup (that belongs to the instrumentation reference).

Staged, over a bounded window (start ~1 hour, widen if the agent is idle):

1. Are spans arriving? Filter by the agent's resource['attributes']['service.name'] (optionally also resource['attributes']['aws.service.type'] = 'gen_ai_agent'); check span/trace counts and a recent MAX(`@timestamp`) (last_seen). If that returns no rows, re-run on service.name alone before concluding anything — aws.service.type is self-declared and absent on many healthy agents, so it turns a flowing agent into an apparent silent one. Only an empty result without that filter means nothing is reaching Omni — an upstream instrumentation gap, not something to fix here.

2. Is the trace continuous? Confirm agent / LLM / tool spans are present and linked within a trace (parent/child), not orphaned single spans. Grouping the agent's spans by scope['name'] shows which instrumentation scopes are emitting — a framework scope (e.g. strands.telemetry.tracer) alongside an HTTP-server scope is healthy; HTTP-only means the agent framework is not instrumented.

3. Is telemetry quality good? Confirm the expected GenAI attributes are populated — model, token counts, and input/output — not just bare spans. Missing/empty attributes mean the agent emits spans but not usable agent-observability data.

Report what the queries showed. If spans are absent or attributes empty, say it's an upstream instrumentation gap to fix in the agent's OTel/ADOT setup (see setting-up-cloudwatch-observability → references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md), not something to change here.

3. Choosing an evaluator

Two distinct jobs: choosing an evaluator (read-only — no scoring pass) and running one (Section 4). When the user asks "which evaluator should I use?" or is unsure, answer WITHOUT scoring anything.

Discover evaluators first — never hardcode ids

aws bedrock-agentcore-control list-evaluators

Each summary carries evaluatorId, evaluatorName, evaluatorType (Builtin | ThirdParty | Custom | CustomCode | CustomDerived), provider (e.g. AWS, DeepEval, AutoEval), level (TRACE | TOOL_CALL | SESSION), description, status, and lockedForModification (whether the evaluator can be modified — relevant before proposing an edit), plus evaluatorArn, createdAt, updatedAt and, where set, kmsKeyArn — enough to choose from without any further call.

Call aws bedrock-agentcore-control get-evaluator --evaluator-id <id> only when a question actually needs more — the rating scale under evaluatorConfig, or a custom evaluator's judge model. The detail response does not carry machine-readable input-requirement flags — judge an evaluator's ground-truth needs from its identity and description (below). For a field's exact semantics, consult the current AWS documentation rather than guessing.

Use only the ids list-evaluators returns. The catalog varies by account and Region and changes over time, so a name baked into this doc goes stale. Current accounts expose Builtin.<Name> (e.g. Builtin.Helpfulness, Builtin.Correctness, Builtin.ToolSelectionAccuracy, Builtin.GoalSuccessRate) and third-party evaluators namespaced ThirdParty.<Provider>.<Name> (e.g. ThirdParty.DeepEval.ToolUse), plus any custom evaluators created in the account — but treat the live list as the only source of truth. An id you did not see in this account's list output may not exist here; do not recommend or run it.

Listed ≠ invocable. If evaluate rejects an id that list-evaluators returned (e.g. ValidationException: Unknown evaluator), treat it as a service-side issue, offer a different evaluator, and never self-assign a score to compensate.

Ground-truth check (before recommending for on-demand)

A live trace carries no ground truth, so an evaluator that needs an expected value cannot score it on-demand. There is no machine-readable requirement flag — decide from the evaluator's identity and description:

• Ground-truth-required (on-demand: no) — the trajectory matchers Builtin.TrajectoryExactOrderMatch, Builtin.TrajectoryInOrderMatch, Builtin.TrajectoryAnyOrderMatch compare the actual tool sequence against an expected trajectory. They only work against a dataset whose examples carry expected_trajectory (Section 6) — never on a raw trace.

• Ground-truth-free (on-demand: yes) — response-quality/safety judges (Builtin.Helpfulness, Builtin.Coherence, Builtin.Faithfulness, and the like) and the tool judges Builtin.ToolSelectionAccuracy / Builtin.ToolParameterAccuracy score a trace on its own merits. For "does my agent pick the right tools?" use Builtin.ToolSelectionAccuracy, not a trajectory matcher. Builtin.Correctness and Builtin.GoalSuccessRate also run without ground truth (they can optionally use it, but do not require it).

When a user's question maps only to a ground-truth-required evaluator, say the raw-trace path can't supply the expected values and offer the dataset path (Section 6) instead of substituting an evaluator that doesn't answer their question.

When you redirect an evaluator, tell the user ALL of this: which evaluator actually answers their question (for "does it pick the right tools?" that is Builtin.ToolSelectionAccuracy, run at TOOL_CALL level); that the trajectory matchers (exact-order, in-order, any-order) are ground-truth-required — they compare the actual tool sequence against an EXPECTED trajectory; that a live trace carries no ground truth, so a matcher cannot score raw traces on-demand and only works against a dataset whose examples carry an expected trajectory; that the dataset path (Section 6) is the alternative if they want the matcher; and that there is no machine-readable input-requirement flag on the detail API — an evaluator's ground-truth need is judged from its identity and description.

Level → target rules

Every scoring call runs at ONE level, and the level decides what gets scored:

• TRACE — whole request/response interactions. The default for "is this trace any good?" questions.

• TOOL_CALL — individual tool invocations within a trace. Use for "did it pick the right tool?" / "were the tool calls correct?". Only works when the trace actually has tool spans.

• SESSION — the whole conversation across the trace.

All evaluators in a single scoring call must share that one level. The level is a property of the scoring call (the --level argument), set explicitly — it is not inferred from the evaluators; each evaluator's own level must match the call's level, which is why evaluators at different levels need separate calls.

When a request implies more than one level (e.g. a request that names both a per-tool-call check and a whole-conversation session-level check together), tell the user ALL of this: every scoring call runs at exactly ONE level and all evaluators in a call share it; the two asks map to different levels (TOOL_CALL vs SESSION) so they need SEPARATE calls; briefly define the levels involved (TRACE = whole request/response, TOOL_CALL = individual tool invocations, SESSION = whole conversation); and TOOL_CALL only scores when the trace actually has tool spans. Then confirm the evaluator choice before running (operating contract).

Recommend

Recommend the evaluator whose level and description match the trace and the user's question. Ask the user to confirm when the intent is genuinely ambiguous (e.g. "helpfulness" vs. "correctness" for the same trace) rather than guessing — and per the operating contract, never start scoring until the user has chosen.

A question that only asks which evaluators to use still starts from this account's list-evaluators output: it is a read, not scoring, so run it and recommend from what it returns rather than from the examples in this doc. For retrieval, Builtin.Faithfulness judges grounding in the retrieved context, alongside a response-quality judge and, when retrieval is a tool call, Builtin.ToolSelectionAccuracy; say which candidates would need a dataset.

4. Running an on-demand evaluation

Best when the user has a specific session/trace and wants a score now — no dataset and no online config are required, so this is the answer to "score this one trace without setting up a whole pipeline." The score has to come from the evaluate API (service bedrock-agentcore, data plane), not from you.

Use the bundled helper — do not hand-build an evaluation request

evaluate needs the trace's spans reconstructed into the exact sessionSpans shape, with each span's kind coerced to a bare OTel string (SERVER/INTERNAL, never a number or null) or it fails with ValidationException: Failed to parse span data. Do not hand-shape spans in the conversation — reshaping turn-by-turn is exactly where a one-shot flow thrashes (reshape → ValidationException → retry → loop). Do not instead call any cloudwatch-omni Evaluate operation or assemble structuredInput (input / output / toolCalls extracted from spans) by hand: that is a different API on a different shape. If you catch yourself writing code to pull messages/tool-calls out of spans, stop and use the helper.

This skill bundles a tested helper, scripts/cloudwatch-omni/evaluate_traces.py, that does resolve-session → fetch spans from traces.default via Omni SQL → normalize kind → call evaluate deterministically:

1. Pick the evaluator (Section 3) — e.g. Builtin.Helpfulness (TRACE level). A raw trace has no ground truth, so pick a ground-truth-free evaluator; ground-truth evaluators belong to dataset-based evaluation (Section 6).

2. Resolve the trace ids if the user named traces by symptom rather than id (see "Query → evaluate chaining" below).

3. Retrieve and run the helper (this skill's supplementary file — fetch it, then run it with your shell):

   Copy

   # one trace, one evaluator
   python evaluate_traces.py --trace-id <traceId> --evaluator-id Builtin.Helpfulness \
     --level TRACE --region <region> --include-explanations
   # many traces and/or many evaluators — ONE run, each trace fetched once, every
   # (trace, evaluator) pair scored:
   python evaluate_traces.py --trace-ids <id1>,<id2>,… \
     --evaluator-ids Builtin.Helpfulness,Builtin.Correctness --level TRACE --region <region> --include-explanations
   # a different level is a separate run — tool judges score at TOOL_CALL, not alongside TRACE:
   python evaluate_traces.py --trace-ids <id1>,<id2>,… \
     --evaluator-ids Builtin.ToolSelectionAccuracy --level TOOL_CALL --region <region> --include-explanations

   It accepts one or many trace ids (--trace-id / --trace-ids) and one or many evaluator ids (--evaluator-id / --evaluator-ids), auto-resolves each trace's session (--session-id pins it for a single trace), fetches every session's spans from CloudWatch Omni (traces.default) via Omni SQL once, normalizes every span's kind, scores each (trace, evaluator) pair via evaluate (level → target: TRACE → traceIds; TOOL_CALL → spanIds; SESSION → no target), and prints a compact JSON receipt — a per-(trace, evaluator) results matrix (traceId / evaluatorId / value / label / explanation, plus errorCode if the evaluator returned one) with a small rollup. Raw spans never enter the conversation.

4. Report the receipt — carry value / label and the explanation as the "why." You are not done until the helper prints a results array (or an error).

Query → evaluate chaining (resolving trace ids)

If the user names traces by symptom ("my slowest traces", "the failing ones") or by service and time window ("traces for checkout-api in the last hour") rather than by id, first resolve the ids with a telemetry query over traces.default, then pass those ids to the helper. Never fetch or reason over raw spans yourself — the helper fetches and reshapes them.

Candidate traces are the user's AGENT traces. When resolving trace ids to score without a named service, fetch the user's agent traces (resource['attributes']['aws.service.type'] = 'gen_ai_agent'), not arbitrary application/service traces. If that returns nothing, the attribute is missing rather than the agent silent — it is self-declared and many agents omit it. Do not simply drop it here, or the query widens to every service in the Space: replace it with the instrumentation-scope filter below (scope['name'] = '<agent framework scope>'), which identifies agent traces more precisely anyway because it is the same allowlist evaluate enforces. Say which discriminator you used.

Filter to a supported instrumentation scope. evaluate accepts only spans whose scope['name'] is on the service's allowlist of agent-framework instrumentations (for Strands, strands.telemetry.tracer; the rejection message lists the current allowlist verbatim). An instrumented agent also emits HTTP-server spans (e.g. opentelemetry.instrumentation.starlette) that arrive as traces of their own — health checks and other requests that never reach the agent — and on a busy agent those shells can outnumber real agent traces twenty to one. A "newest N traces for this agent" query with no scope filter therefore returns mostly shells, and every one of them is rejected with ValidationException: Provided input has no spans with supported scope. Add the scope filter to the trace-resolution query:

SELECT traceId FROM "traces.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '2 HOUR' AND NOW()
  AND resource['attributes']['service.name'] = '<agent service.name>'
  AND scope['name'] = 'strands.telemetry.tracer'
GROUP BY traceId ORDER BY MAX(`@timestamp`) DESC LIMIT <N>

There is deliberately no aws.service.type filter in that query: service.name and the framework scope together already pin it to this agent's own agent-framework spans, so the attribute adds nothing and — being self-declared and often absent — would only turn a good result into an empty one.

If you do not know which scope the agent emits, first group the agent's spans by scope['name'] and pick the framework scope, not the HTTP one. Group without the aws.service.type filter when you do — an agent that does not declare it would otherwise show no scopes at all.

• Ask for one row per trace. Group by the trace id and take the newest timestamp per group. DISTINCT over (trace id, timestamp) is the trap: a trace holds ~15–40 spans with ~15–40 distinct timestamps, so it yields one row PER SPAN, and a row limit then bounds spans rather than traces — a 50-row result can cover as few as 2 traces.

• Ask for the ids alone. A timestamp or row-number column roughly halves how many ids fit in one result, and the helper does not need either.

• Count what you got back, every time. The query result is length-capped, so a request for N ids can return fewer without any error — and a clipped id is unusable (a trace id is exactly 32 hex characters; anything shorter is a fragment, not an id). Count the complete ids and drop any fragment silently — never echo, enumerate or number the ids in your reply; report the COUNT, never the list. If you have fewer than the user asked for, either fetch the remainder (query again for the window older than your earliest id) or say plainly how many you got. Never score a short set silently — "You asked for 80; the query returned 55, so I scored 55" is required.

One run per homogeneous grouping

Each run is a full matrix: every evaluator is applied to every trace, at one level. Group the work by "these traces → this set of evaluators at this level":

• "Score traces A, B, C for helpfulness and correctness" → ONE run (three traces, two evaluators, TRACE).

• "Score A and B for helpfulness, but C for tool selection accuracy" → TWO runs (one per homogeneous grouping), because the trace subsets and evaluators differ. Decide the groupings yourself and make the separate runs.

• A trace is not an agent boundary. When one agent calls another (an orchestrator and a sub-agent, possibly on different frameworks), a single trace holds both agents' spans, and scoring the whole trace judges the pair together: the sub-agent's tool calls and outputs enter the payload. To score one agent, pass --service-name <that agent's service.name> so the helper fetches only its spans, and say which service was scored.

Volume — the helper batches in one run; don't loop it or hand-roll a scorer

Pass every trace id and evaluator id to a single run (--trace-ids, --evaluator-ids); it fetches each trace once and scores every (trace, evaluator) pair. One run covers at most 50 traces, at most 10 evaluators, and at most 100 (trace × evaluator) pairs — three independent bounds, so 60 traces × 1 evaluator is over the limit even though 60 pairs < 100. The helper truncates with a notes entry in the receipt rather than silently. Before running, check the trace count and the evaluator count separately, then the product:

• Within all three → run it in one call.

• Over any of them → do NOT quietly score a subset. Tell the user the limit up front and offer to narrow the time window, use fewer evaluators, or score the first 50 and continue in a follow-up — let them choose. "Evaluate 80 traces" is NOT satisfiable in one pass; say so plainly rather than scoring 50 and presenting it as the answer. Whole-window / "evaluate everything" at large scale is not supported here — say so rather than appearing to do it.

• Account quotas are 100 evaluations/min and 200,000 input tokens/min per Region per account, so a big matrix throttles and takes a while; the helper paces within a run. A throttled call comes back as a run-level failure, not a per-cell one — retry with fewer traces or fewer evaluators rather than assuming the traces were bad.

• If the cap truncated, an error hit partway, or you scored only a subset, report "scored N of M" and name what was skipped (the receipt's notes / fetchErrors / scoreErrors carry this). Never present a partial pass as if it covered everything.

If the evaluator can't score the trace, report that — never self-judge

When the receipt has a top-level error (e.g. the evaluate call raised) or a results[].errorCode, tell the user the evaluator couldn't score this trace and give the exact error/errorCode verbatim, then STOP. Do not fall back to reading the trace and assigning a number yourself. Per-trace errors the service surfaces:

• AgentSpanMappingException — span shape not recognized.

• ValidationException: Failed to parse span data — span payload malformed.

• ValidationException: Provided input has no spans with supported scope — the trace has no span from an allowed agent-framework instrumentation scope, usually an HTTP-only shell trace picked up by a scope-blind "newest N" query. That rejection is deterministic per trace, so do not retry it: re-resolve the trace ids with AND scope['name'] = '<supported scope>' (the error lists the current allowlist) and score those.

For other fixable span issues you may retry the helper once; if it still errors, report the error. When you report an evaluator error — or, with no error text supplied, ask for it — say which of these three it is (or likely is) and, for the unsupported-scope one, state explicitly that it is deterministic per trace and will not be retried as-is; re-resolving to a supported scope is the only fix.

Scores are persisted so they can be read back later

When the user asks to STORE / persist / "query later" scores, tell them ALL of this before writing:

1. evaluate returns the score inline but does NOT store it — the helper writes each back as a gen_ai.evaluation.result record so it is queryable later (Section 7).

2. NAME the results log group /aws/cloudwatch/evaluations/results/<sanitized-service-name> and say the helper CREATES it if absent. The helper sanitizes non-safe characters to _ and truncates to the log-group name limit, so the final segment may not equal the literal service name.

3. WARN it stores the judge's prose explanation, which quotes the agent's own inputs/outputs — the group inherits the conversations' sensitivity and is readable by anyone with log read access.

4. RECOMMEND retention (--retention-days) and a customer-managed KMS key (--kms-key-id); note the data is billed like any other log data.

5. Note writeback is BEST-EFFORT and NUMERIC-only — a categorical-only result stays in the receipt, not the log group; a missing logs:PutLogEvents leaves the score non-durable.

6. Mention --no-writeback for users who do not want scores in their logs.

evaluate returns a score inline but does not store it, so the helper also writes each one back as a gen_ai.evaluation.result telemetry record under /aws/cloudwatch/evaluations/results/<sanitized-service-name> — the same shape online evaluation emits. A score therefore outlives this conversation and can be queried later (Section 7), by a later session or another user, alongside the online results. This is best-effort: the receipt reports persistedTraces (how many traces' scores were written) plus the resultsLogGroupPrefix, or carries a notes entry explaining why a write did not happen (for example the caller has no logs:PutLogEvents — the score is still in results, it just is not durable). Pass --no-writeback when the user does not want scores written into their logs. Writeback covers numeric scores only — a successful row whose value is None (categorical-only) is returned in the receipt but not written back; the helper also filters failed jobs on the write path so they never persist for on-demand.

Writeback creates a log group, and what it stores is sensitive. If /aws/cloudwatch/evaluations/results/<sanitized-service-name> does not exist, the helper creates it — with 90-day retention by default (--retention-days to change; 0 opts out), and optionally a customer-managed KMS key if --kms-key-id is passed — and its stored data is billed like any other log data. The records include gen_ai.evaluation.explanation, the judge's prose, which quotes the agent's own inputs and outputs — so this log group inherits the sensitivity of the conversations being scored, and it is queryable later by anyone with read access to it. Name the results log group to the user before the first writeback (the receipt reports resultsLogGroupPrefix but cannot tell you whether it already existed — use aws logs describe-log-groups --log-group-name-prefix if you need to know), and recommend they set retention (aws logs put-retention-policy) and attach a KMS CMK (aws logs associate-kms-key) so scored prompt and completion text is not held indefinitely in plaintext.

Synthesizing the receipt

A run returns one receipt whose results is the per-(trace, evaluator) matrix — each row a numeric value and/or a categorical label (which one depends on the evaluator's rating scale), the evaluator's explanation, and an error marker when that cell could not be evaluated. Turn the matrix (and across runs if you had to split) into a finding, not a table dump. A per-evaluator mean is the FLOOR, not the answer.

Know the rating scale before you summarize — value and label are two shapes:

• Numerical evaluator (has a value): summarize the distribution — the range and how many sit at the low end — not just a mean. A mean of 0.6 hides "6 great + 4 zero" vs. "all mediocre". Any label here is a natural-language gloss on the number (e.g. 0.5 = "Somewhat Helpful") — carry it, don't treat it as the score.

• Categorical evaluator (has a label, no value): there is nothing to average — report a count per category ("6 Excellent, 3 Good, 1 Poor").

Good vs. bad is NOT in the result — it comes from the evaluator's scale. A low number is not automatically bad and 0 can be the DESIRED outcome (e.g. a refusal or toxicity check). The direction, range, and category ordering live on the evaluator's configured scale (get-evaluator when you need it), never on the raw magnitude or on the label TEXT (a free-form descriptor, not a pass/fail flag). When the direction isn't obvious, read the explanation before you call an outcome good or bad. See "Polarity" in Section 7 for the known inverted evaluators.

Then find the patterns the numbers hide — this is where the insight is:

• Cluster the poor cells by shared CAUSE from their explanations. When several traces come out badly for the same reason, say it once and name those traces ("4 traces failed Correctness — all invented a region value not in the input: T2, T4, T7, T9") rather than restating a reason per row.

• Always carry the WHY, scaled to size. A few notable cells → give each one's reason. Many that share a cause → the cluster's reason once. Many notable cells with VARIED reasons → give a high-level characterization (the main themes, or how many distinct causes) and invite the user to probe specific traces/evaluators — do NOT enumerate every reason, and do NOT drop the reasoning entirely.

• Roll up the OTHER axis: call out traces that come out poorly across MULTIPLE evaluators (a broadly-bad response, worth prioritizing) — distinct from one that only dips on a single dimension.

• Surface the outliers and the errored cells specifically — the worst results and any cell that could not be evaluated — not every row.

Shape of the answer — always deliver the insight; scale only how much raw detail rides with it:

• Small matrix → detail is affordable. A compact view of the actual cells (a small table is fine) plus the takeaway. One trace × several evaluators is a single compact row, NOT a section per evaluator.

• Large matrix → summarize INSTEAD of dumping the grid — never skip the analysis. Lead with the distribution / category counts, the clusters + worst traces, and WHY the notable ones landed there. "Large" means "summarize, not enumerate"; it never means "give a bare number and stop."

• Uniform results → collapse, at any size. If everything landed the same (e.g. all good), one line says so; break out a specific evaluator or trace ONLY where it deviates.

• Never blend evaluators into one combined score (operating contract) — a scoring rule, NOT a layout rule; it does not require a separate table per evaluator.

• Close on what the finding sets up — a failing cluster is a ready-made set to re-check after a fix, or to build an evaluation dataset from (Section 6).

For many sessions or continuous coverage, use online evaluation (Section 5).

5. Online (continuous) evaluation and custom evaluators

Online evaluation — score live traffic over time

Best for continuously monitoring a deployed agent — it samples live sessions and scores them, accruing results in logs.default. When the user wants an agent's quality monitored and has not named the evaluators, start with the traffic profile in Section 8 ("Set up quality monitoring") and choose the evaluators from it before the steps below. All ops on bedrock-agentcore-control:

status (ACTIVE / CREATING / …) is the provisioning state only; executionStatus (ENABLED / DISABLED) is the separate field that switches scoring on or off — ACTIVE says nothing about whether scoring is enabled. And neither field proves scores are being produced: a config with a misresolved data source goes ACTIVE/ENABLED and scores nothing (see the @logGroupName rule below). To check that a config is actually scoring, count the recent gen_ai.evaluation.result records in logs.default whose attributes['aws.bedrock_agentcore.online_evaluation_config.name'] is that config (Section 7). None, once enough sampled sessions should have arrived (allow the 2 to 5 min ingestion lag), points first at a misresolved data source. Online evaluation only scores sessions arriving after it is enabled; it never back-fills. For earlier sessions, offer on-demand scoring (Section 4, within its volume limits) or a dataset built from those traces (Section 6). This sets up and manages scoring; to report what it found, query the evaluation-result telemetry (Section 7), not the config.

When the user asks to set up online / continuous evaluation — whether or not they name a log group — tell them ALL of this before any create: you will run the @logGroupName query against the agent's own spans first (a named group is the group you verify, not permission to skip verification); the group must never be taken from resource['attributes']['aws.log.group.names'], because that yields a config that goes ACTIVE/ENABLED and silently scores nothing; you will show the candidate groups with their span counts for approval; if the group is not confirmed you will not create — you report what you found and wait; the create is a real write — draft → confirm the specifics → call → report the returned onlineEvaluationConfigId; and you will never report it as created without that id. Say this in the answer even when you cannot run the query yet (no Region or Space given).

Gather before creating:

• Instrumentation the service can read. AgentCore Evaluations picks how to read each span from its scope.name. Group a recent window of the agent's spans by scope['name'] (Section 2, step 2) and compare against the supported agent frameworks page. A framework without its own page is still read under generic framework support when its scope starts with openinference.instrumentation. or opentelemetry.instrumentation. and its spans carry the convention's span-kind attribute (openinference.span.kind, or gen_ai.operation.name). If the agent's scopes match neither, warn that the config can go ACTIVE/ENABLED yet score nothing, link that page, and create only if the user confirms. This is a warning, not a block.

• One agent per config. dataSourceConfig.cloudWatchLogs.serviceNames is exactly one service (logGroupNames up to 10, or logGroupNamePrefixes 1–5 as the alternative). To watch N agents, create N configs — never fold several agents into one.

• Log group(s) — read them off the agent's own spans; never derive, construct, or guess a name. The @logGroupName column records the log group each span actually landed in:

  Copy

  SELECT `@logGroupName` AS log_group, COUNT(*) AS spans
  FROM "traces.default"
  WHERE `@timestamp` BETWEEN NOW() - INTERVAL '24 HOUR' AND NOW()
    AND resource['attributes']['service.name'] = '<service>'
    AND resource['attributes']['aws.service.type'] = 'gen_ai_agent'
  GROUP BY `@logGroupName`
  ORDER BY spans DESC
  LIMIT 10

  • An empty first result is a two-query question, not an answer. Delete the aws.service.type line, run it again, and judge from the second result. Whether a span carries that attribute at all is decided by the OTel/ADOT config that shipped it, not by the service — on a Space of ordinary agents, more spans lack it than carry it, and every one of those agents is invisible to the first query. Both counts come back 0 (including recordsScanned), which is also what a dead agent looks like, so the first query cannot tell you which you have. Dropping the line costs nothing: service.name is the filter that narrows this to one agent, and the group resolution asks for nothing else. Rows on the second run are your groups — take them, and note in the report which filter produced them. "No telemetry" is a conclusion only the second query can support.

  • Every returned row is a group proven to hold this service's spans — the row is the proof, so run no separate span check. Pass each name verbatim: never add or remove a leading slash (aws/spans is a real group, /aws/spans is not). Skip NULL/empty rows.

  • Pass every returned group, highest span count first, up to the 10-group cap — a service may legitimately emit to several, and passing only one leaves the rest unevaluated. If more than 10 return, pass the top 10 and say which you dropped.

  • Show the resolved group(s) with their span counts and get the user's approval before creating. service.name is self-declared by whatever emits the telemetry, so a resolved group is a proposal, not a fact to act on.

  • Never resolve from resource['attributes']['aws.log.group.names']. It is absent for many agents (it is self-declared by the OTel/ADOT config, so EKS/ECS/other-hosted ones omit it), and where present it names the runtime's <name>-<id>-DEFAULT application group — which exists and holds stdout lines but zero spans. Creating against it yields a config that goes ACTIVE/ENABLED and scores nothing, silently.

  • Always run the query before any create — no request waives it. A user naming a log group is not permission to skip verification and not consent to an unverified create; it is the group you verify.

  • When the query does not confirm the group — no rows (drop the aws.service.type line and re-run, then widen to 7 days, before concluding this), the query errored, or it returned rows and the named group is not among them — do not create. Report what the query showed (naming the groups it did return), say the group is unconfirmed and that if it is wrong the config will score nothing without erroring, then stop and let the user decide. An unconfirmed group is not necessarily wrong: an idle agent is legitimate to configure.

  • If the user answers that report by telling you to create anyway ("I don't care", "go ahead"), create it and state in the result that the log group is unverified. That override requires an instruction given after you reported the problem — never infer it from the original request.

  HARD STOP — run the @logGroupName query before every create, and never create against an unconfirmed group until the user has been told it is unconfirmed and has answered that you should proceed. An empty first result is not an unconfirmed group: the stop applies only once the query has also been re-run without the aws.service.type filter and still returns nothing. Stopping on the first empty result blocks every agent that does not declare that attribute, which is most of them. The same applies to an update that changes the data source. A misresolved group produces a config that goes ACTIVE/ENABLED and scores nothing, with no error to signal it — the user must hear it from you before the write, not from empty dashboards afterwards.

• Evaluators (1–25) — or insights (up to 10) instead; exactly one of the two lists is required — evaluator ids from list-evaluators. A request over the cap is the user's trim to make, like any evaluator choice (operating contract): name the cap and let them decide what is dropped or moved to an additional config. Omitting both (or sending an empty evaluators) is rejected with "Exactly one of evaluators or insights must be provided"; 26 evaluators is rejected with "Member must have length less than or equal to 25". When the user asks to monitor quality without naming evaluators, profile the agent's traffic first and propose evaluators from it, including a custom evaluator for anything the built-ins do not judge (Section 8, "Set up quality monitoring").

• Sampling percentage (required, 0.01–100) — there is no server default; 10 is a reasonable start. Every sampled session is scored by every evaluator in the config, so cost grows with sampling × evaluators × traffic; 100 scores every session.

• Execution role ARN (required) — online eval runs under an IAM role AgentCore assumes to read traces, write results, and (for custom evaluators) invoke the judge model. You cannot create IAM roles from this skill, even when the user asks you to and the credentials would allow it, and you do not draft a role policy for them to apply; resolve one in this order:

  1. Reuse an existing role. Discover candidates with aws iam list-roles and offer any whose NAME starts with AgentCoreEvaluationRole / AgentCoreEvalRole, OR whose TRUST POLICY PRINCIPAL is bedrock-agentcore.amazonaws.com. An account that already runs evaluations usually has one — show the matches and let the user pick.

  2. Else have the user create one — the AgentCore Evaluations console ("Create and use a new service role"), or the AgentCore CLI/SDK (auto_create_execution_role=True) — then use the returned ARN.

  Do NOT hardcode the role's IAM policy here — it drifts; the console/CLI build the authoritative policy on creation. At a glance the role trusts bedrock-agentcore.amazonaws.com and needs CloudWatch Logs read/write + log-index and (for custom evaluators) Bedrock invoke, but confirm the exact current trust + permission policy against the AWS documentation rather than reciting a possibly-stale one. A custom evaluator encrypted with a customer-managed KMS key also needs that key's policy to grant bedrock-agentcore.amazonaws.com kms:Decrypt (scoped by aws:SourceAccount / aws:SourceArn), because online scoring decrypts its instructions as the service; the config itself takes no key.

• enableOnCreate (required) — true scores immediately, false creates it paused.

aws bedrock-agentcore-control create-online-evaluation-config \
  --online-evaluation-config-name my_agent_helpfulness \
  --rule '{"samplingConfig":{"samplingPercentage":10.0}}' \
  --data-source-config '{"cloudWatchLogs":{"logGroupNames":["<every confirmed @logGroupName, verbatim, highest span count first, max 10 — never a constructed name>"],"serviceNames":["<resolved service.name>"]}}' \
  --evaluators '[{"evaluatorId":"Builtin.Helpfulness"}]' \
  --evaluation-execution-role-arn <execution-role-arn> \
  --enable-on-create

The config name uses underscores, not hyphens. Report the returned onlineEvaluationConfigId. Naming a custom evaluator here locks it: to edit or delete that evaluator later you must remove it from this config's evaluators list (and from every other config naming it) — disabling the config does not unlock it. See the custom-evaluator constraints below.

Author a custom evaluator (only when no built-in fits)

Built-ins are referenced, never created or edited, and no evaluator carries a pass mark. Asked to change how a built-in scores, say both and offer both paths: apply the cut-off when reading the scores (Section 7) or in an alert on them, or draft a custom evaluator with its own rubric. Author a custom evaluator only when the user wants something the built-in set does not score — either an LLM-as-judge (a judge model plus scoring instructions and a rating scale) or a code-based one (CustomCode). All three operations are bedrock-agentcore-control calls: create-evaluator, update-evaluator, delete-evaluator.

These are writes — draft with the user, then confirm. Gather what it should score, the level (TRACE / TOOL_CALL / SESSION — a response-quality dimension such as brand voice, tone, or conciseness is judged at TRACE level), and, for an LLM judge, the judge model, the scoring instructions, and the scale. Present the draft back before creating — its name, what it judges, the level, the judge model, and the rubric — and create only after the user confirms that specific draft. Never invent the rubric, and never treat a "yes" from an earlier turn as approval of a rubric the user has not seen. A delete is destructive and a change alters what every future run is judged on, so confirm those too.

• Read the exact evaluatorConfig shape from the CLI's own model — do not guess it. aws bedrock-agentcore-control create-evaluator help gives the required fields, types, and nesting (llmAsAJudge vs codeBased; the judge model under modelConfig.bedrockEvaluatorModelConfig; the ratingScale). If a create still fails validation, the error carries the input schema — correct against it.

• On update-evaluator, resend the FULL evaluator config; fields you omit revert.

• The name must match [a-zA-Z][a-zA-Z0-9_]{0,47} (start with a letter; letters, digits, underscores; no hyphens; ≤48 chars) and must not collide with a built-in name.

• description is a short label for the listing (≤200 chars) and is not read by the judge — only instructions is. Keep it to one crisp sentence; put the rubric in instructions.

• Creation is asynchronous (CREATING → ACTIVE, terminal CREATE_FAILED) and Region-scoped — wait for ACTIVE before scoring with it, and keep the judge model in the same Region.

• Customer-managed key (--kms-key-arn). When the user names a key, tell them all four points in the draft: only a symmetric KMS key in the evaluator's Region is accepted; it encrypts the evaluator's instructions and rating scale (without one, an AWS owned key does); update-evaluator --kms-key-arn can later rotate to a new key; and there is no way to remove it and go back to the AWS owned key. Online scoring also needs the key policy grant described under the execution role above.

• A custom evaluator named by any online-evaluation config is locked, and only detaching it unlocks. get-evaluator reports lockedForModification: true, and update-evaluator / delete-evaluator fail with Cannot update locked evaluator: <id> / Cannot delete a locked evaluator…. The release comes from removing it from the config's evaluators list — update-online-evaluation-config --evaluators '[…]' listing the ones that stay — not from switching the config off: an evaluator stays locked indefinitely while executionStatus is DISABLED, so disabling only stops the scoring you were paying for and leaves the edit just as blocked. Detach it from every config that names it, not only the one you are working on. The unlock is eventual and can take many minutes after the detach — if the next update-evaluator still reports the lock, that is the lag, not a second problem, so re-poll get-evaluator until lockedForModification is false instead of changing tack.

Judge model (LLM-as-judge). evaluatorConfig.llmAsAJudge.modelConfig.bedrockEvaluatorModelConfig.modelId must be a foundation model available in the Region — don't guess it. Discover it and fold a recommended model into the draft you present rather than asking as a separate step:

• Inference profiles (prefer these): aws bedrock list-inference-profiles --type-equals SYSTEM_DEFINED → us. / global.-prefixed ids, the form these judges use.

• On-demand text models: aws bedrock list-foundation-models --by-inference-type ON_DEMAND --by-output-modality TEXT.

Instructions must embed a placeholder valid for the level. A create with none, or with an unrecognized token, is rejected — a content rule the schema does not express. Use single curly braces ({context}); {{context}} and ${...} / %s forms are rejected.

{context} is valid at every level, so it is the safe default; add the level-specific tokens the rubric actually judges (e.g. {assistant_turn} for TRACE response quality, {actual_tool_trajectory} for SESSION tool use). Example TRACE instruction: "Evaluate the assistant's response for conciseness.\n\nConversation:\n{context}\n\nResponse:\n{assistant_turn}\n\nScore 1-5, where 5 = tight and precise and 1 = excessively verbose." If a create is rejected for placeholders, its validation error enumerates the allowed set for that level — follow that, and re-check single braces.

6. Datasets (create, manage, and build from traces)

A dataset is a versioned collection of schema-typed examples — the target for repeatable / regression evaluation, and the only way to supply ground truth (expected_response, expected_trajectory, assertions) to evaluators that need it. All ops on bedrock-agentcore-control. Optional: on-demand (Section 4) and online (Section 5) evaluation need no dataset.

Writes are real — creating/extending a dataset costs resources. Confirm with the user before any create or delete, and never act on a non-interactive request without confirmation. The bundled helper backs this up: create/add refuse to write (exit non-zero) unless you pass --i-have-confirmed, which you add only after the user has confirmed. The flag is your assertion, not a check — the helper cannot tell whether confirmation happened — and it covers only the helper's own writes, so confirm separately before a direct create-dataset-version or delete.

Dataset operations

Core model — read before creating

• schemaType is chosen at creation and is immutable:

  • AGENTCORE_EVALUATION_PREDEFINED_V1 — author-fixed conversation turns (what trace-derived examples use).

  • AGENTCORE_EVALUATION_SIMULATED_V1 — an LLM actor drives the conversation from a goal; needs actor_profile.goal, so it cannot be derived from a trace. Asked for a simulated dataset "from traces", offer the two real options, a PREDEFINED dataset captured from the traces or a SIMULATED one whose goals the user writes, and say schemaType cannot be changed once the dataset exists.

  • THIRD_PARTY_EVALUATION_V1 — a flat shape (input/expected_output at the top level, no turns), and what the console writes for a dataset built from traces, so an existing dataset is routinely this type. Read the target's schemaType with get-dataset before adding examples by hand: a PREDEFINED-shaped example is rejected with Required field 'input' is missing from example content, which fails the whole batch. The capture helper does this read for you.

  • This type was renamed from GENERIC_EVALUATION_PREDEFINED_V1, and the old name was removed rather than kept as an alias: passing it to create-dataset now fails, and a dataset created with it earlier cannot be read back until its type is migrated. If get-dataset fails on a dataset that predates the rename, that is the cause — the dataset has to be recreated (or its type migrated by the service team), not worked around from here.

• A dataset cannot be created empty — create-dataset needs a source, either inlineExamples.examples (1–1000) or an s3Source.s3Uri pointing at a JSONL file.

• Writes edit the mutable DRAFT; create-dataset-version snapshots an immutable numbered version. Evaluations that need a stable target run against a published version.

• A dataset is keyed by datasetId (the name is a display label, pattern [a-zA-Z][a-zA-Z0-9_]{0,47} — use underscores); an example is keyed by its server-minted exampleId (returned by add-dataset-examples, shown by list-dataset-examples), not its scenario_id.

• Editing or deleting an example keys off exampleId — resolve it first. update-dataset-examples and delete-dataset-examples take exampleIds. A traceId, a scenario_id, or any id the user pasted is not the server-assigned exampleId — call list-dataset-examples and resolve it to the real exampleId before the write. Passing the wrong id deletes or overwrites the wrong example. Each entry in an update MUST carry the exampleId it replaces.

• Validation is all-or-nothing. Every example is validated against the dataset's schemaType before anything lands — one bad example rejects the whole batch. Fix and resubmit; a partial batch never writes.

• Three size limits apply, not one. (1) Per request: at most 1000 inline examples / ~5 MB — 1001 rows is a 400 ("Member must have length less than or equal to 1000"). (2) Per dataset: at most 1000 examples in total across every request — the service returns HTTP 402 ServiceQuotaExceededException ("Maximum number of examples (1000) reached for this dataset") as soon as a write would push the dataset past 1000, so add-dataset-examples against a dataset that already holds examples fails even when the request itself is within limits. Count the existing examples (get-dataset / list-dataset-examples) before adding, and create a second dataset when the total would exceed 1000. (3) Per example: at most 1 MB serialized — a single oversized example is a 400 ("Example at index N exceeds maximum size of 1MB") and, being one bad example, rejects the whole batch; trim or drop long inputs/outputs before sending.

• Writes accept an optional clientToken for idempotency — pass one when retrying a create/add so a retry cannot double-write.

• delete-dataset and delete-dataset-examples are destructive and not undoable — name what will be deleted and get the user's confirmation before calling, and never widen the scope they asked for. update-dataset edits metadata only (the name and schemaType are immutable).

Example shapes

PREDEFINED example (the trace-derived shape):

{
  "scenario_id": "checkout_happy_path",
  "turns": [{ "input": "Add the blue mug and check out", "expected_response": "Order placed — #A1B2." }],
  "expected_trajectory": ["search_catalog", "add_to_cart", "place_order"],
  "assertions": ["The order confirmation number is returned"],
  "metadata": { "sourceTraceId": "<traceId>" }
}

scenario_id + turns (each input non-empty) are the minimum. Optional ground-truth fields gate which evaluators can score the example: expected_response → correctness; expected_trajectory (ordered tool names) → tool-trajectory; assertions → goal-success. Omit them for ground-truth-free scoring.

SIMULATED example (author-supplied only — an LLM actor drives the turns, so this shape cannot come from a trace); input, actor_profile.goal, and actor_profile.context are required:

{
  "scenario_id": "refund_dispute",
  "input": "I want a refund for an order that already shipped",
  "actor_profile": { "goal": "Get a full refund", "context": "Impatient customer",
                     "traits": { "tone": "frustrated" } },
  "max_turns": 8,
  "assertions": ["The agent explains the return policy"]
}

Create a dataset

aws bedrock-agentcore-control create-dataset \
  --dataset-name checkout_regressions \
  --schema-type AGENTCORE_EVALUATION_PREDEFINED_V1 \
  --source '{"inlineExamples":{"examples":[ … ]}}'

Returns a datasetId; the dataset may be CREATING briefly — re-check get-dataset until status is ACTIVE. Then create-dataset-version --dataset-id <id> to publish the immutable version a repeatable evaluation targets.

Build a dataset from traces

When the user asks to turn traces into a re-runnable / regression set, tell them ALL of this: the capture helper builds the examples FROM the traces client-side (no server-side trace→example op); a repeatable target MUST evaluate against a PUBLISHED IMMUTABLE version, not the mutable draft; creation is a real write (draft → confirm → call) and you will report the returned datasetId; and before writing, the service caps apply — ≤1 MB per example and ≤1000 examples per dataset — and both caps reject the whole add rather than trimming it: a per-example >1 MB row fails the entire add-dataset-examples batch with a 400, because validation is all-or-nothing (nothing lands unless every example passes), and the 1000-examples-per-dataset cap refuses the whole add up front with a service-quota error (create a new dataset with --mode create for the remainder). Neither is auto-split; and the capture helper itself caps each invocation at 25 traces (MAX_TRACES): if more than 25 unique trace ids are supplied, only the first 25 become examples and the receipt notes that the rest need another run — so a large trace set needs multiple invocations.

There is no server-side "trace → example" operation, so this skill bundles a tested helper, scripts/cloudwatch-omni/capture_dataset_from_traces.py, that does the whole fetch → reshape → validate → write. Do not hand-reshape spans in the conversation — the one-shot flow is error-prone and bloats context.

1. Retrieve and run the helper (this skill's supplementary file — fetch it, then run it with your shell). It queries each trace's spans from CloudWatch Omni (traces.default), converts each into an example (root-span input, last-finishing AGENT-span answer, ordered tool trajectory) in the shape the target dataset's schemaType declares, validates every example, and writes the batch:

   Copy

   python capture_dataset_from_traces.py --mode create \
     --dataset-name checkout_regressions --trace-ids <id1>,<id2>,… --region <region> \
     --i-have-confirmed

   Pass --i-have-confirmed only after the user has confirmed the write; without it the helper exits non-zero before touching any AWS API. Append to an existing dataset with --mode add --dataset-id <id> (an id or a name; a name is resolved via list-datasets) — it reads that dataset's schemaType and builds the examples in whichever shape it declares, so a console-created one needs nothing extra from you. It looks back 30 days (--window-days) and prints a compact JSON receipt (datasetId, schemaType, examplesWritten, any per-trace conversionErrors) — raw spans never enter the conversation, and a partial/invalid batch never lands.

2. Report the receipt — the returned datasetId and counts. You are not done until the helper prints a datasetId. Relay it honestly: if the receipt lists per-trace conversionErrors, say which traces did not become examples rather than reporting only the successes; and when an example is stamped metadata.partial = true (its trace's span set was truncated, so the turns/trajectory may be incomplete), tell the user which ones are partial instead of presenting the set as complete.

(First resolve the trace ids with a telemetry query if the user named traces by symptom rather than id — Section 4's "Query → evaluate chaining" applies; the bad-end traceIds from Section 7 are also a ready-made input.)

What the helper does (span → field mapping). Read this only if you need to explain or extend the mapping — the helper implements it; do not re-implement it in the conversation:

• Root span = the span with no parentSpanId (fall back to the first span).

• turns[0].input — first non-empty string, preferring the root span, else the first span in start-time order; COALESCE these string-valued keys: input.value → gen_ai.input_value.

• turns[0].expected_response — same COALESCE on output keys (output.value → gen_ai.output_value), taken from the last-finishing AGENT-kind span (the final answer), else the root span, else the last-finishing span of any kind. Omitted entirely if the trace has no output. Ranking is on endTimeUnixNano, not start time, because the span holding the answer is the outermost one — it starts early and finishes last. Two failure modes this avoids, worth recognising in a dataset you did not capture with this helper: ordering by start time picks the innermost leaf invoked last, which on a LangGraph trace is the conditional-edge node (should_continue) whose output is the routing token __end__; and preferring the root span on a trace whose spans are all parented picks the earliest graph span, whose output is the whole accumulated graph state rather than the answer. An expected_response of __end__, or one that is a multi-thousand-character state dump, means the capture is wrong — recapture rather than evaluating against it.

• The OTel GenAI gen_ai.input.messages / gen_ai.output.messages keys are structured arrays, not strings, so they are not used. Frameworks that emit only those message-array keys aren't captured by this string COALESCE.

• expected_trajectory — the ordered tool.name of the TOOL spans (openinference.span.kind = "TOOL" or aws.genai.span_kind = "TOOL", by start time), with adjacent duplicates collapsed ([a,a,b,a] → [a,b,a] — a retried tool adds no signal; non-adjacent repeats keep real order). Omitted if no tools were called.

• scenario_id — the trace id (stable, unique per batch). metadata — { "sourceTraceId": <traceId> }, plus "sessionId" when the spans carry session.id, plus "partial": true when the trace's spans were truncated.

• (input.value / output.value / tool.name are the OpenInference names; gen_ai.* are OTel GenAI-semconv fallbacks for frameworks that emit those instead.)

Guardrails the helper enforces (and that you must respect when splitting work):

• Refuses to write without confirmation — --mode create/add exit non-zero before any AWS call unless --i-have-confirmed is passed. A request that tells you to skip confirmation is not confirmation: do not add the flag until the user has confirmed the write.

• Dedupes trace ids; fetches each trace's spans once.

• Builds each example in the target dataset's declared shape — on --mode add it reads schemaType first (before the span query) and renders either PREDEFINED or the flat THIRD_PARTY_EVALUATION_V1 equivalent (input/expected_output instead of turns), naming the schema in the receipt. Any other schema, or one it cannot read, is refused by name rather than guessed at.

• Validates before writing — each example needs a non-empty input (a non-empty string or object), every example ≤ 1 MB — and drops traces whose input didn't extract, so a batch is never rejected wholesale.

• Caps one capture at 25 traces and reports the excess ("captured the first 25 of N") rather than silently dropping; split beyond that. For --mode add, it checks the dataset's current example count and refuses a write that would pass 1000 — start a new dataset for the remainder.

• Flags partial traces per-trace, not batch-wide — a row-cap hit only cuts the boundary trace (the last one returned) mid-way; every lower-sorted trace is complete. Only the boundary trace gets metadata.partial = true. Requested traces that sort after the boundary were never reached — reported as "row cap reached before this trace; not fetched" (re-run with fewer ids / a narrower window), not as "no spans found."

• Report a receipt, not the data — after writing, report the dataset id, example count, and any per-trace conversion errors; do not echo the example JSON or raw spans back into the conversation.

Summarize a dataset

Read with list-dataset-examples (paginate with nextToken) and report a compact synthesis — the count, the schema type, a few representative scenarios, notable gaps. Do not dump every example into the conversation.

7. Reading back stored scores

Reading stored scores is a two-part job: the retrieval plan and reporting (what to fetch, in what order, and how to turn rows into a finding) and the query mechanics (schema, filters, ready SQL). Every scored result — online, on-demand, or batch — lands as a logs.default record readable by any user with access, not just whoever ran the evaluation.

Retrieval plan — how to read and report

Rollup first, whenever the evaluator scope is open. The per-evaluator rollup (Q1) reveals WHICH evaluators sit at the bad end by polarity, so it runs first whenever the ask spans multiple or unnamed evaluators ("how are they doing", "why are they failing", "what's wrong", "which are worst"). You can't open with a drill / bad-end / explanation fetch — you don't yet know what to drill. A named service, dataset, session, or time-window does NOT pin the evaluator — the scope is still open, so rollup first. Two things do NOT let you skip it: (1) a guessed evaluator set — deciding yourself which evaluators are "failing" is not the user naming them; (2) an invented score cutoff — don't add a fixed WHERE score < 0.5 to define "failing" (no universal pass mark; polarity may be inverted), and don't reproduce one as SELECT buckets either (SUM(CASE WHEN score >= 0.5 …) AS pass_like is the same invented pass mark, just moved out of the WHERE). Ranking by score is fine — sort by each evaluator's bad direction and take the worst N; just don't hard-code an absolute pass/fail line.

A missing region / service.name / time-window is NOT a precondition — lead with the rollup, then offer to scope down. When the ask is to interpret or summarize an evaluation run and no region, service, or window was supplied ("how are my agents doing", "how did the run go"), run the rollup FIRST at the broadest reasonable scope — omit the service.name filter so it spans every scored agent, over a sensible default window (the 7-day window the patterns below use) in the Space already in context — then report it, drilling the WHY of any bad-end evaluator per the second pass below, and OFFER to narrow by region, service, or window. Region / service.name / window are inputs you gather to DRILL DEEPER, never a gate the first rollup waits behind; the placeholder filters in the query patterns below are scope-down options, not inputs you must collect before running. The user asked for a read, so lead with the read rather than stopping to ask for scope.

The number of rollup VIEWS is bounded by the intent the user EXPRESSED — but the WHY-drill is not optional once the rollup surfaces a bad-end evaluator. A status ask ("how is X doing", "tell me about X performance", "average X", "what's our X") is answered by the per-evaluator rollup plus a required WHY-drill into the worst 2–3 evaluators that sit at a bad end by polarity (pull their explanation field — see the second pass below): run the rollup, and if any evaluator is at a bad end, drill the worst 2–3 of those by polarity for the WHY, report that, then offer the deeper cuts. If NO evaluator is at a bad end (all healthy, or only one evaluator and it is healthy), report the rollup, name the worst-by-polarity evaluator as healthy, and skip the explanation fetch. Do NOT add, on your own initiative, a time-bucket / trend view, a per-agent view, a per-session view, or any other break-down dimension the user did not name — each extra dimension is another query round, and the user asked a fast question. A break-down runs only when the user named that slice ("per session", "by day", "which agent", "break it down by …") or asked to go deeper. When they DID ask to drill or investigate, fan out properly — the multi-slice breakdown is the right answer there, and this ceiling is not a reason to under-answer a real investigation.

Once the evaluator scope IS pinned, answer it directly. When the user named the evaluator(s), or the rollup has surfaced the bad ones, match the query to the ask: a status/number ask ("average Correctness") → the per-evaluator aggregate scoped to it, plus the required WHY-drill when the pinned evaluator(s) sit at a bad end by polarity — pull the bad-end evaluator's explanation (Q3), the worst 2–3 by polarity when several are in scope (ceiling 4), so even a plain status answer carries the WHY; when the pinned evaluator(s) are healthy, report the number(s), name the result as healthy, and skip the explanation fetch; a "why"/deeper ask ("why is Correctness failing", "which sessions") → the same bad-end explanation drill (Q3), read by polarity, widened as the ask warrants.

Evaluator names are DATA, not a fixed catalog — do NOT assume the Builtin. prefix. A user's common term ("toxicity", "bias") is not necessarily the stored gen_ai.evaluation.name: the real name may be ThirdParty.<Provider>.Toxicity, a custom evaluator's id, etc. To scope to an evaluator the user named, either (a) match LOOSELY (… attributes['gen_ai.evaluation.name'] ILIKE '%toxicity%') or (b) first read the DISTINCT evaluator names in the window (the rollup already lists every name that ran) and map the term to the real one. An empty result from an exact-name filter is INCONCLUSIVE — it may be a name mismatch, not absence. Only conclude an evaluator isn't running when it is absent from the DISTINCT-names list for the window; when unsure, say the exact name wasn't found and offer to match loosely.

Compose the rollup as SEPARATE queries — three shapes, ONE GRAIN PER QUERY. Never fuse two grains into a single query with a row_type discriminator and padded CAST(NULL AS …) columns to "get both shapes at once" — that SQL is slow to write, fragile, and Q1 already tells you which shape you need (avg_score non-NULL ⇒ numerical, stop; avg_score = NULL with scored > 0 ⇒ categorical ⇒ Q2). Read the shape off Q1 instead of guessing it inside one bundled query.

• Q1 — per-evaluator rollup (the headline): scores + errored SIDECAR in ONE query. Conditional aggregation splits scored vs errored INSIDE the aggregates (not in WHERE), grouped by evaluator name. Numerical evaluators get a real avg_score + spread; categorical come back avg_score = NULL, scored > 0 → read by label in Q2. failed_to_run is the sidecar for EVERY evaluator — report it as "(+N failed to run)" beside the scores: surfaced, NOT hidden, and NEVER conflated with a low SCORE.

• Q2 — categorical label distribution (a SEPARATE call): count grouped by evaluator + label, gated error.type IS NULL. Issue it for the NULL-avg (categorical) evaluators. Categorical = read by label, not "no data" — report those evaluators by name, never drop them or call them scoreless. A NUMERICAL evaluator gets NO label query at all. Once ANY query has come back with a real avg_score for an evaluator (this turn or an earlier one), that evaluator is numerical and the question is settled: do NOT issue Q2 for it, do NOT add label to a GROUP BY on it, and do NOT hedge by asking for both shapes in one query. Its label is only a gloss on the number (carry it if a row already has it); a label roll-up on it adds rows, not information.

• Q3 — the WHY of the low ones (required whenever the rollup surfaces a bad-end evaluator — see the second pass), and/or the error.type breakdown (only on an explicit execution-failure ask). The errored COUNT already shows in Q1's sidecar; Q3 is the detail, not the rollup.

First pass — per-evaluator rollup. For each evaluator on its own — numerical by avg + spread (min/max) + count, categorical by label distribution + count. Never a single blended number across evaluators (operating contract). This holds inside every break-down dimension too: per session (or per day, per online-eval config), roll up per (dimension × evaluator), never a single per-session number that averages distinct evaluators together. The per-evaluator table IS the headline; do NOT open with a "Headline KPIs" dashboard of invented top-line metrics.

Read each evaluator per its polarity — never a blanket "low = bad." Higher-is-better for most evaluators, but for inverted ones — Toxicity, Bias, Refusal, PII-leakage — a higher score is worse. Builtin.Harmfulness and Builtin.Stereotyping are not inverted despite their names: their rating scale is 0.0 = Harmful / Stereotyping, 1.0 = Not Harmful / Not Stereotyping, so the bad end is low (the Evaluation dashboard's Direction column shows "Higher is better" for both). For a categorical evaluator judge by the label, not a number. 0 can be the DESIRED outcome. Don't rank, or pick "worst," by low score alone.

Second pass — drill into the worst evaluator(s), BOUNDED and REQUIRED WHEN THE ROLLUP SURFACES A BAD-END EVALUATOR. Runs after the rollup on every eval-results report that has at least one evaluator at a bad end by polarity — including a plain status ask, not only "why" / "failing" / "what's wrong" asks. "Bad end" is a judgment from the rollup's own spread, not a fixed threshold: an evaluator is at a bad end when its scores cluster toward that evaluator's bad pole by polarity or show a visible bad-end tail — Q1's min/max spread reaching the bad pole (a low min for higher-is-better, a high max for an inverted one), or bad-pole labels in Q2 — never because they crossed an invented universal cutoff. Pull the explanation field for those worst, bad-end evaluators so the answer carries the WHY, then offer the deeper cuts. When every evaluator sits toward its good pole (no bad-end cluster or tail), none is at a bad end: report the rollup, name the worst-by-polarity evaluator as healthy, fetch no explanation, and OFFER the drill rather than forcing it. It is tightly bounded:

• Drill the worst 2–3 bad-end evaluators by polarity (a HARD ceiling of 4) — required whenever a bad-end evaluator exists, not a default to skip. Pick them from the rollup (worst by each evaluator's own polarity, among those at a bad end); do NOT drill middling or clean evaluators — if fewer than 2–3 sit at a bad end, drill only those that do, and if none do, drill nothing. Do NOT widen past the ceiling unless the user explicitly asks ("all", a named evaluator, "go deeper"). Drilling every non-max evaluator is the main cause of a turn running long.

• ONE query PER drilled evaluator, issued together as one batch — ranked, not thresholded. Each query is filtered to one evaluator and ranked top-N (~15) by that evaluator's own polarity, worst first (a categorical evaluator ranks by label / recency — a numeric predicate silently drops every categorical row). The queries are independent, so start all of them before reading any result rather than running them one round at a time. Do NOT fuse them into one query (QUALIFY … PARTITION BY evaluator, or a flat LIMIT): a fused result is read from the top, so later-sorted evaluators get starved or dropped — no WHY for them, and a second drill round. And do NOT filter by an invented cutoff (score < 0.5).

• One representative example per drilled evaluator, then STOP and offer more. Give the shared cause per drilled evaluator + ONE real bad-end example each — "Correctness (6 low): mostly missing-context, e.g. '…'" — not just "6 scored low." The fetched WHY MUST appear in the final answer — the cause is the headline, not an appendix.

• Never conclude "no bad results" for an evaluator you drilled from a capped/truncated slice — that's absence-vs-truncation, not proof. (This applies only to the ones you drill; it is NOT a mandate to drill the others to prove them clean.)

Show one representative example — the actual scored input/output. For a notable finding, fetch the trace's prompt → agent response for ONE representative poorly-scored result — a real bad-end score, NOT an errored (error.type) job (those have no score, no explanation, and often no valid trace to fetch) — and show it concretely ("user asked '…', the agent replied '…'"), not every row and not full transcripts.

Traces for low-scoring evaluations — match the depth to the ask. Each bad-end eval record already carries its scored trace's traceId, so the bad-end results give you the trace ids directly. Handing traces to a next step (add to a dataset, re-evaluate) needs only those traceIds — no trace fetch. Showing the actual trace (prompt / response / spans) means querying traces.default for those ids. Don't over-fetch.

"Failing" evals mean LOW SCORES, NOT execution failures. Disambiguate BEFORE composing the query. "Failing evals", "failing", "bad", "poor", "worst", "underperforming", "unhealthy", "what's wrong" — with NO execution-failure wording — mean evals performing badly: a low/bad SCORE judged by polarity, or a bad-end label (SCORED records, error.type IS NULL). That is the DEFAULT, and it does not change the ORDER — the FIRST query is still the rollup. This holds for the word "evaluation(s)" too: "which online evaluations are failing?" means low-scoring RESULTS scoped to online-produced records, error.type IS NULL — NOT the online-eval configs/jobs failing to run; the online / on-demand / batch word is ONLY the eval-TYPE scope. "Which tool/span are the failures concentrated in?" means: after the rollup, take the bad-END SCORED results, follow their traceIds into traces.default, and group the spans/tool-calls there. Treat it as execution failures (error.type IS NOT NULL) ONLY when the user EXPLICITLY says "failed to run", "failed to execute", "execution failure(s)", "errored", "did not run". An error.type record is NOT a "failing" score; there is no pass/fail and no "pass rate". When errored jobs fall in the window, say so ("N scored, Y failed to run (<error.type>)") rather than silently omitting them.

"Unhealthy / underperforming online evaluators" is this same RESULTS question. Answer it by filtering the scored results by attributes['aws.bedrock_agentcore.online_evaluation_config.name'] and rolling up per (config × evaluator) by polarity (error.type IS NULL) — NOT by reading list-online-evaluation-configs / get-online-evaluation-config. A config's status / executionStatus describe provisioning and on/off, not evaluation quality; an ACTIVE / ENABLED config exposes no health field. Use error.type only on the explicit "failed to run / errored" ask, and report it as its own short line after the scores, never folded into them. When the user also asks which configs exist, list them as a separate first part (list-online-evaluation-configs: name, status, executionStatus, evaluators, sampling) and still answer the health part from the results.

Query surface — dataset and schema

When the user asks to read eval scores or gen_ai.evaluation.* from traces.default, or asks which table holds them, tell them ALL of this: NO — evaluation results are LOG records in logs.default, not spans in traces.default; on traces.default those attributes are not span columns, so the query returns zero rows silently, with no error; that failure is silent — zero rows is indistinguishable from "no evaluations ran" unless the table choice is checked; what IS valid on traces.default is reading the scored trace's own spans by traceId; and the eval-result field names live with the logs.default surface (table below).

Each score is a log record in logs.default (query FROM "logs.default", or the unscoped "default") — NOT traces.default, where the agent's own request/response spans live. Reading gen_ai.evaluation.* off traces.default is not an error — those attributes are not span columns, so they come back NULL for every span row. The eval-read pattern filters on them (WHERE attributes['gen_ai.evaluation.name'] IS NOT NULL), so on traces.default that filter matches nothing and the query returns zero rows — indistinguishable from "no evaluations ran" unless you notice the wrong table. What is valid on traces.default is reading a scored trace's own spans by traceId (Section 7's trace-first join); the eval scores themselves live only in logs.default. Identify eval-result records by the evaluator-name attribute:

attributes['gen_ai.evaluation.name'] IS NOT NULL

Use these attribute paths verbatim — never shortened. The query engine reads the key literally — a wrong or shortened path (e.g. attributes['score.value'] for attributes['gen_ai.evaluation.score.value']) is not an error; it silently returns NULL for every row (an empty avg_score for all evaluators), which then wastes turns re-discovering the schema.

Validity and filter rules

• error.type IS NULL is the "job succeeded" gate. A record with gen_ai.evaluation.name present is not necessarily a score — the job may have errored, in which case score.value / label / explanation are all empty and score.value is non-numeric. Every score query filters AND attributes['error.type'] IS NULL (the Q1 rollup is the deliberate exception — it buckets errored rows with CASE instead). This also prevents CAST(<non-numeric> AS DOUBLE) failures ("failed to coerce" / "results fail to compute"), most visibly where the CAST is in a WHERE / ORDER BY comparison. evaluate_traces.py filters failed jobs on the write path so they never persist for on-demand, but online-evaluation records can still fail server-side and land in logs.default with error.type set — keep the gate on every score query rather than assuming the corpus is failure-free.

• A scored record has a score.value OR a score.label. Categorical evaluators have ONLY a label. The valid-result predicate is error.type IS NULL AND (score.value IS NOT NULL OR score.label IS NOT NULL).

• Do NOT gate on score.value IS NOT NULL alone. CAST(NULL AS DOUBLE) is safe — it yields NULL, which AVG skips and a < 0.5 comparison drops — so that filter adds no safety and silently drops every categorical evaluator. Let error.type IS NULL do the gating.

• On-demand writeback covers numeric scores only (Section 4). An on-demand categorical result lives only in the receipt, not in logs.default; online categorical scores are persisted server-side and do land. When a categorical-label-distribution query returns fewer rows than expected for a service, check whether the missing scores were on-demand categorical runs.

• Scope: the defaults below use IN ('<service>', '<service>.DEFAULT') and so include both online and on-demand records for exactly that service (never a prefix LIKE, which would blend sibling services such as <service>-api into the rollup). An on-demand-only or online-only report has to scope explicitly ("Other filters" below), or an on-demand quality regression can be masked by a larger online sample scoring well over the same window.

Polarity for filtering — pick the "bad" end

When a query filters to the bad results, pick the bad end per the evaluator's polarity — do NOT hardcode one comparison for every evaluator:

• Higher-is-better is the default — a low score is the bad end (< threshold, e.g. < 0.5).

• Inverted evaluators flip to the HIGH end — for Toxicity, Bias, Refusal, PII-leakage a higher score is worse, so the bad end is > threshold. Harmfulness and Stereotyping are NOT inverted: Builtin.Harmfulness scores 0.0 = Harmful, 1.0 = Not Harmful (same shape for Stereotyping), so their bad end is < threshold like the default; a > 0.5 drill on them returns only the safe rows and hides every harmful one.

• Categorical evaluators are judged by their LABEL, not a number — filter on the bad score.label value(s), never on score.value.

• This inverted list is a HINT, not exhaustive. The authority is the evaluator's own scale: get-evaluator ratingScale labels (which value is "Harmful", which is "Not Harmful") or the Evaluation dashboard's Direction column. For an unknown or custom evaluator, judge by its label or its get-evaluator rating scale; never assume higher-is-better, and never flag "failing" with score.value = 0 or score.label LIKE '%fail%' — evaluators sit on their own scales with no universal pass/fail.

A single-comparison bad-end query must be polarity-homogeneous — do not mix inverted and higher-is-better evaluators in one IN (...) list behind one < / >, or it surfaces desired-outcome rows on one side and hides the actual bad end on the other. To drill mixed polarities in one query, use the QUALIFY pattern below, which carries a per-evaluator comparison and ranking.

Query patterns

All over FROM "logs.default", filtering attributes['gen_ai.evaluation.name'] IS NOT NULL and a bounded `@timestamp` window (relative form NOW() - INTERVAL 'N HOUR' — never a bare integer; see query/sql-logs-traces.md). Score queries also add AND attributes['error.type'] IS NULL.

Recent scores for an agent:

SELECT `@timestamp`,
       attributes['gen_ai.evaluation.name']        AS evaluator,
       attributes['gen_ai.evaluation.score.value'] AS score,
       attributes['gen_ai.evaluation.score.label'] AS label,
       attributes['session.id']                    AS session_id
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NULL
  AND resource['attributes']['service.name'] IN ('<service>', '<service>.DEFAULT')
ORDER BY `@timestamp` DESC
LIMIT 100

Q1 — per-evaluator rollup: avg + spread + errored SIDECAR in ONE query. Do NOT filter error.type in WHERE; split it INSIDE the aggregates and always GROUP BY the evaluator name. The CAST sits INSIDE the scored CASE, so it never runs on an errored row (no cast failure); AVG skips a categorical evaluator's NULL value → avg_score = NULL for it (read those by label with Q2) instead of dropping the evaluator. MIN/MAX give the spread without a dialect-specific STDDEV:

SELECT attributes['gen_ai.evaluation.name'] AS evaluator,
       AVG(CASE WHEN attributes['error.type'] IS NULL
                THEN CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) END) AS avg_score,
       MIN(CASE WHEN attributes['error.type'] IS NULL
                THEN CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) END) AS min_score,
       MAX(CASE WHEN attributes['error.type'] IS NULL
                THEN CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) END) AS max_score,
       SUM(CASE WHEN attributes['error.type'] IS NULL     THEN 1 ELSE 0 END) AS scored,
       SUM(CASE WHEN attributes['error.type'] IS NOT NULL THEN 1 ELSE 0 END) AS failed_to_run
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND resource['attributes']['service.name'] IN ('<service>', '<service>.DEFAULT')
GROUP BY evaluator

failed_to_run is the per-evaluator sidecar (it covers categorical evaluators too, since their errored rows have no label). For a strictly scored-only view with no sidecar, add AND attributes['error.type'] IS NULL to the WHERE and drop the CASE wrappers.

Average score per session (or any break-down dimension) AND evaluator. GROUP BY the dimension AND the evaluator name, and SELECT the evaluator name so the breakdown is legible. Never GROUP BY session_id alone — that silently averages a session's Helpfulness, Correctness, Toxicity, … into one meaningless number. Swap session.id for any other dimension the same way:

SELECT attributes['session.id']                 AS session_id,
       attributes['gen_ai.evaluation.name']     AS evaluator,
       AVG(CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE)) AS avg_score,
       COUNT(*)                                 AS n
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NULL
  AND resource['attributes']['service.name'] IN ('<service>', '<service>.DEFAULT')
GROUP BY session_id, evaluator
ORDER BY session_id, evaluator

Categorical evaluators come back with avg_score = NULL here — roll those up with Q2, adding session_id to its GROUP BY the same way. For a NUMERICAL evaluator never add label to the GROUP BY — that's the categorical shape only, and it multiplies rows past the display cap.

Q2 — categorical label distribution per evaluator (for the evaluators Q1 returned with avg_score = NULL). One query returns every categorical evaluator's label breakdown:

SELECT attributes['gen_ai.evaluation.name']        AS evaluator,
       attributes['gen_ai.evaluation.score.label'] AS label,
       COUNT(*)                                    AS n
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NULL
  AND attributes['gen_ai.evaluation.score.value'] IS NULL   -- categorical = label-only, no numeric value
  AND resource['attributes']['service.name'] IN ('<service>', '<service>.DEFAULT')
GROUP BY evaluator, label
ORDER BY evaluator, n DESC

GROUP BY evaluator, label keeps each evaluator's own label set separate. Never GROUP BY label alone — that mixes distinct evaluators' categories. And never run this pattern for an evaluator already known to be NUMERICAL: if a previous query returned a real avg_score for it, it has scores, so this query is the wrong shape for it (its label is a gloss on the number). The score.value IS NULL predicate would return zero rows for it anyway — don't spend a query round proving that.

Q3 — results at the bad end of the scale + why (pull explanation for the bad results). The comparison is per-evaluator polarity: the example assumes higher-is-better (< 0.5, ASC); for an inverted evaluator flip both the comparison to > 0.5 and the ORDER BY to DESC so LIMIT returns the actual worst, not the least-bad breachers near the threshold; for a categorical one filter score.label instead. Scope with IN (...) to evaluators of one polarity:

SELECT attributes['session.id']                    AS session_id,
       attributes['gen_ai.evaluation.name']        AS evaluator,
       attributes['gen_ai.evaluation.score.value'] AS score,
       attributes['gen_ai.evaluation.explanation'] AS why,
       traceId                                     AS trace_id
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NULL
  AND attributes['gen_ai.evaluation.name'] IN ('<eval1>', '<eval2>', '<eval3>')   -- same-polarity evaluators only
  AND resource['attributes']['service.name'] IN ('<service>', '<service>.DEFAULT')
  AND CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) < 0.5   -- higher-is-better; flip to > 0.5 for inverted, or filter score.label for categorical
ORDER BY CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) ASC
LIMIT 50

Multi-evaluator drill — top-N PER evaluator. The default drill is one Q3 per drilled evaluator, issued together as a batch (Retrieval plan, second pass): each query filters to a single evaluator and ranks by its own polarity, so nothing starves. When several evaluators must share ONE query anyway, a single flat LIMIT/ORDER BY STARVES later-sorted ones — all rows land on the first evaluator, and "no bad results for evaluator X" becomes indistinguishable from "truncated by the LIMIT" — so cap PER evaluator with QUALIFY ROW_NUMBER() OVER (PARTITION BY evaluator …), and remember a fused result is still read from the top, so later partitions can be cut by the row cap. Two rules: (1) inside QUALIFY reference the SELECT aliases (evaluator, score), NOT the raw attributes['…'] — post-projection the raw expression is out of scope and errors No field named attributes. (2) size N so N × (num evaluators) ≤ ~50 so the default row cap never truncates the union (e.g. ~15 each for 3 evaluators).

Numerical evaluators — rank "worst" polarity-aware (a plain ORDER BY score ranks one polarity backwards):

SELECT attributes['session.id']                    AS session_id,
       attributes['gen_ai.evaluation.name']        AS evaluator,
       attributes['gen_ai.evaluation.score.value'] AS score,
       attributes['gen_ai.evaluation.explanation'] AS why,
       traceId                                     AS trace_id
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NULL
  AND ( (attributes['gen_ai.evaluation.name'] IN ('<inverted-eval>')
           AND CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) > 0.5)
     OR (attributes['gen_ai.evaluation.name'] IN ('<higher-is-better-eval>')
           AND CAST(attributes['gen_ai.evaluation.score.value'] AS DOUBLE) < 0.5) )
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY evaluator
  ORDER BY CASE WHEN evaluator IN ('<inverted-eval>')
                THEN  CAST(score AS DOUBLE)          -- inverted: higher = worse
                ELSE -CAST(score AS DOUBLE) END DESC -- higher-is-better: lower = worse
) <= 15

Categorical evaluators — no numeric value, so filter/rank by LABEL, never a threshold:

SELECT attributes['session.id']                    AS session_id,
       attributes['gen_ai.evaluation.name']        AS evaluator,
       attributes['gen_ai.evaluation.score.label'] AS label,
       attributes['gen_ai.evaluation.explanation'] AS why,
       `@timestamp`                                AS ts
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NULL
  AND attributes['gen_ai.evaluation.score.value'] IS NULL      -- categorical = label-only
  AND attributes['gen_ai.evaluation.score.label'] IS NOT NULL
  -- optional, when a bad label is known: AND attributes['gen_ai.evaluation.score.label'] IN ('<bad-label>', …)
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY evaluator
  ORDER BY ts DESC          -- no numeric severity to rank on; sample by recency (or per chosen label)
) <= 15

Do NOT combine numeric and categorical bad-ends in ONE query — a numeric score > 0.5 predicate silently drops every categorical row (its value is NULL). Run them as separate queries.

Evaluations that failed to run (run this ONLY when the request is about evals that errored / did not run — error.type IS NOT NULL, the inverse of the score gate):

SELECT attributes['gen_ai.evaluation.name']  AS evaluator,
       attributes['error.type']              AS error_type,
       attributes['error.message']           AS error_message,
       COUNT(*)                              AS n
FROM "logs.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND attributes['gen_ai.evaluation.name'] IS NOT NULL
  AND attributes['error.type'] IS NOT NULL
  AND resource['attributes']['service.name'] IN ('<service>', '<service>.DEFAULT')
GROUP BY evaluator, error_type, error_message
ORDER BY n DESC

Traces behind low-scoring evals (two steps). Each bad-end eval record already carries the scored trace's id as top-level traceId — the bad-end queries above already select it, so the traceIds ARE the trace list; a next step that only needs the ids (add to a dataset, re-evaluate) needs no further query. To show the actual traces (prompt / response / spans), take those traceIds and query traces.default:

SELECT traceId, name, `@timestamp`
FROM "traces.default"
WHERE `@timestamp` BETWEEN NOW() - INTERVAL '7 DAY' AND NOW()
  AND traceId IN ('<traceId1>', '<traceId2>', …)   -- the ids from the bad-end query
ORDER BY `@timestamp` DESC

Other filters: one session → AND attributes['session.id'] = '<sid>'. One trace → AND traceId = '<traceId>' (the eval-result record references the scored trace by traceId; the score is a separate record, not an attribute on the trace's own spans). Online only → attributes['aws.bedrock_agentcore.online_evaluation_config.name'] IS NOT NULL (or = '<config_name>' for one config); on-demand only → the same field IS NULL.

From a trace (trace-first). A trace's eval scores are not attributes on the trace's own spans — they are separate logs.default eval-result records that reference the scored trace by top-level traceId. So "the eval result of this trace" is a logs.default query filtered by that id (add AND attributes['session.id'] = '<sid>' for the session variant). To show a trace's spans and its eval score together, JOIN "logs.default" ⋈ "traces.default" on traceId — JOIN mechanics are in query/sql-logs-traces.md.

Notes

• Results are Region-scoped and appear after the normal ingestion lag (a score shows up a couple of minutes after the evaluation runs). If a just-run evaluation is not visible, widen the window or wait for ingestion.

• On-demand scores are written back best-effort (Section 4) — a score that failed to write (e.g. missing logs:PutLogEvents on the caller) will not appear; online evaluation is the source with guaranteed persistence.

8. From scores to changes

Workflows that start from what evaluation found and end in a change to the agent. They reuse the sections above: scores come from Section 7, scoring from Sections 4 and 5, datasets from Section 6, and the per-trace and per-session span reads from query/trace-exploration.md. Every write along the way (a config, an evaluator, a dataset, a version, a code change) is confirmed first, per the operating contract.

What these workflows expose. They read and repeat the agent's real conversations: prompts, responses, retrieved context, tool arguments, and evaluator explanations that quote them. In a report, a dataset, an alert notification or a CR description, show only the excerpt the finding needs and mask anything that looks like a secret or personal data (payload rules in query/trace-exploration.md section 2). Evaluation results, from online configs as well as on-demand runs, are stored in CloudWatch log groups readable by anyone with access to them; recommend a customer-managed KMS key and access limited to the people allowed to see that data, as for the on-demand results group (Section 4, "Writeback creates a log group").

Set up quality monitoring for an agent

1. Profile the traffic first. Over a bounded window of the agent's spans, look at the traces with the highest and lowest token totals and latency, and at what the prompts ask and which tools get called. This is what the evaluators have to judge.

2. Propose an online evaluation (Section 5) whose evaluators fit that traffic, for example a tool-selection evaluator when tool calls dominate and a faithfulness one when answers rest on retrieved content. Then say whether the built-ins cover what the traffic needs judged; for anything they do not (a domain policy, a tone, a format), propose a custom evaluator (Section 5). The user picks the final set.

3. Propose an alert on the scores so a sustained drop notifies someone: a SQL alert on logs.default whose query is the per-evaluator average of scored results (Section 7) for this agent over a window long enough to ride out single bad sessions, with the threshold on each evaluator's bad side by polarity. Authoring rules are in alerts.md.

Root-cause a quality regression

1. Confirm scored results exist for the agent in the window (error.type IS NULL, both service.name forms) and trend each evaluator over time (by hour or day) to find which one degraded and when.

2. Compare traces scored well and badly by that evaluator, before and after the drop, for what differs: the model id or its parameters, the system prompt, the tool set or tool outputs, retrieval results ("What changed" in query/trace-exploration.md).

3. If the spans carry a version or deployment attribute, split the scores by it; that is the most direct way to name the change that introduced the drop. Without one, ask for the change times and compare before and after each.

Drill one evaluator into a fix

For a dimension that is dropping (tool selection, faithfulness, or any other), read that evaluator's bad-end results by polarity with their explanations (Section 7), follow their traceIds into traces.default, and compare what the explanation complains about with what the spans show:

• Tool selection: the tool chosen against the tools available on that call. Typical causes are tools whose descriptions overlap, several near-identical tools, or a system prompt that never says when to use which.

• Faithfulness: the claims in the answer against the retrieved context the trace shows. Typical causes are answers that go beyond the retrieved text, or a retrieval tool that returns content without the source fields the answer should cite.

Recommend specific edits (tool descriptions, system-prompt instructions, the tool's output shape) and offer to verify them by scoring a dataset before and after. The verdict comes from the evaluator, never from you re-reading the traces.

Is the agent tuned to the evaluators rather than the task?

The signal is high scores on the evaluated dimensions while independent signals stay poor. Compare the scores with evidence the rubric does not reward: a different evaluator on the same traces, user-visible errors or abandoned sessions, a held-out dataset the agent was not tuned on, and answers that echo the rubric's own wording. Report it as a hypothesis with the evidence for and against it, not as a verdict.

Golden datasets from production

Start from the bad-end results (Section 7) for the window and evaluator in question, take their traceIds, and capture them with the helper (Section 6), which records each trace's input and its tool trajectory. The captured expected response is what the agent actually answered, so for traces chosen because they scored badly it is the wrong answer: say so, and leave the expected responses for the user to write (or drop them before any correctness scoring) rather than proposing ones yourself.

Change, re-run, compare

Any plan or report for a change, a prompt comparison, or an A/B states all of this: the baseline and the new run use the same published dataset version; each evaluator's result is given before and after, separately, never combined; the review (CR) is opened only after the second run exists; and if the change does not beat the baseline, that is said plainly, with no claimed gain that was not measured. A run of a new prompt version is reported against the previous version's run, and if nothing here can execute the prompt, say so and point to the Agent Playground. Splitting stored scores by prompt version needs a version attribute on the spans; without one, say the split cannot be made and offer to record one going forward or to score both prompts on one published version.

To show a change improves the agent, measure it against a fixed target:

1. Baseline: evaluate a published dataset version (Section 6) and record each evaluator's result.

2. Make the change (a prompt or tool-description edit grounded in the evidence above), then re-run the same evaluators against the same version.

3. Report each evaluator's before and after separately, and say plainly when the change does not beat the baseline. Never claim an improvement without the re-run.

When the request chains the whole loop (worst traces to dataset to baseline to change to re-score) and the user has not said what "worst" means, take the bad end of the per-evaluator rollup of stored scores (Section 7), read by polarity; if the agent has no stored scores yet, say so and offer to score first. A definition the user gives (slowest, failed) wins. The capture helper takes at most 25 traces per run, so a larger set needs several runs. Confirm each write as it comes, report each id the service returns (datasetId, the version), and stop at the first step that fails, reporting what completed, rather than continuing on partial data. For a code change, open the review only after the re-run, with the before and after in it.

Comparing prompts. The same rules apply, plus:

• Stored scores split by prompt version only if the traces record one. Check with schema discovery (EXPLAIN (ANALYZE_FIELDS) on the agent's spans in traces.default, query/sql-logs-traces.md) for a prompt-version attribute; never assume a name. The score records in logs.default do not carry it, so grouping them by it returns NULL for every row. Map each version to its traceIds from the spans, then group the score records by those traceIds (the trace-first pattern in Section 7), and give each evaluator's scores per version beside the prompt diff. Without such an attribute, say the scores cannot be split that way, and offer to record a version attribute going forward or to score both prompts against one published dataset version.

• Running a prompt over a dataset needs something that executes it. Confirm the published version and the evaluators, and resolve the model id from the Region's inference profiles (as for a judge model above) rather than writing one from memory. If nothing in reach can run the agent or model with the new prompt, say so and point to the Agent Playground; never write the outputs yourself. Report each evaluator against the previous prompt's run on the same dataset version, not on its own.

• Picking a winner. Run both prompts on the same version with the same model and one evaluator for the dimension asked. If no built-in fits, draft a custom TRACE judge and show it for confirmation; a draft is not a create, so offer it even when the user said not to change anything. Report each prompt's average, spread and count with a few contrasting examples, and name a winner only when the gap is clear against the spread.

After analysis — offer follow-up actions

**What would you like to do next?**
1. **Score these traces now** — run an on-demand evaluation with a chosen evaluator
2. **Monitor continuously** — set up an online evaluation on live traffic
3. **Save as a dataset** — capture these traces as a regression / golden set for reuse
4. **Read back stored scores** — roll up what online / on-demand evaluation has already found

Wait for the user to pick an action before proceeding.