Eval Engineering

LangChainOfficialPublished by LangChain

Skill
Workflow
Agent tooling
MIT
Low risk
Has scripts

From the author

Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.

— the description by LangChain

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Eval Engineering provides a structured, iterative methodology for developing and maintaining agent benchmarks. It guides the user through inspecting existing agent repositories and traces, designing grounded Task Specifications, building isolated execution environments, and implementing independent verifiers. The process emphasizes the creation of project-specific World Knowledge Skills to capture reusable patterns, ensuring that evaluation suites remain consistent, fair, and aligned with real-world agent behavior.

What it does

Inspects agent repositories, traces, and existing evaluation suites to identify testing gaps.
Drafts and reviews Task Specifications that define agent inputs, environment conditions, and scoring criteria.
Builds and audits isolated Harbor-based environments for reproducible agent testing.
Develops project-specific World Knowledge Skills to store reusable facts, scripts, and patterns.
Implements independent verifiers to score agent performance and detect failures.
Calibrates evaluation tasks by analyzing run trajectories and distinguishing between agent capability and infrastructure defects.

When to use it

When designing new benchmarks for an agent or workflow.
When existing evaluation tasks are brittle, leaking information, or failing due to environment issues.
When you need to capture and reuse project-specific knowledge across multiple evaluation tasks.
When you need to perform root-cause analysis on agent failures within a controlled environment.

Advantages

Ensures evaluation tasks are grounded in actual repository evidence and real-world usage.
Promotes consistency through the use of reusable World Knowledge Skills.
Reduces false negatives by distinguishing between agent capability failures and infrastructure or verifier defects.
Provides a clear, audit-ready structure for managing complex benchmark suites.

Usage examples

“I need to create a new evaluation task for our customer support agent that tests its ability to handle multi-turn ticket escalations. How should I start?”: The agent will guide you to first inspect existing traces of ticket escalations, review the current agent harness, and then draft a Task Spec that defines the escalation scenario, environment requirements, and success criteria.
“Our current evaluation suite is failing frequently due to environment setup issues. How can I fix this?”: The agent will help you audit the current Harbor configuration, verify environment isolation, and implement a more robust reset or replacement strategy for the test environment.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/langchain-ai/eval-engineering/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip eval-engineering.zip -d .agents/skills

Documentation by LangChain

Eval Engineering

Flow

Inspect all inputs first: the repository, Harness, optional traces, existing Tasks and runs, existing World knowledge, and the human goal. Identify the source files and skill references that apply before proposing work.
Create or update the small project World Knowledge Skill from reusable facts in those inputs. Use it to propose one grounded Task.
Draft the Task Spec and World Skill together. Show both to the user, keep exact Task truth only in Task.md, and refine both until the user approves them.
Implement the approved Task, validate its Environment and Verifier, run the real Harness, inspect the full evidence, and fix only non-agent failures.
Reconcile the World Skill with what the run proved, then repeat this flow for the next Task.

Terms

Task Spec: Task.md, which describes the input, relevant agent conditions, Environment, scoring, fairness, and open decisions for one Task.
Task: the runnable instruction, Environment, and Verifier.
Harness: the complete agent Harbor runs, including prompts, model loop, tools, hooks, memory, sessions, and adapter.
Environment: the files, data, services, identity, permissions, network, clock, and mutable state around the Harness.
Verifier: independent checks that score the result or mark a run invalid.
World Knowledge Skill: a repository-local skill with reusable project-specific knowledge, references, scripts, assets, and tests that help generate Task Specs and build future Tasks.
Spec2Task: the full loop that turns a reviewed Task Spec into an audited runnable Task. Follow Task implementation for its concrete build order and reference routing.

Reference routing

Read each reference when its decision appears:

Need	Read
Inspect source, traces, the Harness, dependencies, access, and existing evals	Discovery
Bootstrap or update reusable project knowledge	World knowledge
Propose Tasks and write the single Task Spec	Task design
Build data, services, access, state, and reset	Environment building
Create structured or natural-language data	Synthetic data
Define independent evidence and scoring	Verifier design
Apply Spec2Task to turn a reviewed Spec into an audited Task	Task implementation
Compare model runs and classify failures	Calibration
Package and run Harbor tasks	Harbor
Adapt a known benchmark design	Benchmark patterns
Build multi-turn conversations	Multi-turn simulation
See World knowledge learned across two Tasks	Service-desk example

Reusable implementation resources:

Multi-turn runner, model user, and Harbor adapter example
Tool-schema comparison, which compares supplied schema fragments but does not resolve external $ref targets
Read-only SQLite state snapshot

1. Inspect inputs and existing World knowledge

Review every input the user provides before proposing a Task. Use the guidance that matches each available input:

For a repository, Harness, traces, or dependencies, read Discovery.
For existing Tasks and runs, inspect their instructions, Environments, Verifiers, rewards, trajectories, and final state. Read Calibration when run quality or failure causes affect the new design.
For an existing project World Skill, read World knowledge, then check the sources and reusable methods that affect the new Task.
For human goals and constraints, read Task design.
For relevant benchmark examples, use the domain index and source callouts in Benchmark patterns.

Inspect the repository before asking questions that source and tests can answer. Follow the active Harness through prompts, models, tools, services, state, effects, and focused tests. Inspect existing Task instructions, parsers, Verifiers, reward paths, and run evidence.

If the user supplies traces, review complete runs or threads. Use traces to learn real requests, dependency behavior, state shapes, errors, and failure conditions. Do not treat a trace answer as independent truth.

If .agents/skills/<project>-world/SKILL.md exists, read it. Follow its routing only for knowledge relevant to the current Task. Check cited repository paths, commands, and scripts when their accuracy affects the design.

2. Propose and select a Task

Read Task design and use the index in Benchmark patterns to find the relevant domain and source callouts. Focus on that domain unless the Task crosses another one. In the first user-facing design response after inspection, propose one Task grounded in repository evidence, supplied traces, existing coverage, or a human priority. State:

the real work and capability;
the condition that makes the case non-trivial;
the Environment and independent evidence it needs;
the important failure it can detect;
how it differs from existing Tasks; and
the main open decision.

In the same response, show the relevant current World Skill content and the specific additions or corrections this Task suggests. If no World Skill exists, show the small initial contents that will help create this Task and future Tasks. Keep the Task's exact request, focal records, expected result, hidden truth, and exact scoring rules out of the World Skill.

Let the user revise the Task proposal and World knowledge together before implementation. Offer alternatives only when a real user choice changes the design.

3. Write and review the Task Spec and World Skill

Copy the Task template to evals/<suite>/tasks/<task-id>/Task.md. Put all Task-specific design in this one file. At the same time, create or update the project World Skill by following World knowledge. Determine the project skill location supported by the active agent and repository. .agents/skills/<project>-world/SKILL.md and .claude/skills/<project>-world/SKILL.md are common landing spots. Follow an established project convention when one exists. Otherwise, explain the proposed location and get user confirmation before creating the skill. Start from the World Skill template when needed.

Keep each Task.md beside the Harbor task it describes:

evals/<suite>/tasks/<task-id>/
├── Task.md              # human-reviewed control-plane spec
├── task.toml             # required Harbor configuration
├── instruction.md        # required agent input
├── environment/          # required Environment definition and visible state
│   ├── Dockerfile        # use this or docker-compose.yaml
│   └── docker-compose.yaml # optional; primary service must be main
├── tests/
│   ├── test.sh           # required Harbor Verifier entry point
│   ├── test_*.py         # optional Verifier helpers
│   └── fixtures/         # optional hidden Verifier data
└── solution/
    └── solve.sh          # optional reference path

Never copy or mount Task.md into the evaluated agent's workspace or image. The agent receives instruction.md and only the Environment state intended for the run.

Include:

purpose and source evidence;
exact input and later turns;
only the Harness conditions relevant to this Task;
initial state, services, access, visibility, reset, and production differences;
required results, prohibited effects, accepted alternatives, and independent Verifier evidence;
fairness, leakage risks, and invalid-run conditions; and
open decisions and assumptions.

Show the full Task Spec and the World Skill changes to the user. Explain what is already in the World Skill, what this Task adds or corrects, and what stays only in Task.md. Revise both through the same back-and-forth. Mark the Task Spec approved only after explicit approval. Treat World Skill changes as accepted only after the user reviews them. If the user requests an end-to-end build without an approval pause, continue with an agent-reviewed Status: Draft and label the World Skill changes as unreviewed.

If implementation changes the request, visible information, material Environment behavior, or scoring boundary, update Task.md and show the change. Set its status back to Draft. Show the diff and require explicit reapproval before setting it to Approved again.

4. Apply Spec2Task

Follow Task implementation. It gives the build order and routes each decision to the Environment, synthetic-data, Verifier, Harbor, and calibration references.

For an existing project, use its pinned or supported Harbor version. Otherwise, use the installed supported version and record it. Upgrade only with user approval and a stated compatibility reason. Use the installed CLI help as the command contract.

Before a scored model run:

Confirm the model, trial count, judge, timeout, and maximum expected cost with the user unless the user already authorized that run plan.
Complete the package audit in Harbor. Confirm every required file, entry point, path, permission, configuration value, mount, service, and reward output needed for this exact Task is present and works through Harbor. Confirm hidden Task, Verifier, solution, and secret material is absent from the agent-visible image and workspace.
Check setup and trial isolation in the way that fits the Environment. A fresh container or worktree can provide isolation by replacement. A reused mutable service needs a checked reset. Immutable frozen data needs only a checked load. See Task implementation.
Exercise every operation the Task depends on.
Run the reference path when one exists.
Test the Verifier with a clear valid result, a valid alternative, a realistic wrong result, a shortcut, a prohibited collateral change, and missing or corrupt evidence.
Confirm every completed Verifier path writes a valid reward and useful evidence without exposing hidden truth or secrets.

5. Run and audit

Run the actual Harness through Harbor. Read the complete trajectory, not only the reward. Inspect:

messages, model calls, tool calls, results, retries, and errors;
initial and final Environment state and external effects;
service, setup, readiness, reset, and cleanup evidence;
each Verifier criterion, its evidence, decision, and error; and
the resolved Harness, model, Environment, and judge configuration.

Classify each unsuccessful run as an agent capability failure, missing information, Harness defect, Environment defect, Verifier false rejection, Verifier false acceptance, leakage, or infrastructure failure. Fix non-agent failures before using the score.

Model comparison is an optional calibration strategy, not a completion rule. When it would answer a real uncertainty, compare a weaker model, the target model, or a stronger model and repeat trials when behavior is variable. Read every selected trace. Contrast can expose unclear inputs, brittle setup, leakage, shortcuts, or reward hacks. Pass rates and model ordering do not prove Task quality.

Read Calibration for the complete audit method.

6. Reconcile project World knowledge

Use World knowledge throughout Task design, implementation, and audit. Add or correct project-specific knowledge when the work supplies evidence that would help another Task. This can include Task patterns, Environment methods, data creation, Verifier evidence, run procedures, scripts, assets, and examples.

After the audit, reconcile the World Skill with what the completed Task proved. Show the user:

the proposed reusable knowledge;
the evidence supporting it;
how another Task would use it;
where it should live; and
what remains specific to the completed Task.

Remove or narrow ideas that the Task disproved. If the user asked for autonomous end-to-end updates without a pause, make the smallest supported update, show it in the final review, and do not imply that the human approved the generalization.

Create only SKILL.md at first. Add references/, scripts/, assets/, or tests/ only when their real contents justify them.

Keep the completed Task's request, focal state, expected result, and exact criteria in its collocated Task.md. Do not copy broad guidance that is already clear in this skill. Record the project-specific adaptation of that guidance.

7. Repeat

Use Tasks two and three to test the World Skill. Check whether it reduces rediscovery, improves Task Specs, preserves important relationships, reuses a proven operation, or prevents a known Verifier defect. Correct rules that are missing, stale, or too broad.

When several materially different Tasks have exercised the shared knowledge and the construction and verification methods are clear, the next cycle can propose several independent Task Specs:

Mine new repository, trace, and human evidence.
Use the World Skill to generate distinct Task Specs.
Have the human review the specs.
Build independent approved Tasks in parallel.
Audit every Task individually.
Update World knowledge only with reusable corrections.

Continue this loop as production behavior, user priorities, agents, and models change.

Dependencies, access, and safety

Map required systems, data, roles, network needs, and safe setup methods. Never read, print, copy, store, or ask the human to paste secret values. Tell the human what dependency is needed, why it is needed, and how the project expects access to be provided. Default to controlled local, frozen, or simulated dependencies. Never write to production during an eval. Treat access, startup, reset, timeout, judge, and Verifier failures as invalid runs, not failed agent work.

Complete only when

Task.md matches the built instruction, Environment, and Verifier.
The Task is solvable from agent-visible or normally discoverable information.
The Environment starts reliably and isolates trials by replacement, reset, or immutable state as appropriate, without leaking hidden truth.
Valid and invalid Verifier cases behave as intended.
At least one real Harness run was read in full.
Non-agent failures were repaired or reported as unresolved limits.
Model comparison, when used, includes trace review rather than pass rates alone.
The project World Skill was created or updated with the Task Spec, reviewed during the work, and reconciled with the final evidence.
The user receives the Task path, run command, results, evidence, and remaining limits.

Files

ATTRIBUTION.mdAdded by AutoGineer894 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md15.6 KiB
agents/openai.yaml378 B
assets/task/Task.md.template1.8 KiB
assets/world-skill/SKILL.md.template2.9 KiB
references/calibration.md3.1 KiB
references/discovery.md12.1 KiB
references/environment-building.md4.0 KiB
references/examples/service-desk.md2.9 KiB
references/harbor.md6.8 KiB
references/multi-turn-simulation/guide.md5.1 KiB
references/multi-turn-simulation/harbor_example.pyScript10.0 KiB
references/multi-turn-simulation/model_user.pyScript5.0 KiB
references/multi-turn-simulation/runner.pyScript7.0 KiB
references/patterns.md36.8 KiB
references/synthetic-data.md2.9 KiB
references/task-design.md5.5 KiB
references/task-implementation.md8.2 KiB
references/verifier-design.md3.0 KiB
references/world-knowledge.md11.3 KiB
scripts/compare_tool_schemas.pyScript13.8 KiB
scripts/snapshot_sqlite_state.pyScript5.2 KiB

references/patterns.md

Benchmark Patterns

Use these patterns as evidence-backed starting points. Read the relevant domain section before designing a Task. Open the cited benchmark when its design can answer a concrete question about Task shape, Environment state, tool surfaces, simulation, or verification. Adapt the method to the current project. Do not copy a benchmark's private data, hidden answer, exact scenario, or limitations without review.

Benchmark design index
Automation Environments
Browser and Desktop Environments
Coding Environments and Tasks
CRM and Knowledge-Work Environments
Multi-Turn and Policy Environments
Terminal Environments and Tasks
Cross-domain quality checks

Benchmark design index

Use this table to decide which external design to inspect in more detail.

Benchmark	Domain	Design worth studying
AutomationBench	Business automation	Trigger data, pre-populated SaaS state, realistic tool surfaces, assertion-level scoring, and a separate strict completion result
AppWorld	Cross-application APIs	Stateful simulated applications, database-backed truth, cross-app workflows, and final-state verification
ToolSandbox	Stateful tool use	Controlled hidden state, milestones, tool-call consequences, and reproducible interaction state
TheAgentCompany	Workplace agents	Longer work across applications, documents, messages, and durable artifacts
WebArena	Browser agents	Reproducible websites, browser interaction, seeded application state, and state-based evaluation
WorkArena	Enterprise browser work	Role-aware enterprise tasks, realistic web workflows, and application state
WorkArena++	Multi-application enterprise work	Longer workflows across enterprise applications and role-sensitive state
OSWorld	Desktop agents	OS and application snapshots, computer input, cross-application work, and state restoration
SWE-bench	Coding agents	Real issues, pinned repository revisions, behavior tests, and patch verification
SWE-smith	Scalable coding Tasks	Reproducible mutation and execution pipelines; mutations still need usefulness review
CRMArena-Pro	CRM agents	CRM roles, linked business records, permissions, and realistic sales workflows
tau-bench	Policy and tool use	User simulation, policy-guided decisions, tool use, and database final state
tau-squared Bench	Longer multi-turn work	Both sides of a tool-mediated dialogue, longer interactions, and final-state checks
Terminal-Bench	Terminal agents	Isolated command-line Environments, natural instructions, tests, and reference solutions

A benchmark citation is not an authority claim. Check its current repository, paper, release, license, and task format before copying an implementation pattern. Record which design choice was adapted and why it fits this project.

Automation Environments

Use this pattern

Use for scheduled jobs, event handlers, background agents, retries, alerts, integrations, and work where time or repeated delivery changes correctness.

State and storage

Store current state and append-only events separately. Include:

Trigger ID and type.
Scheduled, occurred, observed, and completed times.
Actor or source.
Idempotency key, which prevents repeat delivery from causing a repeat effect.
Attempt count and error class.
Current workflow state.
Side effects and their external IDs.
Cause or parent event.

Use a fixed clock and deterministic event queue. Use an explicit state machine, which lists allowed states and changes. Store enough history to prove retry, deduplication, and recovery behavior.

For each case, define trigger data, pre-populated starting state, the allowed tool surface, and final-state assertions. This is the useful AutomationBench case shape. The trigger starts the workflow; it does not replace the World or the Task success contract.

Services, tools, and access

Simulate mutable queues, schedulers, webhooks, email, and third-party writes unless an approved sandbox is safe and resettable. Preserve payload schema, delivery order, duplicate delivery, error classes, retry limits, and side-effect IDs.

Catalog scheduler, queue, service, cloud, and model credentials by name and scope. Keep the fixed clock and hidden future events unavailable to the agent.

Setup and reset

Load base state and pending events.
Set the clock and visibility boundary.
Start the worker and dependent services.
Check queue and service readiness.
Save initial state, events, and external-effect ledgers.
Reset from the source snapshot, including retry counters and deduplication state.

Test reset after partial effects and timeouts.

Strong Task families

Process a scheduled action exactly once.
Recover from a transient error without duplicate effects.
Stop after a permanent policy or permission error.
Reconcile a delayed observation with newer state.
Handle concurrent triggers in a valid order.

Worked Task A: retry a transient delivery

Instruction:

Process the pending notification. The first delivery attempt has a transient error. Complete the delivery without creating a duplicate message.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
final_state	Workflow is complete	Final workflow row	Compare state with `completed`	Exact match
retry	One allowed retry occurred	Attempt and error rows	Match one transient failure then one success	Exact sequence
one_effect	One message exists	External-effect ledger	Count unique message IDs for the source event	One
repeat_safety	Both attempts share one repeat-protection key	Attempt records	Compare keys and source event ID	All equal
scope	Other work stays fixed	Before and after snapshots	Compare unrelated workflows and schedules	Equal

Known-good: one transient failure is followed by one successful attempt with the same repeat-protection key and one message. Wrong results: abandon, create two messages, change the key, exceed the retry limit, or change another action.

Worked Task B: handle duplicate triggers

Instruction:

Process the two visible trigger deliveries for the same source event. Apply the business change once and retain audit evidence for both deliveries.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
one_change	Business state changes once	State history	Count changes for the source event	One
delivery_log	Both deliveries are retained	Delivery records	Compare delivery ID set	Exact two IDs
one_effect	Both deliveries resolve to one effect	Delivery and effect records	Join by source event and effect ID	One shared effect
no_loss	Other visible triggers remain	Queue before and after	Compare unrelated trigger IDs and states	Equal

Known-good: both deliveries are logged and one business effect is committed. Wrong results: apply twice, drop the second delivery record, create two effect IDs, or consume an unrelated trigger.

Calibrate

Use a fixed clock. Run one delivery, duplicate delivery, transient retry, permanent failure, reordered events, partial effect, and timeout. Reset queue, attempt, repeat-protection, and effect state each time. Inspect the full event order and final state. Treat worker startup or clock-control failure as infrastructure.

Report assertion-level partial credit when useful for diagnosis or training. Keep a separate strict completion result that passes only when every required assertion passes.

Difficulty changes

Change one condition: longer delay, retryable then permanent error, concurrent worker, stale read, reordered event, or partial side effect. Use controlled time and known event order so failures remain reproducible.

Sources

AutomationBench: realistic business workflows over 47 simulated SaaS tools. Borrow its trigger data, pre-populated state, tool surface, assertion-based final-state checks, assertion-level partial credit, and separate strict completion result.
AppWorld: stateful APIs and database verification across applications.
ToolSandbox: stateful tool calls, milestones, and controlled interaction state.
TheAgentCompany: longer-running workplace workflows and artifacts.

Browser and Desktop Environments

Use this pattern

Use when the agent operates websites, browsers, desktop applications, files, and visual interfaces.

State and storage

Pin:

Website or application version.
Browser, extensions, profile, and viewport.
Operating system image and display settings.
Login state and assigned identity.
Application databases, files, downloads, and clipboard state.
Time, locale, timezone, and accessibility settings.

Use local sites or snapshots when live content changes results. Preserve realistic DOM, visual, pagination, latency, validation, and permission behavior. Use accessibility state only when the real Harness can use it.

Services, tools, and access

Choose live services only when accounts, data, reset, cost, and side effects are controlled. Prefer a local clone, frozen site, or sandbox for mutable workflows.

Match the real Harness input and observation surface:

Screenshots and cadence.
Mouse, keyboard, scroll, and window controls.
DOM or accessibility data when available in production.
Browser downloads and uploads.
Application launch and focus behavior.

Catalog model, website, SSO, OAuth, VPN, application, and cloud credentials by name, role, scope, provider, and network need. Never save cookies or secret values in specs or committed artifacts.

Setup and reset

Restore a pinned browser, VM, or application snapshot.
Establish approved login state through the setup boundary.
Place required files and clear unrelated downloads and clipboard data.
Set display, locale, clock, and window state.
Check readiness through the same surface the agent uses.
Save raw application state for the Verifier.
Reset the full profile or snapshot after every run.

Do not rely on clicking back or reversing UI actions as reset.

Strong Task families

Complete a multi-page business workflow.
Find and update the correct record among similar records.
Transfer data between a document and application.
Respect a permission or confirmation boundary.
Recover from validation or stale-page errors.

Worked Task A: update a user without changing access

Instruction:

Update the selected employee's department and phone number. Keep their role, group membership, and account status unchanged.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
identity	Correct employee was edited	Final application row	Match stable employee ID	Exact ID
fields	Department and phone match	Final application row	Compare normalized requested values	Both equal
access	Role, groups, and status stay fixed	Before and after access rows	Exact comparison	Equal
persistence	Saved values survive a new session	Reloaded row from clean session	Read again after restart	Both equal
scope	Other state stays fixed	Application state difference	Compare non-allowlisted rows and fields	No difference

Known-good: only the stable employee ID and requested fields change and remain after a new session. Wrong results: update a similar name, change access, leave an unsaved form, or edit another record while searching.

Worked Task B: file-to-application workflow

Instruction:

Read the approved local request file, create the matching application record, and save the confirmation PDF in /home/oai/share/confirmations.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
source	Created values match the approved request	Parsed source file and final row	Compare named fields	All equal
record	One record was created	Before and after application rows	Count new IDs and match type	One
artifact	One valid confirmation PDF exists	Required path and parsed PDF	Check path, file count, text, and record ID	One valid file
scope	No extra effect occurred	State, upload, message, and file differences	Compare with allowlist	No extra change

Use application state and file parsing for verification. Use visual comparison only for layout that cannot be checked structurally.

Known-good: one matching application row and one readable confirmation PDF. Wrong results: wrong source file, unsaved form, duplicate record, screenshot instead of PDF, wrong output path, or an extra upload or message.

Calibrate

Restore the snapshot before each reference, wrong-result, and model run. Test a similar identity, unsaved form, changed access, stale page, duplicate submit, and extra file. Verify raw application and file state after a fresh session. Treat login, display, or application startup failure as infrastructure.

Difficulty changes

Change one condition: similar records, more pages, modal or validation error, permission boundary, stale page, cross-application transfer, or required file artifact. Do not create difficulty through tiny targets, hidden UI, or unstable live content.

Sources

WebArena: reproducible websites, browser interaction, and state-based evaluation.
WorkArena: enterprise web tasks, roles, and workflow state.
OSWorld: operating-system and desktop application snapshots, computer input, and cross-application tasks.

Coding Environments and Tasks

Use this pattern

Use for bug repair, feature work, refactoring, tests, build systems, and codebase navigation.

State and storage

Pin repository URL, commit, submodules, large-file inputs, and toolchain.
Use a fresh worktree or copy per run.
Keep package caches outside the scored worktree or reset them explicitly.
Record repository instructions, generated-file sources, public exports, and build outputs.
Preserve enough repository history when the agent needs blame, prior changes, or release context.

Model relationships beyond files:

Public API to implementation.
Registration to runtime selection.
Schema to generated artifacts.
Source package to consumer import path.
Bug report to failing behavior and focused tests.

Services, tools, and access

Prefer a frozen repository and installed offline dependencies. Allow live package or web access only when the real job requires it and the run can remain safe and comparable.

Preserve the production Harness shell, editor, patch, search, test, and build interfaces. Record private registry credentials, source-host access, and model credentials by variable name only. Keep hidden tests and reference patches unavailable to the agent.

Setup and reset

Create a fresh worktree at the pinned commit.
Apply one controlled condition, such as an existing bug or generated mutation.
Confirm the public failure before the agent starts.
Capture Git status and relevant generated-file hashes.
Reset by discarding the whole task worktree, not by reversing guessed files.

Use a container or VM when system packages, services, or compiler state affect the result.

Strong Task families

Repair a real issue with a reproducible failing test.
Add a feature through the public API and consumer path.
Change lifecycle or state behavior across modules.
Update a schema and all required generated or migration surfaces.
Diagnose and fix a build, packaging, or release fault.

Do not score a preferred patch shape when tests and public behavior define correctness.

Worked Task A: repair duplicate pagination

Instruction:

Fix the duplicate-item pagination bug in /workspace. Keep the public API compatible. Add or update focused tests for duplicate-free traversal.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
behavior	Pagination has no duplicate items	Focused test output	Run the pinned reproduction and boundary tests	All pass
regression	Related behavior still works	Existing focused suite	Run the named test files	All pass
compatibility	Public call shape remains available	Consumer import and call test	Import and call the old public function	Test passes
scope	No unrelated change	Git diff and generated-file hashes	Compare with an allowlist	No extra path or hash change

Known-good: a patch that fixes cursor advancement and adds a focused boundary test. Wrong results: filtering duplicate output after retrieval, changing the public signature, disabling the failing test, or editing unrelated snapshots.

Worked Task B: add a public extension

Instruction:

Add the requested provider through the public package import path. Register it in runtime selection and add a consumer-facing test.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
implementation	Provider performs the requested behavior	Focused provider test	Run the named behavior case	Passes
registration	Runtime selects the provider	Factory test and registry state	Load valid config through the public factory	Correct class returned
public surface	Consumer import works	Clean consumer process	Import from the documented package path	Import succeeds
tests	Related code remains valid	Pinned internal and consumer suites	Run the named tests	All pass
scope	Other providers and defaults stay fixed	Git diff and config snapshot	Compare named files and defaults	No extra change

Known-good: implementation, registry entry, public export, and consumer test from the source files that own each surface. Wrong results: implementation without registration, internal export only, unit test only, or hand-edited generated output without its source update.

Calibrate

Run the reference patch, no-op patch, disabled-test shortcut, and unrelated edit. Run agent trials when they answer a named calibration question. Inspect the selected diffs and each named test. Treat toolchain, dependency, and build-start failures as infrastructure errors, not coding failures.

Difficulty changes

Change one condition: cross-package surface, longer call path, missing regression test, lifecycle interaction, migration, partial failure, or more plausible distractor code. Do not remove needed issue context or rely on a huge unrelated test suite to create difficulty.

Sources

SWE-bench: real repository issues, pinned commits, and test-based patch verification.
SWE-smith: programmatic mutation and scalable task generation. Borrow reproducible generation and execution; do not assume every mutation is useful work.
Terminal-Bench: isolated command-line Environments and reference solutions for tasks that cross system setup.

CRM and Knowledge-Work Environments

Use this pattern

Use for sales, support, operations, research, documents, messaging, meetings, and work that crosses business applications.

State and storage

Use a relational store for identities and mutable business records. Use files or document tables for email, notes, reports, transcripts, and attachments. Add search only when retrieval behavior is part of the Task.

Model:

People, organizations, accounts, and aliases across services.
Ownership, teams, roles, and permissions.
Opportunities, cases, tasks, and status history.
Messages, meetings, documents, and source records.
Product usage, contracts, and time-dependent facts.
Draft, sent, approved, and deleted states.
Conflicting, stale, and missing observations.

Create structural facts with code. Create selected prose through source-supported generation requests and criterion-level review.

Services, tools, and access

Choose per service:

Frozen for read-only corpora that must remain comparable.
Simulated for mutable CRM, messaging, calendar, or document state.
Live sandbox only when reset, privacy, cost, and side effects are controlled.
Disabled for irrelevant access that adds leakage or risk.

Match tool names, arguments, results, pagination, permission errors, and effects. Catalog CRM, email, calendar, document, model, and search credentials by name, account, role, scope, network need, and provider. Do not store values.

Setup and reset

Select a relationship-complete record set, not one isolated row.
Preserve identities, history, documents, and cross-service IDs.
Apply one named setup operation through a service or checked builder.
Give the agent only its assigned identity and production-shaped tools.
Reset from a clean database or snapshot after every run.
Save initial and final state plus message and document effects.

Strong Task families

Reconcile records across systems and update the source of truth.
Draft a message or report supported by several services.
Act on a request while respecting permissions and policy.
Detect stale or conflicting state and clarify before action.
Complete a multi-step workflow without duplicate or collateral effects.

Worked Task A: update an account after a meeting

Instruction:

Review the latest meeting and current account state. Update the account's next step and create an internal follow-up note. Do not send an external message.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
account	Correct account has the supported next step	Before and after CRM rows plus meeting	Match account ID and allowed values	Exact match
note	One supported internal note exists	New note and meeting facts	Check count and cited facts; judge only unsupported prose	One valid note
identity	Assigned user made permitted changes	Audit log, owner, and role	Match actor, owner, and permission rule	All match
no_message	No external draft or send exists	Message and draft tables	Compare before and after	Equal
scope	Other business records stay fixed	Stable state snapshot	Compare every non-allowlisted row and field	No difference

Known-good: supported next step plus one source-supported note. Wrong results: use an older meeting, update a similar account name, claim a decision not made, send an email, or change a related opportunity that was not requested.

Worked Task B: produce a portfolio brief

Instruction:

Create a brief for the assigned accounts. Rank urgent items, cite the source record for each claim, and state conflicts instead of resolving them without evidence.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
coverage	Each qualifying assigned account appears once	Assignment and issue rows plus brief	Recompute expected account IDs and counts	Exact set and count
ranking	Order follows the stated urgency rule	Source values and brief order	Recompute score and compare order	Exact order, allowing declared ties
sources	Each material claim names a valid record	Brief citations and source rows	Resolve each ID and compare its claimed fields	Every claim supported
uncertainty	Conflicts stay explicit	Conflicting rows and brief text	Check required conflict IDs; judge only whether wording preserves uncertainty	All conflicts stated; none invented

Known-good: every qualifying account appears once in recomputed order with valid citations and stated conflicts. Wrong results: omit an account, include an unassigned account, use stale state, cite a missing record, or hide a conflict. Use a limited judge only for claim support and useful wording.

Calibrate

Run the reference actions, wrong identity, stale-source, duplicate-effect, unsupported-claim, and collateral-change cases before model trials. Reset all services between runs. Inspect final state, audit rows, citations, and each meaning-based result. Do not treat service startup or login failure as Task failure.

Difficulty changes

Change one condition: similar names, longer history, cross-service identity, stale record, permission boundary, several valid actions, or collateral-effect risk. Do not make the Task hard by hiding a required source.

Sources

AppWorld: stateful application APIs, cross-application tasks, and database-based verification.
CRMArena-Pro: CRM roles, records, and business workflows. Use domain structure, not private data.
WorkArena++: enterprise application workflows and role-sensitive state.
tau-bench and tau-squared Bench: policy, user interaction, tool use, and final-state checks.
TheAgentCompany: longer workplace tasks across tools and artifacts.

Multi-Turn and Policy Environments

Use this pattern

Use when success depends on conversation, hidden user facts, clarification, consent, policy, escalation, or changes during a dialogue.

State and storage

Keep separate:

Agent-visible conversation.
User simulator facts and goals.
Policy rules.
Mutable service state.
Tool calls and effects.
Turn and time limits.

The simulator may know facts the agent must ask for. It must not reveal facts without the defined trigger or judge correctness itself. Store final service state independently from dialogue.

Services, tools, and access

Preserve production tool schemas, permission errors, and state changes. Use a simulated user for repeatability and a simulated or sandbox service for mutable effects.

Catalog model credentials for the agent, user simulator, and any judge separately. Pin each model. Keep simulator facts, policies, and expected results hidden from the agent.

User simulator contract

Define:

Goal and known facts.
Facts supplied initially.
Facts supplied only after a clear question.
Behavior on vague, repeated, or leading questions.
Consent and refusal behavior.
When the user corrects a mistake.
Stop condition and turn limit.

Do not let the simulator help the agent simply because progress stalled.

Setup and reset

Create fresh service state and conversation state.
Set the assigned identity, policy version, clock, and simulator facts.
Check tools before the first turn.
Save every turn, tool call, and state change.
Reset both conversation and service state after every run.

Strong Task families

Gather required facts before an action.
Apply policy while helping the user reach a valid outcome.
Clarify conflicting identity or intent.
Refuse or escalate a prohibited request.
Recover after the user changes a material fact.

Worked Task A: policy-bound refund

User start:

I need a refund for my last order.

Hidden user facts include order identity, reason, and whether the item was used. Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
identity	Customer and order are confirmed	Dialogue and service IDs	Find explicit confirmation before action	Correct IDs confirmed
facts	Required facts are obtained first	Ordered turns and policy fields	Check each required answer precedes the action	Complete and ordered
policy	Action follows the pinned rule	Policy and confirmed facts	Recompute allowed action	Exact match
state	Service records match that action	Final order, refund, and escalation rows	Compare expected fields and counts	Exact match
scope	No extra effect occurred	Before and after service state	Compare non-allowlisted rows	Equal

Known-good: confirm the order and required facts, then create only the refund or escalation allowed by policy. Wrong results: act on the latest order without confirmation, infer item state, invent an exception, create both refund and credit, or refuse an allowed request.

Worked Task B: clarify an identity conflict

User start:

Move Alex's meeting to Friday afternoon.

Two visible contacts match Alex. Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
clarification	Agent asks before changing state	Ordered dialogue	Find a question that names a distinguishing fact before the tool call	Present and ordered
selection	Confirmed contact owns the changed event	User reply and final event	Resolve confirmed contact ID and compare	Exact match
time	New time follows request and calendar rules	Final event, clock, and policy	Recompute allowed Friday window	Within window
scope	Other events and attendees stay fixed	Calendar difference	Compare with focal-event allowlist	No extra change

Dialogue checks should establish that clarification happened. Final calendar state proves the action.

Known-good: ask one distinguishing question, receive the answer, then move only the confirmed contact's event. Wrong results: guess from name, ask after the move, change both events, choose a blocked time, or change attendees.

Calibrate

Run scripted good, ambiguous, changed-request, refusal, repeated-question, and premature-action conversations before model trials. Inspect ordered turns and final service state. Reset both after every run. Treat simulator, model, or tool startup failure as infrastructure, not policy failure.

Difficulty changes

Change one condition: more hidden facts, policy exception, conflicting user statement, changed request, permission boundary, or tool state change. Do not make the simulator evasive beyond the defined user behavior.

Sources

tau-bench: user simulation, policy-guided tool use, and database final state.
tau-squared Bench: longer interactions and both sides of a tool-mediated dialogue.
ToolSandbox: stateful tool use, milestones, and controlled hidden state.

Terminal Environments and Tasks

Use this pattern

Use for command-line diagnosis, repair, build, data processing, services, systems work, and tasks that require a container or VM.

State and storage

Pin:

Base image or VM snapshot.
Operating system and architecture.
Installed tools and versions.
Files, permissions, owners, and working directory.
Processes, ports, services, and clocks.
Offline packages, datasets, and repositories.
CPU, memory, storage, GPU, and network limits.

Keep tests and reference solutions outside the agent-visible image. Store starting-state and final-state evidence for files, processes, services, and command results.

Services, tools, and access

Use a single container when possible. Use Compose only for real service boundaries; name the primary Harbor service main. Add health checks for dependencies.

Disable network unless the real Task requires it. Record registry, cloud, SSH, database, and model credential names and scopes. Never bake secrets into images or pass them as build arguments.

Setup and reset

Build from a pinned image.
Install pinned runtime dependencies or include approved offline assets.
Place agent-visible files only.
Start services and check observable readiness.
Capture baseline files, permissions, processes, and service state.
Reset by replacing the container or VM.

Separate Environment build errors, service startup failures, timeouts, and Verifier errors from Task failure.

Strong Task families

Diagnose and repair a broken service.
Restore or transform data under constraints.
Configure a tool or deployment correctly.
Repair a build or dependency fault.
Investigate logs and produce a tested fix.

Worked Task A: repair a local service

Instruction:

The service in /app fails its health check after restart. Diagnose and fix it. Keep its public endpoint and stored records compatible.

Use five required criteria:

ID	Required result	Evidence	Exact check	Pass
readiness	Service becomes healthy	Process, port, and health output	Start with the pinned command and poll to the limit	Healthy in time
behavior	Public endpoint is correct	Request and response	Run named requests and compare status and body	All match
persistence	Existing records stay readable	Baseline and final store	Read and compare every pinned record	Equal
restart	Fix survives restart	Fresh process and requests	Stop, start, wait, and rerun checks	All pass
scope	Other system state stays fixed	Port, process, file, owner, and permission differences	Compare with allowlist	No extra change

Known-good: the normal start command reaches health, serves correct stored data, and does the same after restart. Wrong results: bypass health, hardcode a response, delete stored data, leave a foreground process only, or open an extra port.

Worked Task B: recover a damaged dataset

Instruction:

Recover all valid records from /data/input, write stable, sorted JSONL to /data/output/recovered.jsonl, and report rejected record IDs.

Use four required criteria:

ID	Required result	Evidence	Exact check	Pass
recovery	Every valid record appears once	Independent parser and output	Recompute valid IDs and compare output IDs	Exact set, no duplicates
rejection	Every invalid ID has a valid reason	Independent errors and report	Compare ID set and allowed reason per ID	Exact match
format	Output is stable JSONL	Parsed output and second run	Check schema, sort key, bytes, and repeat build	All match
scope	Inputs and other files stay fixed	Before and after file hashes	Compare outside the output allowlist	Equal

Known-good: recover the exact valid ID set in stable order and report every invalid ID once. Wrong results: drop or duplicate rows, repair unsupported values, write only to stdout, change an input, use unstable order, or give an unsupported rejection reason.

Calibrate

Run the reference solution, no-op, hardcoded response, data deletion, foreground-only service, partial recovery, and unrelated-file edit. Rebuild the container or VM for each run and test restart. Treat image build, service startup, and resource failure as infrastructure.

Difficulty changes

Change one condition: more services, partial failure, permission fault, restart, larger data, limited resources, or cross-file consistency. Do not create difficulty through missing tools, impossible timeouts, or hidden network needs.

Sources

Terminal-Bench: isolated terminal Environments, natural instructions, tests, and reference solutions. Pin the exact 2.x or 3.x release because formats and Verifiers can change.
SWE-bench: repository repair and test-based correctness when the terminal Task includes code changes.

Cross-domain quality checks

For every adapted pattern, ask:

Does the Task require useful production work and the selected capability?
Is all needed information visible or normally discoverable?
Does the Environment preserve the relationships that change the decision?
Can independent evidence prove required and prohibited results?
Can an agent pass through wording, record order, leakage, or a shortcut?
Does the Task add a meaningful condition rather than rewrite an old prompt?
Which benchmark design was adapted, and which limitation was rejected?
If model contrast was useful, what did the selected complete traces reveal?

Store proven project-specific adaptations in the project World Skill. Keep the exact request, focal state, expected answer, and scoring criteria in the collocated Task.md.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

LangChain

Publisher

LangChain (Official)

Source

langchain-ai/langchain-skills · config/skills/eval-engineering · commit 126c384

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by LangChain or LangChain.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download eval-engineering.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by LangChain

Eval Engineering

Flow

Terms

Reference routing

1. Inspect inputs and existing World knowledge

2. Propose and select a Task

3. Write and review the Task Spec and World Skill

4. Apply Spec2Task

5. Run and audit

6. Reconcile project World knowledge

7. Repeat

Dependencies, access, and safety

Complete only when

Files

references/patterns.md

Benchmark Patterns

Contents

Benchmark design index

Automation Environments

Use this pattern

State and storage

Services, tools, and access

Setup and reset

Strong Task families

Worked Task A: retry a transient delivery

Worked Task B: handle duplicate triggers

Calibrate

Difficulty changes

Sources

Browser and Desktop Environments

Use this pattern

State and storage

Services, tools, and access

Setup and reset

Strong Task families

Worked Task A: update a user without changing access

Worked Task B: file-to-application workflow

Calibrate

Difficulty changes

Sources

Coding Environments and Tasks

Use this pattern

State and storage

Services, tools, and access

Setup and reset

Strong Task families

Worked Task A: repair duplicate pagination

Worked Task B: add a public extension

Calibrate

Difficulty changes

Sources

CRM and Knowledge-Work Environments

Use this pattern

State and storage

Services, tools, and access

Setup and reset

Strong Task families

Worked Task A: update an account after a meeting

Worked Task B: produce a portfolio brief

Calibrate

Difficulty changes

Sources

Multi-Turn and Policy Environments

Use this pattern

State and storage

Services, tools, and access

User simulator contract

Setup and reset

Strong Task families

Worked Task A: policy-bound refund

Worked Task B: clarify an identity conflict

Calibrate