AutoGineer
Generated banner for Eval Harness First
Generated logo for Eval Harness First

Eval Harness First

Seth HobsonCommunityCreated by Seth Hobson

  • Skill
  • Workflow
  • Software development
  • MIT
  • No issues found
Download ZIP

From the author

Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for building an evaluation harness that acts as the primary gate for fine-tuning efforts. It emphasizes data curation, error analysis, and the creation of a regression suite that serves as both the training data source and the final checkpoint validator. By establishing deterministic graders and calibrated LLM-judges, the harness ensures that every fine-tuning run is measured against a consistent, versioned baseline, preventing regressions and providing a clear metric for model improvement.

What it does

  • Guides the collection and axial coding of traces into failure buckets.
  • Defines a directory structure for versioned goldens, graders, and drift suites.
  • Provides templates for deterministic and LLM-based grading.
  • Outlines a calibration protocol for LLM-judges to ensure reliable verdicts.
  • Establishes a baseline against the unmodified base model for future comparisons.
  • Ensures evaluation data is strictly separated from training data.

When to use it

  • Before starting any fine-tuning effort.
  • When converting production traces into a structured evaluation set.
  • When calibrating an LLM-judge against human-labeled data.
  • When establishing a CI regression suite for model checkpoints.

Advantages

  • Prevents regressions by gating checkpoints against a fixed baseline.
  • Provides a clear, data-driven basis for model promotion.
  • Reduces noise in evaluation by using binary pass/fail grading.
  • Ensures evaluation metrics are reproducible and versioned like code.

Usage examples

“How do I start the Phase 0 evaluation setup for my fine-tuning project?”
The agent will guide you through collecting 100+ traces, performing axial coding to define 4-8 failure buckets, and creating the initial eval/ directory structure.
“I need to set up a grader for my JSON schema compliance failure bucket.”
The agent will provide the schema-compliance Python template and explain how to integrate it into the eval/graders/ directory.
“How do I calibrate my LLM-judge for the tone-consistency bucket?”
The agent will walk you through the labeling, splitting, and TPR/TNR calculation protocol defined in the calibration reference.
“What should I do before running my first fine-tuning training config?”
The agent will verify the Phase 0 Exit Checklist, ensuring goldens, graders, judge calibration, and the baseline file are all present and valid.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/wshobson/eval-harness-first/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip eval-harness-first.zip -d .agents/skills

Documentation by Seth Hobson

Eval Harness First

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:

  1. Collect traces — production/agent spans, or synthetic tasks if none exist yet.

  2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.

  3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.

  4. Prioritize by frequency × severity × value.

  5. The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID.

  6. Train.

  7. Re-run the same harness on the checkpoint — not a different, looser one.

  8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis.

Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate.

Building Goldens

  • From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. strict-schema extraction) may land at 1–2 buckets with per-field sub-metrics inside one grader — don't invent artificial splits with no evidence behind them.

  • Synthetic, when traces don't exist yet: dimension-based generation — enumerate the axes that matter (task type, difficulty, edge case, persona) and sample the cross-product; free- generated prompts cluster around whatever's easiest to write.

  • Goldens are versioned like code — commit eval/goldens.jsonl, diff it in review, tag it per release. It doubles as the CI regression suite.

Graders

One grader per failure bucket from error analysis — not one for the whole eval set. A single blended score hides which bucket regressed.

  • Deterministic first. Regex, schema validation, or execution checks are cheaper, reproducible, and need no calibration.

  • LLM-judge only for genuinely subjective criteria — tone, faithfulness, "which response is better" — where no deterministic check can express it.

  • Binary pass/fail over Likert. A 1–5 or 1–10 scale is noisier to calibrate and harder to apply consistently; collapse to pass/fail.

  • Drift-suite MMLU-style scoring: prefer logprob over generate-and-extract — a tight token budget makes generate-and-extract parse-brittle for models that preamble, conflating format compliance with the knowledge being measured. Templates for all four grader shapes and this scoring note: references/grader-templates.md.

Judge Calibration Is a Prerequisite

Any bucket routed to an LLM-judge needs calibration before its verdicts count for anything beyond exploration — a hard prerequisite, not a nice-to-have. N/A when no bucket routes to a judge — an all-deterministic harness has nothing to calibrate; state that rather than leaving this section unaddressed.

  • Label ≥100 items, split train/dev/sealed test (report once, no re-touching after).

  • Report TPR and TNR, not one blended accuracy number — a judge can hit 90% by always saying "pass" on a skewed set.

  • Pin the judge to a fixed model snapshot and recalibrate on judge-model change, quarterly regardless.

  • The judge must come from a different model family than the model under test.

  • A judge that misses the agreed TPR/TNR bar ships advisory-only — flags for human review, never gates a promotion. Full protocol, bias correction, and recalibration checklist: references/judge-calibration.md.

The Baseline

Before Phase 1 (method selection) starts, run the full harness — goldens plus the capability-drift suite — against the unmodified base model. This is the number every later checkpoint gets compared against.

eval/baseline-<model>.json is the gate token. No baseline file, no comparison basis for checkpoint-promotion — a checkpoint that "looks better" against nothing measured isn't a finding.

Directory Contract

eval/
├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
├── graders/                # one module per failure bucket
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json   # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
    └── results.json         # per-run harness output, one per checkpoint

eval/ persists across runs and lives outside runs/ — the fixed measuring stick, not a run artifact. runs/ is disposable; eval/ is not. Never let a run script write into eval/. Canonical location: every per-trace results.json — the Phase 0 baseline included — lives at runs/<run-id>/results.json, never under eval/runs/...; an instruction requesting the latter is wrong, not this contract.

Phase 0 Exit Checklist

Before finetuning-method-selection, confirm:

  1. ≥100 traces open-coded; 4–8 failure buckets (N/A floor for synthetic goldens on a single-failure- surface task — see the Building Goldens exception; bucket count then comes from post-baseline error analysis instead).

  2. eval/goldens.jsonl committed and versioned.

  3. One grader per bucket, deterministic first.

  4. Judges calibrated — TPR/TNR, snapshot pinned, different family (N/A when no bucket routes to an LLM-judge; state that explicitly).

  5. eval/drift-suite.yaml frozen.

  6. eval/baseline-<model>.json written.

Missing any of the six (or its stated N/A)? Not Phase 0 complete — /finetune checks the baseline file before a run.

Related Skills

General-purpose evaluation guidance (dashboards, A/B testing, non-fine-tuning harnesses) lives in the llm-application-dev plugin's llm-evaluation skill — this skill covers only the fine-tuning coupling: goldens that double as training data, and the baseline that gates a checkpoint.

  • finetuning-method-selection — routes here first.

  • dataset-curation — formats these traces into training rows.

  • trace-to-training-data — turns graded traces into training examples.

  • checkpoint-promotion — consumes baseline-<model>.json, re-runs this harness on each candidate checkpoint.

References

  • references/grader-templates.md — runnable grader examples per shape, plus a drift-suite.yaml example and MMLU logprob-scoring note.

  • references/judge-calibration.md — the calibration protocol, including the all- deterministic N/A path.

Files

references/grader-templates.md

Last verified: 2026-07-14

Grader Templates

Runnable examples for the four grader shapes named in SKILL.md's Graders section: schema-compliance, exact-match with normalization, execution-based, and LLM-judge. Every grader returns a binary pass/fail — never a Likert score — per the plugin-wide rule. Wire each one to exactly one failure bucket from error analysis; don't blend buckets into one grader.

Schema Compliance

For buckets where the failure mode is "the output isn't shaped right" — tool calls, structured extraction, JSON responses:

import json
from jsonschema import validate, ValidationError

RESPONSE_SCHEMA = {
    "type": "object",
    "required": ["action", "arguments"],
    "properties": {
        "action": {"type": "string"},
        "arguments": {"type": "object"},
    },
    "additionalProperties": False,
}

def grade_schema_compliance(completion: str) -> bool:
    """Binary pass/fail: does the completion parse as
    JSON and match RESPONSE_SCHEMA? Malformed JSON is
    an automatic fail, not an exception to handle
    upstream — the grader owns that decision.
    """
    try:
        payload = json.loads(completion)
    except json.JSONDecodeError:
        return False
    try:
        validate(instance=payload, schema=RESPONSE_SCHEMA)
    except ValidationError:
        return False
    return True

Exact Match with Normalization

For buckets with a single ground-truth string (math final answers, extracted entities, classification labels) where naive == fails on formatting noise:

import re

def normalize(text: str) -> str:
    """Lowercase, collapse whitespace, strip
    punctuation and surrounding markup — apply the
    identical normalization to both prediction and
    ground truth so neither side gets an unfair pass.
    """
    text = text.strip().lower()
    text = re.sub(r"[^\w\s.]", "", text)
    text = re.sub(r"\s+", " ", text)
    return text

def grade_exact_match(completion: str, ground_truth: str) -> bool:
    return normalize(completion) == normalize(ground_truth)

Execution-Based

For buckets where correctness means "the code runs and does the right thing" — the strongest signal available when applicable, since it needs no normalization or judgment call.

WARNING — this grader executes model-generated code and REQUIRES an isolated environment: a network-disabled container, gVisor/firejail, or a dedicated CI sandbox, with no secrets or credentials in the environment — no HF tokens, experiment-tracker keys, cloud credentials, or SSH keys. Never run it directly on a host holding credentials. The timeout below protects grading-loop liveness only — it is NOT a security boundary; isolation comes entirely from sandbox_cmd.

import logging
import subprocess
import tempfile
from pathlib import Path

logger = logging.getLogger(__name__)

def grade_execution(completion: str, test_code: str, sandbox_cmd: list[str]) -> bool:
    """Write the completion plus a pytest test file to
    a scratch dir, run pytest via `sandbox_cmd`, and
    pass only on a clean exit code. Timeouts and
    non-zero exits are both failures — never treat a
    hung process as a pass by default.

    SECURITY: executes model-generated code. This
    function REQUIRES an isolation boundary — it does
    not run anything on the host by itself.

    `sandbox_cmd` (list[str], required) is a command
    prefix that wraps pytest in that boundary, e.g. a
    network-disabled, resource-capped Docker container:

        # sandbox_cmd = [
        #     "docker", "run", "--rm", "--network=none",
        #     "--memory=1g", "--cpus=1",
        #     "-v", f"{workdir}:/work:ro", "-w", "/work",
        #     "python:3.12-slim",
        # ]

    If `sandbox_cmd` is falsy, this function refuses to
    execute anything and returns False — it never falls
    back to running pytest on the host. The subprocess
    environment is scrubbed to a minimal PATH.
    """
    if not sandbox_cmd:
        logger.warning(
            "grade_execution: no sandbox boundary provided "
            "— refusing to execute model-generated code"
        )
        return False

    scrubbed_env = {"PATH": "/usr/bin:/bin"}
    with tempfile.TemporaryDirectory() as tmp:
        solution = Path(tmp) / "solution.py"
        test_file = Path(tmp) / "test_solution.py"
        solution.write_text(completion)
        test_file.write_text(test_code)
        try:
            result = subprocess.run(
                [*sandbox_cmd, "python", "-m", "pytest",
                 str(test_file), "-q"],
                cwd=tmp,
                capture_output=True,
                timeout=30,
                env=scrubbed_env,
            )
            return result.returncode == 0
        except subprocess.TimeoutExpired:
            return False

LLM-Judge Template

Only for buckets that failed the deterministic-first check in SKILL.md — genuinely subjective criteria. Few-shot slots and a binary output contract are both mandatory; a judge without few-shot anchors drifts toward its own prior instead of the calibrated labels.

JUDGE_PROMPT = """You are grading whether a response
meets the following criterion: {criterion}

Examples of PASS:
{few_shot_pass_examples}

Examples of FAIL:
{few_shot_fail_examples}

Now grade this response. Output exactly one word,
PASS or FAIL, with no other text.

Task: {task}
Response: {response}
Verdict:"""

class JudgeParseError(Exception):
    """Raised when the judge returns anything other than
    exactly PASS or FAIL — a transport or format failure,
    not a grading verdict. Never caught and coerced to
    False; the caller retries the judge call or routes the
    item to human review."""

def grade_llm_judge(task: str, response: str, criterion: str,
                     few_shot_pass: str, few_shot_fail: str,
                     judge_client) -> bool:
    prompt = JUDGE_PROMPT.format(
        criterion=criterion,
        few_shot_pass_examples=few_shot_pass,
        few_shot_fail_examples=few_shot_fail,
        task=task,
        response=response,
    )
    # judge_client is pinned to a fixed snapshot per
    # SKILL.md's Judge Calibration section — never an
    # unpinned "latest" alias.
    verdict = judge_client.complete(prompt, temperature=0).strip().upper()
    if verdict not in ("PASS", "FAIL"):
        raise JudgeParseError(f"unparseable judge verdict: {verdict!r}")
    return verdict == "PASS"

The grader's return value stays binary pass/fail per this file's plugin-wide rule — JudgeParseError is not a third grading state, it's an operational failure. Never catch it and coerce to False; log it and re-run the judge call or route the item to human review instead.

drift-suite.yaml Example

Frozen general-capability benchmarks plus a domain-adjacent slice, per SKILL.md's Directory Contract. Benchmark subsets are frozen at a fixed seed/split so re-runs are comparable run over run:

# eval/drift-suite.yaml
frozen_benchmarks:
  - name: mmlu-subset
    source: mmlu
    split: test
    n_items: 500
    seed: 42
  - name: gsm8k-subset
    source: gsm8k
    split: test
    n_items: 250
    seed: 42
  - name: ifeval
    source: ifeval
    split: test
    n_items: 300
    seed: 42

domain_adjacent:
  path: eval/goldens.jsonl
  filter: "tag == 'drift-suite'"
  n_items_range: [200, 500]

drift_budget:
  noise_tolerance_pts: 1
  rerun_seed_variation_pts: [2, 5]
  hard_fail_threshold_pts: 5   # >5pt drop is a hard fail regardless of task-metric gains

Minimum viable drift suite for a small/dogfood run: the 500/250/300 general-benchmark sizes and the 200-500 domain_adjacent range above are scoped for production-scale runs and can be hours of wall-clock at greedy single-stream decode on modest hardware. Scope down deliberately rather than silently truncating — pick n per benchmark using checkpoint-promotion's item-count-from-budget rule (95% CI half-width comfortably under half the hard-fail threshold), and document the scoped-down suite inline in drift-suite.yaml (a comment naming the budget it was sized against). The domain_adjacent block itself is optional when the golden set is small and already fully consumed by the task harness — if eval/goldens.jsonl has no separate drift-suite-tagged holdout because every golden is already spoken for by the task grader, omit the block rather than fabricating a slice with nothing behind it; the frozen general benchmarks alone still provide a capability-drift signal.

Drift-Suite MMLU-Style Scoring: Logprob Over Generate-and-Extract

"Letter-choice logprob or generate-and-extract" are not equivalent scoring methods, though earlier guidance in this plugin implied they were interchangeable:

  • Generate-and-extract (sample a completion, extract the first A–D letter from a tight token budget) is parse-brittle for models that preamble before answering. Observed on a real drift run: a near-base checkpoint hit 9/100 unparsed-scored- as-wrong items (all prose restarts like "The energy levels of...") on a max_new_tokens=8 budget, versus 0–1 unparsed for other checkpoints in the same series — this conflates answer-format compliance with the knowledge MMLU is supposed to measure, and can hard-fail a checkpoint on formatting alone.

  • Logprob scoring (compare the model's log-probability on each of the four answer-letter tokens directly, no generation or parsing involved) is robust to this failure mode entirely — there is nothing to parse, so a verbose or preambling model scores on its actual answer distribution.

Prefer logprob scoring whenever the harness has logit access (local inference, not a hosted API that only returns text). When logit access isn't available and generate-and-extract is the only option, use a generous token budget and retry with a longer budget on any unparsed output before scoring it as wrong — don't let a tight budget silently convert "the model reasoned before answering" into "the model failed the benchmark."

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Seth Hobson
Publisher
Seth Hobson (Community)
Source
wshobson/agents · plugins/llm-finetuning/skills/eval-harness-first · commit 4bde316
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Seth Hobson or Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download eval-harness-first.zip