Trace To Training Data

Seth HobsonCommunityCreated by Seth Hobson

Skill
Workflow
Data and AI
MIT
No issues found

From the author

Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.

— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill processes graded model traces into structured training data. It assumes that an evaluation harness has already assigned verdicts and rewards to traces. The skill provides methods to transform these graded outputs into Supervised Fine-Tuning (SFT) examples or Direct Preference Optimization (DPO) pairs, ensuring that data is properly formatted, deduplicated, and sanitized of PII or secrets before being used for training.

What it does

Converts graded JSONL traces into SFT message formats.
Constructs DPO preference pairs from passing and failing trajectories on the same task.
Applies reward-based filtering to select training examples.
Redacts PII and secrets from production logs before export.
Enforces holdouts to prevent evaluation goldens from leaking into training sets.
Maintains data provenance by linking training rows to source run and trace IDs.

When to use it

When you have graded evaluation results and need to generate training data.
When applying rejection sampling to model outputs.
When building DPO pairs from existing passing and failing runs.
When you need to incorporate expert-corrected failures into your SFT set.

Advantages

Automates the transformation of graded traces into standard training formats.
Ensures training data hygiene through automated PII and secret redaction.
Prevents data leakage by filtering out evaluation goldens.
Supports fine-grained control such as step-level masking for multi-step traces.

Usage examples

“Convert my latest graded evaluation results into SFT training rows.”: The skill processes the results.json file, strips grading metadata, and outputs a JSONL file formatted for SFT training.
“Create DPO pairs from my recent production logs.”: The skill identifies pairs of passing and failing traces for the same task and generates chosen/rejected pairs based on reward distribution.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/wshobson/trace-to-training-data/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip trace-to-training-data.zip -d .agents/skills

Documentation by Seth Hobson

Trace To Training Data

This skill assumes eval-harness-first already graded the traces being converted here — goldens, graders, and runs/<run-id>/results.json all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream.

Input: graded traces — eval/goldens.jsonl plus runs/<run-id>/results.json, each row carrying a task_id, a verdict from the grader, and a reward when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier):

{"task_id": "t-042", "trace_id": "t-042-a3",
 "messages": [{"role": "user", "content": "..."}],
 "verdict": "pass", "reward": 0.91,
 "grader": "exact_match"}

Output format: rows shaped exactly like dataset-curation's Format Selection table — SFT messages rows or DPO prompt/chosen/rejected pairs — so this skill's output is that skill's input with no reshaping step in between.

The Principle

The eval harness already did the labeling work: every trace in results.json carries a verdict, and often a reward, before this skill ever touches it. Converting a graded trace into a training row is mechanical — pick a shape from dataset-curation's table, map fields, write JSONL. Curation is the work that remains — which traces clear a quality bar, which pairs are informative, and which rows must never enter the training set at all.

Treat any conversion step that requires re-judging a trace as a sign the harness is missing a grader, not a gap this skill should paper over. A trace with no verdict or reward isn't convertible yet — route it back to eval-harness-first first, don't hand-label it here to unblock conversion.

SFT From Traces

Keep the top-reward fraction of successful trajectories, not every passing one. Rank passing traces by reward and take a fraction (the Agent-lightning pattern) rather than every trace that merely cleared the pass bar — a trace that barely passed is a weaker SFT signal than one that scored well above threshold.
Expert-corrected failures become gold SFT examples directly (the Langfuse pattern) — when a human edits a failing trace's output into a correct one, that correction needs no reward threshold; a human already validated it. Route corrections straight into the SFT set.
Step-level masking beats whole-trajectory discard for multi-step traces. When only some steps in a multi-step trajectory are bad, mask the loss on the bad steps and keep the good ones, rather than discarding the whole trajectory. SRFT reports 32.2% vs. 30.9% on SWE-bench for step-level critic masking over trajectory discard — a real, if modest, gap from the finer-grained cut.

Preference Pairs From Traces

Build pairs from passing-vs-failing trajectories on the SAME task, never from unrelated best- and worst-scoring traces pulled across different tasks — cross-task pairs teach the model to prefer one task over another, not one response over another.
Select the rejected member at μ−2σ of the reward distribution for that task, never the absolute minimum. preference-optimization's Pair Construction section owns the full selection formula; this skill supplies the graded trajectories it consumes.
Judge-scored delta selection cuts pair volume without cutting signal. Score each candidate pair by chosen-minus-rejected judge delta and keep only the highest-delta subset — the top 5k of a 16.5k candidate pool matched the full pool's downstream result. Build the full candidate set first, then filter by delta; don't cap generation at 5k up front.

Hygiene

Scan for secrets and PII before any row ships, and redact what's found. Traces sourced from production logs can carry credentials, API keys, tokens, or customer data — run a secret/PII scan over every SFT and DPO row and redact matches; conversion fails closed (the row is dropped, not shipped with the raw content) if sensitive fields remain after redaction. Never commit secrets.
Eval goldens must never leak into training data. Hold every eval/goldens.jsonl ID out of every converted SFT and DPO set — a trace that also appears as a golden trains on the exact item the checkpoint gets graded against later, silently inflating every subsequent eval run.
Dedup against the training set, not just within the newly converted rows — exact-match or embedding-similarity, matching dataset-curation's dedup method field, run against whatever training data already exists before this batch merges in.
Provenance goes into the dataset card. Every converted row must trace back to its source run_id and trace_id — dataset-curation's Provenance field checks for exactly this link back to trace-to-training-data output; a row with no traceable source isn't ready to merge.

eval-harness-first — produces the graded traces this skill converts; a trace with no verdict or reward isn't convertible yet, route it back there before conversion.
dataset-curation — owns the target formats and the dataset card this skill's provenance data feeds; converted rows must match its Format Selection table field names exactly, not an approximation of them.
preference-optimization — consumes the DPO pairs this skill builds and owns the full μ−2σ rejection-selection formula referenced above.

Worked JSONL-to-JSONL conversions — graded trace to SFT row, trace pair to DPO pair, correction to SFT row, the rejection-sampling loop, and the goldens-holdout check — live in references/conversion-recipes.md.

Files

ATTRIBUTION.mdAdded by AutoGineer994 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md6.1 KiB
references/conversion-recipes.md7.6 KiB

references/conversion-recipes.md

Last verified: 2026-07-13

Conversion Recipes

Concrete JSONL-to-JSONL conversions for every pattern in SKILL.md: a graded trace to an SFT row, a pair of graded traces to a DPO pair, an expert correction to an SFT row, the rejection-sampling loop with reward-threshold selection, and the goldens-holdout check that must run before any converted row merges into a training set. Input records match eval-harness-first's runs/<run-id>/results.json shape; output records match dataset-curation's Format Selection table exactly — field names are copied, not paraphrased. No base-model names appear below; where a model matters, load it from finetuning-method-selection's references/model-catalog.md.

1. Graded Trace to SFT Example

Input — one row from runs/<run-id>/results.json, a single-turn trace that passed with a reward above the batch's top-fraction threshold:

{"task_id": "t-118", "trace_id": "t-118-a1", "messages": [{"role": "user", "content": "Summarize the incident report in two sentences."}, {"role": "assistant", "content": "A misconfigured retry policy caused a 12-minute outage in the billing service. Root cause was fixed and a regression test was added."}], "verdict": "pass", "reward": 0.94, "grader": "rubric_judge"}

Output — dataset-curation's ChatML shape, grading metadata stripped since the trainer only needs messages:

{"messages": [{"role": "user", "content": "Summarize the incident report in two sentences."}, {"role": "assistant", "content": "A misconfigured retry policy caused a 12-minute outage in the billing service. Root cause was fixed and a regression test was added."}]}

Only messages survives the conversion. task_id and trace_id still get written to the dataset card's provenance field (see the holdout check below) — they are dropped from the training row itself, not discarded entirely.

2. Two Graded Traces to a DPO Pair

Input — two traces sharing one task_id, from the same rollout batch, with different rewards:

{"task_id": "t-204", "trace_id": "t-204-a1", "messages": [{"role": "user", "content": "Write a commit message for a null-check fix."}, {"role": "assistant", "content": "Fix null pointer exception in user lookup by validating the session before dereferencing it."}], "verdict": "pass", "reward": 0.88, "grader": "rubric_judge"}
{"task_id": "t-204", "trace_id": "t-204-a4", "messages": [{"role": "user", "content": "Write a commit message for a null-check fix."}, {"role": "assistant", "content": "misc changes"}], "verdict": "fail", "reward": 0.11, "grader": "rubric_judge"}

Selection, per preference-optimization's Pair Construction formula — chosen is the top-reward trace for the task_id; rejected is whichever trace in that task's trajectory set sits closest to μ−2σ of the reward distribution, not the lowest-reward trace by default (here, with only two candidates, the low trace happens to be the μ−2σ pick; a batch with more sampled candidates per task selects a rejected member above the minimum):

def select_pair(trajectories):
    """trajectories: same task_id, each a dict with
    'reward' and 'messages'. Returns (chosen, rejected) —
    always two distinct records; raises if fewer than two
    trajectories are given."""
    if len(trajectories) < 2:
        raise ValueError("select_pair needs >=2 trajectories to form a pair")
    ranked = sorted(trajectories, key=lambda t: t["reward"])
    chosen = ranked[-1]
    candidates = [t for t in trajectories if t is not chosen]
    rewards = [t["reward"] for t in trajectories]
    mu = sum(rewards) / len(rewards)
    variance = sum((r - mu) ** 2 for r in rewards) / len(rewards)
    sigma = variance ** 0.5
    target = mu - 2 * sigma
    rejected = min(candidates, key=lambda t: abs(t["reward"] - target))
    return chosen, rejected

Output — dataset-curation's DPO pair shape, with prompt pulled from the shared user turn and chosen/rejected from each trace's final assistant turn:

{"prompt": "Write a commit message for a null-check fix.", "chosen": "Fix null pointer exception in user lookup by validating the session before dereferencing it.", "rejected": "misc changes"}

3. Correction Record to SFT Example

Input — a failing trace plus a human expert's corrected output, no reward field required since a human already validated the correction:

{"task_id": "t-311", "trace_id": "t-311-a2", "messages": [{"role": "user", "content": "Extract the invoice total as a JSON number."}, {"role": "assistant", "content": "The total is around $4,200"}], "verdict": "fail", "grader": "schema_compliance", "correction": {"content": "{\"total\": 4200.00}", "corrected_by": "reviewer-07"}}

Output — the corrected content replaces the failing assistant turn; the original failing content never enters the training set:

{"messages": [{"role": "user", "content": "Extract the invoice total as a JSON number."}, {"role": "assistant", "content": "{\"total\": 4200.00}"}]}

Route corrections into the SFT set directly, per SKILL.md's SFT From Traces section — skip the reward-threshold gate below for these rows.

4. Rejection-Sampling Loop

Sample several candidate completions per prompt, grade each, and keep only the top-reward fraction — the Agent-lightning pattern named in SKILL.md:

MAX_CANDIDATES = 32  # ceiling on model calls per prompt for this recipe

def rejection_sample(prompt, policy, grader, n=8, keep_fraction=0.25):
    """Generate n candidates for prompt, grade each, and
    keep the top keep_fraction by reward. This recipe is
    for small fixed batches — n is capped at
    MAX_CANDIDATES; a larger sampling budget needs a
    dedicated rollout pipeline with its own concurrency
    and cost controls, not this loop."""
    if n > MAX_CANDIDATES:
        raise ValueError(f"n={n} exceeds MAX_CANDIDATES={MAX_CANDIDATES}")
    candidates = [policy.generate(prompt) for _ in range(n)]
    graded = [(c, grader.score(prompt, c)) for c in candidates]
    graded.sort(key=lambda pair: pair[1], reverse=True)
    keep_n = max(1, int(len(graded) * keep_fraction))
    kept = graded[:keep_n]
    return [
        {"messages": [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": completion},
        ]}
        for completion, reward in kept
    ]

At keep_fraction=0.25 and n=8, two candidates per prompt survive into the SFT set — tune keep_fraction against the batch's reward distribution rather than a fixed count, since a harder prompt set shifts the whole distribution down.

5. Goldens-Holdout Check

Run this before any converted batch merges into the training set — per SKILL.md's Hygiene section, a golden ID leaking into training data silently inflates every later eval run against that same golden:

import json

def load_golden_ids(goldens_path):
    with open(goldens_path) as f:
        return {json.loads(line)["task_id"] for line in f}

def filter_holdout(candidate_rows, golden_ids):
    """candidate_rows: dicts still carrying task_id
    before provenance stripping. Returns only rows
    whose task_id never appears in the goldens."""
    kept, dropped = [], []
    for row in candidate_rows:
        if row["task_id"] in golden_ids:
            dropped.append(row)
        else:
            kept.append(row)
    return kept, dropped

Run filter_holdout before the messages-only stripping shown in recipe 1 — once task_id is gone, the check has nothing to match against. Log dropped rather than silently discarding it; a large dropped count usually means the trace collection step is resampling goldens instead of production traffic.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Seth Hobson

Publisher

Seth Hobson (Community)

Source

wshobson/agents · plugins/llm-finetuning/skills/trace-to-training-data · commit 4bde316

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Seth Hobson or Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download trace-to-training-data.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Seth Hobson

Trace To Training Data

The Principle

SFT From Traces

Preference Pairs From Traces

Hygiene

Related Skills

Files

references/conversion-recipes.md

Conversion Recipes

1. Graded Trace to SFT Example

2. Two Graded Traces to a DPO Pair

3. Correction Record to SFT Example

4. Rejection-Sampling Loop

5. Goldens-Holdout Check

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP