AutoGineer
Generated banner for Dataset Curation
Generated logo for Dataset Curation

Dataset Curation

Seth HobsonCommunityCreated by Seth Hobson

  • Skill
  • Workflow
  • Data and AI
  • MIT
  • No issues found
Download ZIP

From the author

Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured procedure for curating datasets for model training. It covers format selection based on the training method, mechanics for chat templates and loss masking, strategies for synthetic data generation, and the mandatory requirements for a dataset card. It serves as a gatekeeper for the training phase, ensuring that data is properly formatted, packed, and validated before a run begins.

What it does

  • Converts raw data into specific formats for SFT, DPO, ORPO, KTO, or GRPO training.
  • Applies chat templates to conversation data before sequence packing.
  • Configures sequence packing to reduce padding and improve compute efficiency.
  • Generates synthetic training data using methods like Magpie and rejection sampling.
  • Validates datasets against a 25% real-data floor to prevent model collapse.
  • Enforces the completion of a six-field dataset card required for training.

When to use it

  • When preparing raw demonstrations or preference judgments for a fine-tuning run.
  • When configuring datasets for multi-turn chat or preference optimization.
  • When integrating synthetic data into a training set.
  • When preparing to launch a training job that requires a validated dataset card.

Advantages

  • Prevents silent training failures by enforcing template and masking rules.
  • Optimizes compute usage through structured sequence packing.
  • Provides a clear checklist for dataset readiness before training.
  • Ensures provenance and consistency through mandatory dataset card fields.

Usage examples

“How do I format a multi-turn conversation dataset for SFT?”
The agent provides the ChatML messages list structure and explains the requirement to apply the chat template before packing.
“What are the requirements for a dataset card before I start training?”
The agent lists the six mandatory fields: Provenance, Counts, Synthetic/real ratio, Dedup method, Template used, and Packing config.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/wshobson/dataset-curation/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip dataset-curation.zip -d .agents/skills

Documentation by Seth Hobson

Dataset Curation

This skill assumes finetuning-method-selection already routed here — the next step is preparing data, not choosing a method. What follows: format selection by target method, the template/packing mechanics behind the most common silent training failures, rules for mixing in synthetic data without collapse, and the dataset card that closes out Phase 2 before a run starts.

Input: raw examples (demonstrations, preference judgments, or task prompts) plus a routing decision from finetuning-method-selection. Output format: a formatted, packed, validated JSONL dataset plus a completed dataset card — the Phase 2 artifact /finetune checks before launching training.

Format Selection

MethodShapeRows
SFT, single-turnInstruct (instruction/response or prompt/completion)~1,000+ floor
SFT, multi-turnConversation / ChatML messages list~1,000+ floor
DPO / ORPOPreference pair (prompt, chosen, rejected)Method-dependent, see preference-optimization
KTOUnpaired (prompt, completion, label)Method-dependent, see preference-optimization
GRPO / RLVRPrompt-only (prompt + verifier metadata)Method-dependent, see grpo-rlvr-training

  • ~1,000+ rows is the recommended floor for SFT, not a target. Below it, a handful of low-quality or duplicate examples can dominate the gradient; above it, quality over quantity — a smaller verified, deduplicated set beats a larger noisy one.

  • The ChatML shape, for orientation; the other four formats plus a ShareGPT conversion note live in references/formats-and-templates.md:

    {"messages": [
  {"role": "user", "content": "..."},
  {"role": "assistant", "content": "..."}
]}

Chat Templates and Loss Masking

Apply the target model's chat template before any concatenation or packing, never after — packing raw text and templating the packed blob afterward corrupts turn boundaries, landing role markers in the wrong place relative to each example.

  • Train on assistant responses only. Mask the loss (-100 in the labels tensor) over system/user turns and the template's own role markers — only assistant-turn content tokens contribute to loss.

  • Template/tokenizer mismatches are a top silent failure mode. A model trained against one chat template but served or evaluated with a different one degrades without erroring. Verify the same template string used in training is applied at inference and eval time.

  • Keep the dataset in messages shape and let the trainer template and mask it (assistant_only_loss=True in current TRL) — pre-rendering to a flat text field destroys the turn boundaries masking needs. Full code sketch: references/formats-and-templates.md. Sanity-check before training — decode only unmasked positions; expect only assistant text:

    keep = batch["labels"][0] != -100
print(tokenizer.decode(batch["input_ids"][0][keep]))

Packing

Without packing, 40–70% of compute is spent on padding — variable-length examples batched at a fixed sequence length waste the gap between each example's length and the batch's max. Packing concatenates multiple examples into one sequence up to the max length, cutting most of that waste.

  • Packing changes batch semantics. A packed sequence can contain several original examples, so "steps per epoch" and any LR schedule keyed to example count shift once packing is on — recompute schedule milestones against packed-sequence count.

  • MANDATORY: decode and manually inspect 5–10 packed sequences before scaling to a full run. Confirm example boundaries land where expected, template markers are intact per sub-example, and the loss mask is still assistant-only within each packed sequence. Not optional — packing bugs are silent (the loss curve looks normal) and only surface in eval quality, hours later:

    for seq in packed_dataset.select(range(10)):
    print(tokenizer.decode(seq["input_ids"]))

Synthetic Data Rules

  • Keep ≥25% real data as a collapse guard. Training on a growing share of model-generated data without a real-data floor drives measurable quality collapse over successive generations — 25% real is the minimum that holds the line. General-domain replay rows count toward this floor — "real" means "not generated for this task from this student," not "human-authored." An all-synthetic-by-construction dataset can meet the ≥25% floor through replay alone (see references/synthetic-data.md's Replay-Mix Construction recipe); state which rows count as "real" in the dataset card rather than leaving the floor structurally unmeetable.

  • Magpie and rejection sampling are the workhorses. Magpie extracts prompts from the model's own template prior; rejection sampling generates several candidates per prompt and keeps only the ones a filter passes. Both beat naive single-shot generation.

  • Targeted, student-aware generation beats static generation by 1.3–2x sample efficiency — aiming at the student's actual failure modes hits a quality bar with fewer filtered examples.

  • Typical accept rates after filtering run 10–30%. Plan volume accordingly — a 10,000-row target at 15% accept needs ~65,000+ raw generations.

  • Generation-method ranking, filter funnel, replay- mix construction, and distillation pattern: references/synthetic-data.md.

The Dataset Card

Every dataset that reaches training gets a card — the required Phase 2 artifact /finetune checks before launching. The card is not free-form documentation; it MUST carry these fields:

  • Provenance — where every row came from (real source(s), synthetic method(s), or both), traceable to trace-to-training-data output.

  • Counts — total rows, and rows per split (train/eval/held-out) if split.

  • Synthetic/real ratio — the measured ratio, checked against the ≥25% real floor above.

  • Dedup method — exact-match, semantic (embedding threshold), or both; see the filter funnel in references/synthetic-data.md.

  • Template used — the exact chat template string/identifier, kept consistent through inference and eval — this is what ties an eval-harness-first run back to the checkpoint.

  • Packing config — whether packing was used, max sequence length, and confirmation the 5–10-sequence manual inspection above was done.

A dataset missing any of these six fields isn't ready for /finetune — the card is a gate, not a summary written after the fact.

Phase 2 Exit Checklist

Before handing off to /finetune, confirm:

  1. Format matches the method (table above).

  2. Template applied before concatenation.

  3. Loss masked to assistant turns only.

  4. 5–10 packed sequences decoded and read.

  5. ≥25% real data in the final mix.

  6. Dataset card complete — all six fields.

References

  • references/formats-and-templates.md — JSONL examples per format, current-TRL masking code, and the ShareGPT conversion note.

  • references/synthetic-data.md — generation-method ranking, filter funnel, replay-mix construction, and teacher→student distillation pattern.

Related skills: finetuning-method-selection routes here; lora-qlora-recipes, vision-sft, and preference-optimization consume the datasets this skill produces; trace-to-training-data is the provenance source for graded-trajectory datasets; eval-harness-first grades the resulting checkpoint.

Files

references/synthetic-data.md

Last verified: 2026-07-14

Synthetic Data: Generation, Filtering, Distillation

Full detail backing SKILL.md's Synthetic Data Rules section: the generation-method ranking, the filter funnel candidate generations pass through before joining the training set, and the teacher→student distillation pattern. Base models are never named as recommendations here — TEACHER and STUDENT are placeholders for whichever checkpoints a given run uses; see finetuning-method-selection's references/model-catalog.md for actual model choice.

Generation-Method Ranking

Methods below are ordered roughly weakest to strongest for sample efficiency and downstream quality at the same generation budget. Each level subsumes the previous — a rejection-sampling pipeline typically generates its candidates with Magpie or persona-conditioned prompts underneath, rather than replacing them:

  1. Self-Instruct — bootstrap new prompts from a small seed set by having a model paraphrase and extend them. Cheapest, weakest: prompt diversity plateaus quickly and quality tracks the seed set closely.

  2. Evol-Instruct — iteratively rewrite prompts to increase complexity (add constraints, deepen reasoning, broaden scope) across generations. Improves difficulty coverage over Self-Instruct but still seed-dependent.

  3. Magpie — extract prompts directly from the target model's own chat-template prior by sampling from the template's user-turn position with no seed prompt at all. Removes seed-set bias entirely; this is why it's a workhorse rather than a niche technique.

  4. Persona-conditioned generation — condition prompt generation on a sampled persona/role description to broaden style and topic coverage beyond what a single generation policy produces unconditioned.

  5. Rejection sampling / verifier-filtered — generate multiple candidate completions per prompt and keep only those a filter, verifier, or judge accepts. This is the other workhorse from SKILL.md, and it composes with any of the prompt-generation methods above — it's a completion-side filter, not a prompt-generation method by itself.

Targeted, student-aware generation — steering prompt or persona selection toward the current student model's actual failure modes rather than sampling uniformly — layers on top of any method above and is what delivers the 1.3–2x sample efficiency gain cited in SKILL.md. It requires an eval signal on the student to know what its failure modes currently are; without that signal, generation defaults to untargeted/static.

Filter Funnel

Apply filters in this order — each stage is cheaper than the next, so cheap stages should eliminate volume before expensive stages run on what's left:

  1. Exact dedup. Hash-based exact-match removal of identical rows (after normalization — whitespace/casing collapsed before hashing). Cheapest stage, run first, removes generation-loop repeats before anything downstream sees them.

  2. Semantic dedup (~0.92 similarity threshold). Embed each candidate and drop rows whose nearest-neighbor cosine similarity to an already-kept row exceeds ~0.92. Catches paraphrase-level duplicates exact dedup misses.

  3. Length filter. Drop candidates below a minimum or above a maximum token length for the task — too-short responses are usually degenerate, too-long ones are usually rambling or off-task.

  4. Language ID filter. Drop candidates that fail a language-ID check against the target language(s) — generation occasionally drifts language, especially from multilingual base models on under-specified prompts.

  5. Score-based top-30% filter. Score remaining candidates (reward model, heuristic, or self-consistency score) and keep roughly the top 30% — this is a coarse quality cut before the most expensive stage runs.

  6. Judge ≥ threshold. Run the most expensive check last, on the smallest remaining set: an LLM-judge or human-equivalent quality check against a fixed threshold. Rows that fail here are dropped regardless of how they scored upstream.

The 10–30% typical accept rate from SKILL.md is the funnel's end-to-end yield across all six stages, not any single stage's pass rate — budget raw generation volume against the full-funnel yield, not against any one stage's rate.

Teacher→Student Distillation Pattern

  1. Generate teacher traces. Sample completions (with reasoning traces, where applicable) from TEACHER against the target task's prompt distribution.

  2. Verify. Run the traces through the Filter Funnel above — a distillation set is a synthetic dataset like any other and still needs the ≥25% real-data floor from SKILL.md respected in the final training mix, plus the same dedup and quality stages.

  3. SFT the student. Train STUDENT on the verified traces using the standard SFT format and template rules from SKILL.md and references/formats-and-templates.md — a distillation dataset is not a special format, it's a provenance label on an otherwise-ordinary instruct or ChatML dataset.

Record TEACHER identity and generation configuration (sampling temperature, prompt template used to elicit traces) in the dataset card's provenance field — "distilled from TEACHER" is a provenance fact /finetune and downstream audits both expect to find there, not something to leave implicit.

Replay-Mix Construction

The implementation recipe behind checkpoint-promotion's catastrophic-forgetting escalation ladder (SKILL.md's owning document for when and how far to move the replay fraction — this section covers how to build the rows, the single most common REJECT remediation and the part most often improvised ad hoc under time pressure). Five decisions, in the order they come up:

1. General-Domain Source Selection

Pick a source that is genuinely general-domain for the capability being protected, not a narrow slice that happens to be convenient. Two failure modes to avoid:

  • Don't teach to the gate. If the replay source is drawn from the exact same distribution as the drift suite's benchmarks (e.g. the same GSM8K train split the drift suite's test split comes from), the resulting drift score partially measures "did this model see similar items in training," not "did fine-tuning preserve the underlying capability." This is not automatically disallowed — see checkpoint-promotion's instruction-reuse disclosure rule — but it must be disclosed, and a broader source (not scoped to the drift suite's own benchmarks) is the more defensible default when one exists.

  • Match the source to the forgetting signature. If error analysis on the failing checkpoint shows a specific lost capability (e.g. chain-of-thought math reasoning, not general knowledge), a replay source targeting that capability recovers it faster than a generic instruct-tuning mix — but narrows the "general-domain" claim; state in the dataset card which capability the replay mix targets and why.

2. Prompt Shape

Decide what shape replay rows take in the messages-shaped SFT set — this is a real choice, not a detail:

  • Natural instruction — however the source data's own prompts are phrased. Lowest effort, least targeted.

  • The drift harness's exact phrasing — matches the eval's instruction wording. Most directly addresses an instruction-following forgetting signature (e.g. "ignores the show-your-work instruction"), but triggers the instruction-reuse disclosure rule in checkpoint-promotion and inflates the post-replay score on that specific benchmark.

  • Bare input, no instruction wrapper — closest to raw continued-pretraining signal; weakest at restoring instruction-following specifically.

Pick based on the forgetting signature from error analysis, not by default — and disclose the choice in the dataset card regardless of which one.

3. Answer Reformatting

Decide whether replay reference answers get reformatted toward the target task's output convention, or kept in the source format as-is. Example: rewriting a math dataset's #### N final-answer terminator to match the target task's own extraction convention. This is a judgment call that changes what the model learns to emit on replay-domain prompts — record the exact transformation applied (or "none — kept source format") in the dataset card, since it changes what a downstream error-analysis pass should expect to see.

4. Val-Split Treatment

Decide whether the validation split gains replay rows or stays task-only:

  • Task-only val split keeps eval_loss directly comparable across runs that only differ in replay fraction — the training loop has zero visibility into replay fit, and replay recovery is only measurable at the next Phase 5 re-gate.

  • Replay rows in val too gives in-loop visibility into replay fit, at the cost of eval_loss no longer being an apples-to-apples comparison against a prior run's task-only val split.

Neither is universally correct; state which was chosen and why in the dataset card, and don't compare eval_loss across runs that made different choices here without noting the confound.

5. Disjointness Verification

Before training, verify replay rows don't overlap the drift suite or the goldens set — required, not optional, regardless of which source was picked in step 1:

  • Split-level separation — draw replay rows only from a source split (e.g. a train split) disjoint from whatever split the drift suite's items are drawn from.

  • Exact-match text filter — normalize and exact-match replay row question/prompt text against the drift suite's selected items and eval/goldens.jsonl; drop any hit. Record the overlap count found (expect 0) in the dataset card — a nonzero count found and silently dropped is still worth recording, since it signals the source pool needs a tighter split boundary next time.

When a Later Run Changes the Replay Fraction: Swap, Don't Add

If a checkpoint-promotion gate calls for moving the replay fraction (see checkpoint-promotion's escalation ladder), implement the change by swapping rows, not adding them: drop target-task rows out of the training set as replay rows go in, so the total row/step count holds constant between the old and new run. Adding replay rows on top of the existing set changes replay fraction and total optimizer steps in the same move, making it impossible to attribute a later drift-score change to either variable alone — this confound has produced misleading run-to-run trajectories in practice, so treat swap-not-add as a hard rule for this recipe, not a style preference.

Row count is not token count. Swapping rows 1-for-1 holds the row count constant, but replay rows and target-task rows are rarely the same length — a swap can still shift total training tokens (and therefore max_steps under a fixed batch size and sequence-packing scheme) even though the row count didn't move. Hold total training tokens, or max_steps directly, constant between the old and new run — not just row count — and record the packed-token count for each run (not just the row count) in the dataset card before attributing a drift-score change to the replay-fraction change alone. A run that swapped rows but grew packed tokens 10% has the same attribution problem as one that added rows outright.

Synthetic-Only Datasets and the ≥25% Real Floor

SKILL.md's Synthetic Data Rules require ≥25% real data as a collapse guard. When a training set is 100% synthetic by construction (a greenfield task with no real-data pool to draw from at all — not merely a lot of synthetic augmentation on top of a real base), that floor is unmeetable by definition unless something in the mix counts as "real."

Resolution: general-domain replay rows count toward the ≥25% floor. "Real" in this rule means "not generated for this specific task from this specific student model" — a replay row pulled from an existing general-instruct dataset (human-authored or otherwise pre-existing, not freshly generated by the student or its teacher for this run) satisfies that definition even though the target-task rows around it are 100% synthetic. Build the replay mix per the five decisions above, then compute the synthetic/real ratio the dataset card requires treating replay rows as the "real" share — and state explicitly in the card that this is how the ratio was met, so a later audit doesn't misread an all-synthetic-target-data run as having silently skipped the collapse guard.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Seth Hobson
Publisher
Seth Hobson (Community)
Source
wshobson/agents · plugins/llm-finetuning/skills/dataset-curation · commit 4bde316
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Seth Hobson or Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download dataset-curation.zip