Nemotron Asr Finetune

NVIDIAOfficialCreated by Nemotron Speech TeamDistributed by NVIDIA

Skill
Workflow
Data and AI
Apache-2.0
Medium risk
Has scripts

From the author

Orchestration skill for NVIDIA Nemotron Speech (Riva) / NeMo ASR domain and language adaptation. Given a goal like "improve/fine-tune ASR for my domain or language", it scopes the task, picks the cheapest sufficient path (word boosting → n-gram LM → fine-tuning), delegates each stage to the right sub-skill (data generation, training, evaluation, deployment), and answers cost/time/data questions along the way.

— the description by Nemotron Speech Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill acts as an intelligent orchestrator for adapting NVIDIA Nemotron Speech (Riva) ASR models to specific domains or languages. Rather than jumping straight to training, it guides users through a cost-effective ladder of customization techniques, starting with word boosting and n-gram language models before escalating to full acoustic fine-tuning. It manages the end-to-end lifecycle by scoping requirements, delegating tasks to specialized sub-skills for data generation, training, evaluation, and deployment, and providing data-driven estimates for time, cost, and hardware requirements.

What it does

Scopes ASR customization goals based on domain, language, and performance targets.
Identifies the cheapest sufficient path for improvement, from word boosting to full fine-tuning.
Delegates execution to specialized sub-skills for data, training, evaluation, and deployment.
Performs pre-flight checks for dataset quality and environment compatibility.
Provides estimates for data volume, GPU requirements, and expected costs.
Maintains a structured workflow to ensure efficient, measurable model adaptation.

When to use it

When you need to improve ASR accuracy for a specific industry domain or jargon.
When you need to adapt an ASR model to a new language.
When you want to reduce Word Error Rate (WER) in noisy environments like call centers.
When you need to plan a cost-effective ASR customization strategy.
When you are unsure whether to use word boosting, language models, or full fine-tuning.

Advantages

Prevents unnecessary, expensive training by prioritizing cheaper, effective methods first.
Ensures a structured, repeatable approach to model adaptation.
Provides clear guidance on hardware selection and data requirements.
Reduces risk of catastrophic forgetting by enforcing data volume and mixing best practices.
Centralizes planning while leveraging specialized sub-skills for execution.

Usage examples

“I need to improve ASR accuracy for my medical domain, it gets jargon wrong.”: The agent will scope the jargon issues, suggest a word-boosting pilot to test offline, and outline the data requirements for potential fine-tuning if boosting is insufficient.
“How much data do I need to fine-tune an ASR model for a new language?”: The agent will explain the requirements for cross-language transfer versus training from scratch, emphasizing the need for real audio and providing a data-volume estimate.
“What is the cheapest way to improve my ASR model's performance?”: The agent will initiate a diagnostic process to measure the baseline WER and recommend the lowest-cost rung on the customization ladder, such as word boosting or n-gram LM.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/nemotron-asr-finetune/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip nemotron-asr-finetune.zip -d .agents/skills

Documentation by Nemotron Speech Team

Nemotron Speech ASR Customization — Orchestration Skill

Note: "Nemotron Speech" is the public-facing name for what NVIDIA documents today as Riva / Riva NIM; the acoustic models are trained and fine-tuned with NVIDIA NeMo. Commands, config paths, imports, and doc URLs still use "Riva" / "NeMo" — the rename is brand-only. Do not rename them.

What This Skill Is

This is a high-level orchestration skill, not a step-by-step training manual. Its job, given a goal such as "I want to fine-tune ASR for my domain/language", is to:

Scope the problem (how much real audio, target eval set, latency/hardware budget, language/domain).
Choose the cheapest sufficient path — word boosting, n-gram LM fusion, or fine-tuning — and escalate only when quality falls short.
Delegate each stage to the right sub-skill (data generation, training, evaluation, deployment/optimization).
Answer cost/time/data questions along the way (how many hours to hit X% WER, synthetic vs real, L40S vs H100, expected cost).

It owns the plan and the routing; the sub-skills own the execution. When a needed sub-skill does not exist yet, this skill names it as a placeholder and gives interim guidance.

When to Use

Use for any request to make a Nemotron Speech / Riva ASR model work better on a specific domain or language — improving accuracy, reducing WER, adding a language, or planning a fine-tune. Start here even when the user names a specific technique: treat it as a candidate until the orchestration step confirms it is the cheapest sufficient path, then sequence the right sub-skills.

Orchestration Workflow

Run the loop below; each stage names the sub-skill it invokes. Full detail in references/workflow.md.

#	Stage	What happens	Sub-skill
1	State the goal	Capture the target: domain/language, the errors, the metric.	Orchestration (this skill)
2	Clarify & scope	Ask the discovery questions: how much real audio? target eval set? latency/HW budget? deployment target?	Orchestration
3	Choose the path	Pick the cheapest sufficient rung (boosting → n-gram LM → fine-tune). Escalate only if quality is short; experiment while proposing the full plan.	Orchestration → Research/Training
4	Get the data right	If data is scarce/noisy: synthetic (TTS), TTS-friendly formatting, noise profiling/harvest, blend, score vendor samples; align customer data to training format; flag missing real data.	SDG / Data
5	Train	Apply the recipe (configs, hyperparameters, replay/curriculum, GPU/OOM preflight) and run.	Research / Training
6	Evaluate	Normalized WER on the domain set + A/B forgetting check on a general set; error-driven analysis to find the next lever.	Evaluation
7	Loop or ship	If short of target, loop to 4/5 with targeted data; else select/average checkpoints. Consult the user before more cycles.	Orchestration
8	Deploy	Export to NIM/HF, hot-swap the checkpoint, serve.	Deployment / Optimization

Stages 4–8 are the fine-tune path (data → NeMo train → NeMo eval → Riva deploy). Cheaper rungs (boosting, custom vocab, n-gram LM) take a shorter branch owned by a single sub-skill — don't force them through the full loop. See the branch-by-rung table in references/workflow.md (§3b).

Before training (Stage 5), run the pre-flight dataset quality check (§4a): verify/convert audio to 16 kHz, and, on user request, audit transcript quality by running a reference pretrained model and checking WER against the provided ground truth. See references/workflow.md §4a.

Also before training (Stage 5), run the pre-flight environment/dependency check (§4b): if no NeMo is provided, pull the latest main (local execution) or the latest published container tag (container execution). If a NeMo checkout/install is already provided, check its version, but still recommend switching to latest main (staleness risk) and ask the user — if they insist on the provided one, proceed with it and only revisit once a concrete version issue (e.g. unsupported functionality) is actually hit. See references/workflow.md §4b.

Throughout, answer the "along the way" questions (data volume, synthetic vs real, hours to reach a WER target, cost, GPU choice) — see references/planning-answers.md.

Sub-Skills This Skill Calls

Detailed registry, invocation, and handoff contracts in references/sub-skills.md.

Role (per the architecture)	Purpose	Sub-skill to invoke
Research / Training	NeMo configs, recipes, fine-tuning, checkpoint averaging; also owns the NeMo-side word-boosting and n-gram LM pilots	`nemo-speech-asr-finetune`
SDG / Data Designer	Synthetic transcripts/text, noise profiling, vendor-data impact, blends	`data-designer` (synthetic text; audio via TTS in `nemotron-speech`); placeholder: `asr-data-profiling`
Evaluation	Normalized WER, A/B forgetting, error analysis	Offline file WER → `nemo-speech-asr-finetune`; served-endpoint WER → `nemotron-speech`
Deployment / Optimization	NIM/Riva export, checkpoint swap, NIM-build optimization, serving	`nemotron-speech`

If a sub-skill is unavailable, say so, give the interim guidance from the reference, and continue the plan.

Choosing The Path (cheapest first)

The scoping in Stage 3 selects the lowest-cost rung that can meet the target. Summary; full docs-grounded ladder in references/path-selection.md.

Word boosting — a bounded set of known words/names/jargon. Two realizations that are different artifacts: pilot (NeMo) to prove lift offline via GPU-PB context biasing (nemo-speech-asr-finetune), or deploy (Riva) to ship it via runtime boosted_lm_score (nemotron-speech) — different score systems, don't reuse one for the other. Runtime, no training either way. See references/path-selection.md.
Custom vocabulary / pronunciation — OOV or consistently mispronounced terms. Deploy-time. → Deployment sub-skill.
N-gram (KenLM) LM — domain phrasing/word-sequences when you have text but little audio. Two realizations, built the same way but deployed differently by architecture: pilot (NeMo, NGPU-LM) to prove lift offline (nemo-speech-asr-finetune), or deploy (Riva) to ship it (nemotron-speech). For CTC, rebuild the pilot corpus into a Riva word-level LM — don't ship the pilot artifact as-is. For RNN-T/TDT, the opposite: hand the pilot's .nemo artifact to Riva unchanged, no rebuild. See references/path-selection.md.
Fine-tune — real acoustic gaps (accents, noise, channel) with enough transcribed audio (NIM guide: 100+ h; ~10 h floor only if mixed to avoid catastrophic forgetting). Below ~10 h, do not recommend fine-tuning — recommend word boosting instead (severe catastrophic-forgetting/overfitting risk). → Research/Training.
Train from scratch / cross-language transfer — a new language with no suitable checkpoint (last resort). → Research/Training.

Ordering and per-model support follow the NVIDIA Speech NIM ASR customization guide: https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html.

Key Principles

Scope before you pick. Don't recommend fine-tuning before the discovery questions and a measured baseline.
Cheapest sufficient path. Escalate rungs only when the current one provably can't hit the target; you may experiment on a cheap rung while presenting the full fine-tuning plan.
Measure with a contract. Report normalized WER on the domain set plus an A/B forgetting check on a general set — never in-training logs alone.
Verify before you train. Run the pre-flight dataset quality check before Stage 5: verify/convert audio to 16 kHz, and, when the user asks, audit ground-truth transcripts by comparing them to a reference model's WER. Don't train on an unresampled or unaudited-on-request dataset. See references/workflow.md §4a.
Know your environment before you train. If no NeMo is provided, pull the latest main (or latest published container tag). If one is provided, check its version but still recommend latest main and ask the user first — a provided checkout always carries staleness risk. If they insist on keeping it, don't pull preemptively; only revisit once a concrete version issue (e.g. unsupported functionality) is actually hit. Record what was resolved in the ledger. See references/workflow.md §4b.
Delegate, don't reimplement. Route execution to the sub-skills; keep this skill focused on the plan, sequencing, and cost/time/data answers.
Real target-domain audio is the usual bottleneck. Prefer real data; use synthetic to fill measured gaps, kept separately weighted so it can be ablated.
Consult the user before extra tuning cycles, and when a needed sub-skill is a placeholder.

Source of Truth

Topic	Location
NIM Speech docs home	https://docs.nvidia.com/nim/speech/latest/index.html
ASR customization guide (methods, per-model support)	https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html
ASR support matrix (models & features)	https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html
NeMo fine-tuning (flags/config)	`docs/source/asr/fine_tuning.rst`, and the `nemo-speech-asr-finetune` sub-skill
Riva ASR tutorials (boosting, LM, fine-tune)	https://github.com/nvidia-riva/tutorials
Tokenizer extension to new language + acoustic fine-tune	https://github.com/nvidia-riva/tutorials/blob/main/asr-extend-tokenizer-to-newlang-ft-acoustic-model.ipynb

Limitations

Orchestration only — execution happens in the sub-skills. Where a sub-skill is a placeholder, guidance is interim until it exists.
GPU required for the training rungs; deployment/serving is owned by the nemotron-speech sub-skill.
Model names, config paths, flags, and per-model feature support drift across NeMo/Riva releases — verify against the support matrix and the current checkout.
Public branding is "Nemotron Speech"; commands, imports, config paths, and doc URLs still use "Riva" / "NeMo" — do not rename.

Files

ATTRIBUTION.mdAdded by AutoGineer935 B
BENCHMARK.md10.7 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md11.6 KiB
assets/experiment-ledger-template.md3.5 KiB
evals/EVAL.md8.7 KiB
evals/evals.json20.4 KiB
references/path-selection.md14.8 KiB
references/planning-answers.md3.3 KiB
references/sub-skills.md10.8 KiB
references/workflow.md19.9 KiB
scripts/link-nemo-subskill.shScript3.6 KiB
scripts/main.pyScript10.9 KiB
skill-card.md4.9 KiB
skill.oms.sig6.3 KiB

evals/EVAL.md

Nemotron ASR Orchestration Eval Guidance

Use evals/evals.json to verify activation, scoping, path selection, and sub-skill delegation for the orchestration skill nemotron-asr-finetune.

What to grade

The skill should activate for any goal to make a Nemotron Speech / Riva ASR model work better on a domain or language: improving accuracy, reducing WER, adding a language, or planning a fine-tune.
It is an orchestration skill. The central behaviors are:
1. Scope first — run the discovery questions (real audio hours, domain text, eval set/target, latency/HW budget, deployment target) before recommending a technique.
2. Choose the cheapest sufficient path — word boosting → custom vocab/pronunciation / n-gram LM → fine-tune → (last resort) from scratch — and escalate only when the target is missed. Hard floor: below ~10 hours of real transcribed target-domain audio, do not recommend fine-tuning at all — catastrophic forgetting and overfitting risk are both severe at that volume — recommend word boosting instead.
  - Word boosting has a NeMo pilot and a Riva deploy realization — different artifacts with different, non-interchangeable score systems. Pilot: GPU-PB context biasing via the boosting_tree config (CTC: ctc_decoding.*.boosting_tree; RNN-T/TDT: rnnt_decoding.*.boosting_tree, both through speech_to_text_eval.py, weighted by boosting_tree_alpha) — owned by nemo-speech-asr-finetune, entirely self-contained, no Riva required. Deploy: Riva's per-request boosted_lm_score — owned by nemotron-speech. Do not conflate the two score systems, and do not force a customer working the NeMo pilot to also touch Riva.
  - The n-gram LM also has a NeMo pilot (NGPU-LM) and a Riva deploy realization, but carry-over to Riva is architecture-dependent, not a blanket rule. Pilot: build with train_kenlm.py (save_nemo=True), score via speech_to_text_eval.py's ctc_decoding.*.ngram_lm_model/rnnt_decoding.*.ngram_lm_model — CTC also has a second, older NeMo-side mechanism (its own Flashlight decoder, word-level KenLM + lexicon). Deploy — CTC needs a rebuild into a word-level LM for Riva's Flashlight decoder. Deploy — RNN-T/TDT reuses the pilot .nemo artifact unchanged — verified against NVIDIA's own RNNT tutorial. Exact riva-build flags/syntax for either case are owned by nemotron-speech (references/pipelines.md) — the orchestrator should state the routing/carry-over fact and defer to that skill for flags, not recite them from its own memory. Do not apply the CTC rebuild rule to RNNT, and do not assume the RNNT exception applies to CTC.
3. Delegate to sub-skills — data (data-designer for synthetic text; TTS audio via nemotron-speech if needed), training (nemo-speech-asr-finetune), evaluation (nemo-speech-asr-finetune's eval stage), deployment (nemotron-speech). The remaining placeholder is asr-data-profiling (name it as such with interim guidance). Do not route ASR work to LLM-only skills (nemotron-customize) or to the platform eval CLI (nemo-evaluator-plugin).
4. Answer cost/time/data questions with ranges + assumptions, preferring a measured pilot over a fabricated hours-to-WER number.
5. Gate on data quality before training (Stage 4a) — verify/convert audio to 16 kHz offline before handing a manifest to Stage 5, and, when the user asks, audit transcript quality by running a reference pretrained checkpoint (not the fine-tune target) and comparing its WER against the provided ground truth, surfacing flagged utterances rather than dropping them silently. If the user insists on feeding 8 kHz files as-is instead of offline-converting, confirm that choice explicitly (it is not recommended — online resampling during training is slow) and, once confirmed, keep model.sample_rate at the checkpoint's expected rate (typically 16000) so NeMo's loader upsamples online; do not set model.sample_rate=8000, which trains a different, 8 kHz-native model rather than fine-tuning the target checkpoint.
6. Gate on the training environment before training (Stage 4b) — three-way branch:
  - Nothing provided/found: pull the latest main from the current canonical repo (NVIDIA-NeMo/Speech) for local execution, or the latest published container tag for container execution. Mandatory, not optional.
  - Something is provided: check its version (nemo.__version__, git commit/branch for a source checkout), but still recommend switching to latest main (staleness risk — a provided checkout can predate a feature the chosen technique needs) and ask the user rather than silently picking either side.
  - User insists on keeping the provided checkout: proceed with it, do not pull preemptively. Only recommend/pull latest main again if a concrete version-specific issue is actually hit later (e.g. an unsupported config key or missing feature) — treat that as new evidence, not as overriding the user's choice. Record what was resolved (and any later escalation) in the ledger.
7. Make nemo-speech-asr-finetune reachable before Stage 5 — it is project-local (lives inside the NeMo/Speech repo's own .claude/skills/), not catalog-published, so it is not invocable just because it was named. Make it discoverable (try npx skills add NVIDIA-NeMo/Speech --skill nemo-speech-asr-finetune ..., fall back to scripts/link-nemo-subskill.sh <checkout_path>), then change the actual working context (cd, or a scoped sub-agent) into that checkout before running any of its commands — they are paths relative to the repo root, and linking alone only makes the skill discoverable, not runnable.
Evaluation contract: normalized WER on the domain set plus an A/B forgetting check on a general set; error-driven analysis to choose the next lever; consult the user before extra tuning cycles.
Grade down: recommending fine-tuning for a problem boosting/LM would solve; failing to escalate when the residual error is acoustic; skipping scoping; reimplementing sub-skill execution instead of delegating; training on audio that was never verified as 16 kHz; silently dropping/ignoring flagged bad-transcript utterances instead of surfacing them to the user; for a user who wants to skip offline conversion of 8kHz audio, either silently offline-converting without asking or setting model.sample_rate=8000 (which changes the model, not just the input pipeline) instead of keeping model.sample_rate at the checkpoint's expected rate; or recommending/proceeding with fine-tuning when real transcribed target-domain audio is under ~10 hours without flagging the severe catastrophic-forgetting/overfitting risk and recommending word boosting instead; proceeding to Stage 5 without resolving and recording a NeMo version (silently assuming one exists, or silently training against an unverified/ stale checkout instead of pulling latest main or checking the provided one); silently accepting a provided checkout without recommending latest main and asking the user; or, once the user has explicitly chosen to keep a provided checkout, second-guessing that choice / pulling latest main anyway without a concrete version issue having actually surfaced; assuming nemo-speech-asr-finetune is invocable at Stage 5 without first making it discoverable and changing the working context into its NeMo checkout; for a NeMo-only word-boosting request, routing to nemotron-speech/Riva instead of the NeMo pilot, or presenting NeMo's context_score/ boosting_tree_alpha as if it were Riva's boosted_lm_score (or vice versa); or, for the n-gram LM, applying a single blanket rule across CTC and RNNT instead of checking which architecture applies — e.g. telling an RNNT user to rebuild a word-level KenLM/decoding_vocab they don't need, or telling a CTC user their pilot artifact ships to Riva unchanged when it doesn't.
Positive cases should load SKILL.md and the relevant reference (workflow.md, path-selection.md, sub-skills.md, or planning-answers.md). scripts/main.py is harness-only.
Negative cases stay silent for pure export/deployment of an existing model (defer to nemotron-speech), OpenAI Whisper, and text-LLM fine-tuning.

Harness-only script

scripts/main.py exists only because the evaluation harness requires a script entry point. It is a deterministic prompt-to-reference router whose default is the orchestration workflow. It does not invoke sub-skills and is not part of the agent-facing workflow; do not use it as grading evidence for positive cases.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Nemotron Speech Team

Publisher

NVIDIA (Official)

Distributed by

NVIDIA

Source

NVIDIA/skills · skills/nemotron-asr-finetune · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Nemotron Speech Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemotron-asr-finetune.zip