Nemotron Asr Finetune

NVIDIAOfficialCreated by Nemotron Speech TeamDistributed by NVIDIA

Skill
Workflow
Data and AI
Apache-2.0
Medium risk
Has scripts

From the author

Orchestration skill for NVIDIA Nemotron Speech (Riva) / NeMo ASR domain and language adaptation. Given a goal like "improve/fine-tune ASR for my domain or language", it scopes the task, picks the cheapest sufficient path (word boosting → n-gram LM → fine-tuning), delegates each stage to the right sub-skill (data generation, training, evaluation, deployment), and answers cost/time/data questions along the way.

— the description by Nemotron Speech Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill acts as an intelligent orchestrator for adapting NVIDIA Nemotron Speech (Riva) ASR models to specific domains or languages. Rather than jumping straight to training, it guides users through a cost-effective ladder of customization techniques, starting with word boosting and n-gram language models before escalating to full acoustic fine-tuning. It manages the end-to-end lifecycle by scoping requirements, delegating tasks to specialized sub-skills for data generation, training, evaluation, and deployment, and providing data-driven estimates for time, cost, and hardware requirements.

What it does

Scopes ASR customization goals based on domain, language, and performance targets.
Identifies the cheapest sufficient path for improvement, from word boosting to full fine-tuning.
Delegates execution to specialized sub-skills for data, training, evaluation, and deployment.
Performs pre-flight checks for dataset quality and environment compatibility.
Provides estimates for data volume, GPU requirements, and expected costs.
Maintains a structured workflow to ensure efficient, measurable model adaptation.

When to use it

When you need to improve ASR accuracy for a specific industry domain or jargon.
When you need to adapt an ASR model to a new language.
When you want to reduce Word Error Rate (WER) in noisy environments like call centers.
When you need to plan a cost-effective ASR customization strategy.
When you are unsure whether to use word boosting, language models, or full fine-tuning.

Advantages

Prevents unnecessary, expensive training by prioritizing cheaper, effective methods first.
Ensures a structured, repeatable approach to model adaptation.
Provides clear guidance on hardware selection and data requirements.
Reduces risk of catastrophic forgetting by enforcing data volume and mixing best practices.
Centralizes planning while leveraging specialized sub-skills for execution.

Usage examples

“I need to improve ASR accuracy for my medical domain, it gets jargon wrong.”: The agent will scope the jargon issues, suggest a word-boosting pilot to test offline, and outline the data requirements for potential fine-tuning if boosting is insufficient.
“How much data do I need to fine-tune an ASR model for a new language?”: The agent will explain the requirements for cross-language transfer versus training from scratch, emphasizing the need for real audio and providing a data-volume estimate.
“What is the cheapest way to improve my ASR model's performance?”: The agent will initiate a diagnostic process to measure the baseline WER and recommend the lowest-cost rung on the customization ladder, such as word boosting or n-gram LM.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/nemotron-asr-finetune/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip nemotron-asr-finetune.zip -d .agents/skills

Documentation by Nemotron Speech Team

Nemotron Speech ASR Customization — Orchestration Skill

Note: "Nemotron Speech" is the public-facing name for what NVIDIA documents today as Riva / Riva NIM; the acoustic models are trained and fine-tuned with NVIDIA NeMo. Commands, config paths, imports, and doc URLs still use "Riva" / "NeMo" — the rename is brand-only. Do not rename them.

What This Skill Is

This is a high-level orchestration skill, not a step-by-step training manual. Its job, given a goal such as "I want to fine-tune ASR for my domain/language", is to:

Scope the problem (how much real audio, target eval set, latency/hardware budget, language/domain).
Choose the cheapest sufficient path — word boosting, n-gram LM fusion, or fine-tuning — and escalate only when quality falls short.
Delegate each stage to the right sub-skill (data generation, training, evaluation, deployment/optimization).
Answer cost/time/data questions along the way (how many hours to hit X% WER, synthetic vs real, L40S vs H100, expected cost).

It owns the plan and the routing; the sub-skills own the execution. When a needed sub-skill does not exist yet, this skill names it as a placeholder and gives interim guidance.

When to Use

Use for any request to make a Nemotron Speech / Riva ASR model work better on a specific domain or language — improving accuracy, reducing WER, adding a language, or planning a fine-tune. Start here even when the user names a specific technique: treat it as a candidate until the orchestration step confirms it is the cheapest sufficient path, then sequence the right sub-skills.

Orchestration Workflow

Run the loop below; each stage names the sub-skill it invokes. Full detail in references/workflow.md.

#	Stage	What happens	Sub-skill
1	State the goal	Capture the target: domain/language, the errors, the metric.	Orchestration (this skill)
2	Clarify & scope	Ask the discovery questions: how much real audio? target eval set? latency/HW budget? deployment target?	Orchestration
3	Choose the path	Pick the cheapest sufficient rung (boosting → n-gram LM → fine-tune). Escalate only if quality is short; experiment while proposing the full plan.	Orchestration → Research/Training
4	Get the data right	If data is scarce/noisy: synthetic (TTS), TTS-friendly formatting, noise profiling/harvest, blend, score vendor samples; align customer data to training format; flag missing real data.	SDG / Data
5	Train	Apply the recipe (configs, hyperparameters, replay/curriculum, GPU/OOM preflight) and run.	Research / Training
6	Evaluate	Normalized WER on the domain set + A/B forgetting check on a general set; error-driven analysis to find the next lever.	Evaluation
7	Loop or ship	If short of target, loop to 4/5 with targeted data; else select/average checkpoints. Consult the user before more cycles.	Orchestration
8	Deploy	Export to NIM/HF, hot-swap the checkpoint, serve.	Deployment / Optimization

Stages 4–8 are the fine-tune path (data → NeMo train → NeMo eval → Riva deploy). Cheaper rungs (boosting, custom vocab, n-gram LM) take a shorter branch owned by a single sub-skill — don't force them through the full loop. See the branch-by-rung table in references/workflow.md (§3b).

Before training (Stage 5), run the pre-flight dataset quality check (§4a): verify/convert audio to 16 kHz, and, on user request, audit transcript quality by running a reference pretrained model and checking WER against the provided ground truth. See references/workflow.md §4a.

Also before training (Stage 5), run the pre-flight environment/dependency check (§4b): if no NeMo is provided, pull the latest main (local execution) or the latest published container tag (container execution). If a NeMo checkout/install is already provided, check its version, but still recommend switching to latest main (staleness risk) and ask the user — if they insist on the provided one, proceed with it and only revisit once a concrete version issue (e.g. unsupported functionality) is actually hit. See references/workflow.md §4b.

Throughout, answer the "along the way" questions (data volume, synthetic vs real, hours to reach a WER target, cost, GPU choice) — see references/planning-answers.md.

Sub-Skills This Skill Calls

Detailed registry, invocation, and handoff contracts in references/sub-skills.md.

Role (per the architecture)	Purpose	Sub-skill to invoke
Research / Training	NeMo configs, recipes, fine-tuning, checkpoint averaging; also owns the NeMo-side word-boosting and n-gram LM pilots	`nemo-speech-asr-finetune`
SDG / Data Designer	Synthetic transcripts/text, noise profiling, vendor-data impact, blends	`data-designer` (synthetic text; audio via TTS in `nemotron-speech`); placeholder: `asr-data-profiling`
Evaluation	Normalized WER, A/B forgetting, error analysis	Offline file WER → `nemo-speech-asr-finetune`; served-endpoint WER → `nemotron-speech`
Deployment / Optimization	NIM/Riva export, checkpoint swap, NIM-build optimization, serving	`nemotron-speech`

If a sub-skill is unavailable, say so, give the interim guidance from the reference, and continue the plan.

Choosing The Path (cheapest first)

The scoping in Stage 3 selects the lowest-cost rung that can meet the target. Summary; full docs-grounded ladder in references/path-selection.md.

Word boosting — a bounded set of known words/names/jargon. Two realizations that are different artifacts: pilot (NeMo) to prove lift offline via GPU-PB context biasing (nemo-speech-asr-finetune), or deploy (Riva) to ship it via runtime boosted_lm_score (nemotron-speech) — different score systems, don't reuse one for the other. Runtime, no training either way. See references/path-selection.md.
Custom vocabulary / pronunciation — OOV or consistently mispronounced terms. Deploy-time. → Deployment sub-skill.
N-gram (KenLM) LM — domain phrasing/word-sequences when you have text but little audio. Two realizations, built the same way but deployed differently by architecture: pilot (NeMo, NGPU-LM) to prove lift offline (nemo-speech-asr-finetune), or deploy (Riva) to ship it (nemotron-speech). For CTC, rebuild the pilot corpus into a Riva word-level LM — don't ship the pilot artifact as-is. For RNN-T/TDT, the opposite: hand the pilot's .nemo artifact to Riva unchanged, no rebuild. See references/path-selection.md.
Fine-tune — real acoustic gaps (accents, noise, channel) with enough transcribed audio (NIM guide: 100+ h; ~10 h floor only if mixed to avoid catastrophic forgetting). Below ~10 h, do not recommend fine-tuning — recommend word boosting instead (severe catastrophic-forgetting/overfitting risk). → Research/Training.
Train from scratch / cross-language transfer — a new language with no suitable checkpoint (last resort). → Research/Training.

Ordering and per-model support follow the NVIDIA Speech NIM ASR customization guide: https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html.

Key Principles

Scope before you pick. Don't recommend fine-tuning before the discovery questions and a measured baseline.
Cheapest sufficient path. Escalate rungs only when the current one provably can't hit the target; you may experiment on a cheap rung while presenting the full fine-tuning plan.
Measure with a contract. Report normalized WER on the domain set plus an A/B forgetting check on a general set — never in-training logs alone.
Verify before you train. Run the pre-flight dataset quality check before Stage 5: verify/convert audio to 16 kHz, and, when the user asks, audit ground-truth transcripts by comparing them to a reference model's WER. Don't train on an unresampled or unaudited-on-request dataset. See references/workflow.md §4a.
Know your environment before you train. If no NeMo is provided, pull the latest main (or latest published container tag). If one is provided, check its version but still recommend latest main and ask the user first — a provided checkout always carries staleness risk. If they insist on keeping it, don't pull preemptively; only revisit once a concrete version issue (e.g. unsupported functionality) is actually hit. Record what was resolved in the ledger. See references/workflow.md §4b.
Delegate, don't reimplement. Route execution to the sub-skills; keep this skill focused on the plan, sequencing, and cost/time/data answers.
Real target-domain audio is the usual bottleneck. Prefer real data; use synthetic to fill measured gaps, kept separately weighted so it can be ablated.
Consult the user before extra tuning cycles, and when a needed sub-skill is a placeholder.

Source of Truth

Topic	Location
NIM Speech docs home	https://docs.nvidia.com/nim/speech/latest/index.html
ASR customization guide (methods, per-model support)	https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html
ASR support matrix (models & features)	https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html
NeMo fine-tuning (flags/config)	`docs/source/asr/fine_tuning.rst`, and the `nemo-speech-asr-finetune` sub-skill
Riva ASR tutorials (boosting, LM, fine-tune)	https://github.com/nvidia-riva/tutorials
Tokenizer extension to new language + acoustic fine-tune	https://github.com/nvidia-riva/tutorials/blob/main/asr-extend-tokenizer-to-newlang-ft-acoustic-model.ipynb

Limitations

Orchestration only — execution happens in the sub-skills. Where a sub-skill is a placeholder, guidance is interim until it exists.
GPU required for the training rungs; deployment/serving is owned by the nemotron-speech sub-skill.
Model names, config paths, flags, and per-model feature support drift across NeMo/Riva releases — verify against the support matrix and the current checkout.
Public branding is "Nemotron Speech"; commands, imports, config paths, and doc URLs still use "Riva" / "NeMo" — do not rename.

Files

ATTRIBUTION.mdAdded by AutoGineer935 B
BENCHMARK.md10.7 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md11.6 KiB
assets/experiment-ledger-template.md3.5 KiB
evals/EVAL.md8.7 KiB
evals/evals.json20.4 KiB
references/path-selection.md14.8 KiB
references/planning-answers.md3.3 KiB
references/sub-skills.md10.8 KiB
references/workflow.md19.9 KiB
scripts/link-nemo-subskill.shScript3.6 KiB
scripts/main.pyScript10.9 KiB
skill-card.md4.9 KiB
skill.oms.sig6.3 KiB

evals/evals.json

[
  {
    "id": "nemotron-asr-orchestrate-scope-001",
    "question": "I want to fine-tune ASR for my domain. Where do I start?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should run the discovery/scoping questions before recommending a technique, then outline the staged workflow and the sub-skills it will delegate to.",
    "expected_behavior": [
      "Route to references/workflow.md and run Stage 1-2 discovery questions (real audio hours, domain text, eval set/target, latency/HW budget, deployment target).",
      "Do not jump to a fine-tuning command before scoping and a baseline.",
      "Explain the cheapest-sufficient-path approach and name the sub-skills for data, training, evaluation, and deployment.",
      "Offer to answer cost/time/data questions along the way.",
      "Announce steps (Step N/8)."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-path-001",
    "question": "My Riva ASR model gets our German medical jargon wrong. What's the cheapest way to fix it?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "A known-vocabulary problem is the word-boosting rung. The orchestrator should recommend word boosting first and route configuration to the deployment sub-skill, not fine-tuning.",
    "expected_behavior": [
      "Route to references/path-selection.md and diagnose known-jargon as the boosting rung.",
      "Recommend word boosting (runtime, no training) before an LM or fine-tune.",
      "Hand runtime configuration off to the nemotron-speech sub-skill.",
      "Avoid recommending fine-tuning for a bounded known-word problem.",
      "Advise measuring baseline vs post-change WER."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-escalate-001",
    "question": "Boosting and an n-gram LM didn't help — the model mishears our noisy call-center audio with strong accents. What now?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "An acoustic gap after cheaper rungs escalates to fine-tuning; the orchestrator delegates data and training to the sub-skills.",
    "expected_behavior": [
      "Route to references/path-selection.md and recognize accents+noise as an acoustic gap.",
      "Escalate to fine-tuning; consider data via the SDG/Data sub-skill and train via nemo-speech-asr-finetune.",
      "Confirm sufficient transcribed audio (100+ h ideal; ~10 h only if mixed to avoid catastrophic forgetting).",
      "Delegate evaluation (normalized WER + forgetting check) to the evaluation stage of nemo-speech-asr-finetune.",
      "Do not restart at boosting/LM since they were already shown insufficient."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-subskills-001",
    "question": "Which sub-skills will you use to add Farsi support to our ASR, and what does each do?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should map the stages to sub-skills (data-designer, nemo-speech-asr-finetune for training and evaluation, nemotron-speech) and flag the remaining data-profiling placeholder.",
    "expected_behavior": [
      "Route to references/sub-skills.md.",
      "Map data generation to data-designer (+ nemotron-speech TTS), training to nemo-speech-asr-finetune, deployment to nemotron-speech.",
      "Use nemo-speech-asr-finetune for evaluation; flag the data-profiling skill as a placeholder with interim guidance.",
      "Note a new language may need cross-language transfer and lots of data.",
      "Describe the handoff inputs/outputs between orchestrator and each sub-skill."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-planning-001",
    "question": "How much data and how many GPU hours will it take to hit 10% WER, and should I use L40S or H100?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should answer with ranges and assumptions, refuse a fabricated hours-to-WER number, recommend a pilot, and give GPU-choice guidance.",
    "expected_behavior": [
      "Route to references/planning-answers.md.",
      "Explain there is no universal hours-to-WER mapping; establish baseline/target and run a small pilot to read the trend.",
      "Give data guidance by rung (100+ h fine-tune; text-only for LM; little/none for boosting).",
      "Give L40S vs H100 guidance based on model size, batch/precision, VRAM headroom, and cost.",
      "State assumptions and prefer measured pilots over guesses."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-eval-001",
    "question": "How should the orchestrator decide whether the fine-tune worked and whether to loop or ship?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "Evaluate normalized domain WER plus an A/B forgetting check on a general set, do error-driven analysis, then loop with targeted data or ship — consulting the user before extra cycles.",
    "expected_behavior": [
      "Route to references/workflow.md (Stage 6-7) and use nemo-speech-asr-finetune's evaluation stage.",
      "Report normalized WER on the domain set and a general-set forgetting delta.",
      "Use error-driven analysis to choose the next lever and change one thing per loop.",
      "Select/average checkpoints when the target is met; keep a blind holdout.",
      "Consult the user before another tuning cycle."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-preflight-001",
    "question": "Before you fine-tune on my new call-center recordings, verify the audio is 16 kHz and check whether the ground-truth transcripts are any good.",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should run the pre-flight dataset quality gate before training: audit every file's sample rate and resample offline to exactly 16 kHz for any that are not, and, since the user asked, audit transcript quality by running a reference pretrained checkpoint (not the fine-tune target) and comparing its WER against the provided ground truth, flagging the worst utterances rather than silently dropping them.",
    "expected_behavior": [
      "Route to references/workflow.md (Stage 4a) as the pre-flight dataset quality check ahead of Stage 5 training.",
      "Audio: verify every file's sample rate and resample offline to 16 kHz (sox/ffmpeg) for any file that is not already 16 kHz; do not rely on online resampling during training.",
      "Text: since requested, pull a stock pretrained reference checkpoint (not the fine-tune target), run inference, and compute per-utterance WER against the provided transcripts to flag likely-bad ground truth.",
      "Do not silently drop flagged utterances; report the flagged rate/count and let the user decide.",
      "Do not proceed to Stage 5 with unresolved sample-rate mismatches."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-preflight-8khz-001",
    "question": "For the ASR fine-tune, my call-center audio is only available at 8kHz and I don't want to spend time upsampling it — just train directly on the 8kHz files.",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should not silently offline-convert the files, and must not set model.sample_rate=8000 either — that would reconfigure the model into a different, 8 kHz-native architecture rather than fine-tuning the target checkpoint. It should tell the user that skipping the offline conversion is not recommended for a throughput reason (NeMo's own guidance: online resampling during training is slow), confirm the user accepts that trade-off, and, if confirmed, keep model.sample_rate at the checkpoint's expected rate (typically 16000) so NeMo's loader upsamples each 8 kHz file online at load time, producing the same model as the offline-converted path just with slower training.",
    "expected_behavior": [
      "Route to references/workflow.md (Stage 4a) and recognize this as the 'feed 8kHz files as-is' opt-out path, distinct from both the default offline-resample path and from training a dedicated 8kHz-native model.",
      "State that skipping the offline conversion is not recommended and explain why (online per-sample resampling during training is slow, per NeMo's telephony guidance) — not an accuracy concern.",
      "Ask the user to confirm they accept the slower training throughput before proceeding; do not silently offline-convert and do not silently proceed without confirming either.",
      "Do NOT set model.sample_rate=8000 — that trains a materially different, 8 kHz-native model rather than fine-tuning the target checkpoint.",
      "Keep model.sample_rate at the checkpoint's expected rate (typically 16000) so NeMo's audio loader resamples each 8 kHz file online, matching the model's actual expectation.",
      "Only mention model.sample_rate=8000 as a distinct, rarer alternative for a user who explicitly wants a model that is 8 kHz end-to-end including at deployment, not as the answer to this request."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-lowdata-001",
    "question": "Our Riva ASR model mishears industry jargon and I only have about 6 hours of transcribed audio for this domain — should we fine-tune?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "6 hours is below the ~10 h fine-tune floor. The orchestrator should strongly recommend against fine-tuning at this data volume, explain that both catastrophic forgetting and overfitting risk are severe below the floor, and recommend word boosting instead since the problem (mishearing specific jargon) is a known-vocabulary problem that boosting handles without any training.",
    "expected_behavior": [
      "Route to references/path-selection.md and recognize 6 hours as below the ~10 h fine-tune floor.",
      "Strongly recommend against fine-tuning at this data volume; do not present it as the plan.",
      "Explain the specific risk: both catastrophic forgetting (general capability degrades) and overfitting (model memorizes the tiny set instead of generalizing) are severe below the floor.",
      "Recommend word boosting instead, and note it also matches the stated problem (known jargon) independent of the data-volume issue.",
      "Only mention fine-tuning again if the user explicitly accepts the risk after being told it, or once more real data is available."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-envcheck-001",
    "question": "Before we start the NeMo ASR fine-tune, what NeMo version are we going to use — do I need to have it installed already?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should run the pre-flight environment/dependency check before training. If no NeMo is provided/found, it must pull the latest main from the current canonical repo (NVIDIA-NeMo/Speech, noting the 2026 repository split from NVIDIA-NeMo/NeMo) for local execution, or the latest published container tag for container execution. If a NeMo checkout/install/container is already provided, it should check its version, but still recommend switching to latest main (staleness risk) and ask the user rather than silently using either option.",
    "expected_behavior": [
      "Route to references/workflow.md (Stage 4b) as the pre-flight environment/dependency check ahead of Stage 5 training.",
      "Ask whether a NeMo checkout/install/container is already available; do not assume one exists or silently skip the check.",
      "If not provided, pull the latest main from the current canonical NeMo repo for local execution, or the latest published container tag for container execution — not a hardcoded/stale version.",
      "If provided, check its version (nemo.__version__, and git commit/branch if it's a source checkout) rather than assuming it is current or compatible, and still recommend switching to latest main due to staleness risk rather than silently accepting the provided one.",
      "Ask the user which they want (switch to latest main vs. keep the provided checkout) rather than deciding unilaterally.",
      "Record the resolved NeMo version/commit in the ledger.",
      "Do not silently proceed to Stage 5 training without having resolved and recorded a NeMo version."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-envcheck-insist-001",
    "question": "I have a NeMo checkout at /home/user/NeMo from a few months back. I know it might not have every recent feature, but I want to keep using it for this fine-tune rather than switching — don't pull main unless we actually hit a problem.",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "Since the user has explicitly considered the staleness risk and insisted on keeping the provided checkout, the orchestrator should not pull latest main preemptively. It should check and record the provided checkout's version, proceed with it, and only revisit pulling latest main later if a concrete version-specific issue is actually hit (e.g. a config key or feature the recipe needs turns out to be unsupported in that checkout) — not second-guess the user's decision without new evidence.",
    "expected_behavior": [
      "Route to references/workflow.md (Stage 4b).",
      "Recognize the user has already been asked/has already decided to keep the provided checkout; do not re-litigate or push back further, and do not pull latest main preemptively.",
      "Check and record the provided checkout's version/commit in the ledger.",
      "Proceed to Stage 5 with the provided checkout.",
      "State the condition under which the recommendation would resurface: only if a concrete version-specific issue is hit later (e.g. an unsupported config key or missing feature), not as a standing objection."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-subskill-reachability-001",
    "question": "Before we run the NeMo fine-tune, is nemo-speech-asr-finetune actually going to work here, or do I need to set something up first?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "The orchestrator should not assume nemo-speech-asr-finetune is invocable just because it is named in Stage 5. It should explain that this sub-skill is project-local (lives inside the NeMo/Speech repo's own .claude/skills/, not the nvidia-skills catalog), so it must first be made discoverable (try npx skills add against the owning repo, fall back to scripts/link-nemo-subskill.sh) and then the working context must actually be changed into that checkout before any of its commands (which are relative paths) will work.",
    "expected_behavior": [
      "Route to references/workflow.md (Stage 4b) and/or references/sub-skills.md for the Research/Training sub-skill's setup notes.",
      "State that nemo-speech-asr-finetune is project-local, not catalog-published, unlike the other sub-skills.",
      "Describe the two-tier setup: try `npx skills add NVIDIA-NeMo/Speech --skill nemo-speech-asr-finetune ...` first; fall back to `scripts/link-nemo-subskill.sh <checkout_path>` if that doesn't work.",
      "Explain that linking/installing only makes the skill discoverable; the working context (cd, or a scoped sub-agent) must still be changed into the actual NeMo checkout before Stage 5 runs any of its commands, since they are relative paths.",
      "Do not assume the sub-skill is ready to use without verifying both reachability and working context."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-wordboost-nemo-pilot-001",
    "question": "I have a local NeMo RNNT checkpoint and want to try word boosting on it before we even think about deploying to Riva. What do I do?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "Word boosting has a NeMo-only pilot realization owned by nemo-speech-asr-finetune that does not require Riva at all: GPU-PB context biasing via the boosting_tree config, using rnnt_decoding.*.boosting_tree (RNN-T/TDT) through speech_to_text_eval.py, weighted by boosting_tree_alpha. This uses a different score system than Riva's boosted_lm_score, so nothing from the pilot needs to be reused or converted for the customer to get a full offline answer.",
    "expected_behavior": [
      "Route to references/path-selection.md and recognize this as the NeMo-pilot realization of word boosting, not the Riva-deploy realization.",
      "Describe GPU-PB / boosting_tree as the mechanism: decode-time shallow fusion, no retraining.",
      "Give the RNN-T/TDT-specific config keys (rnnt_decoding.*.boosting_tree, boosting_tree_alpha) via speech_to_text_eval.py, not the CTC ones.",
      "State that this is a complete, self-contained loop -- it does not require touching nemotron-speech or Riva at all.",
      "Do not conflate NeMo's context_score/boosting_tree_alpha with Riva's boosted_lm_score scoring convention.",
      "Route execution to nemo-speech-asr-finetune (and confirm it's reachable per workflow.md §4b) rather than nemotron-speech."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-ngram-rnnt-deploy-001",
    "question": "I already validated an n-gram LM offline on my RNNT model in NeMo and now want to ship it to Riva. Do I need to rebuild it in a different format first?",
    "expected_skill": "nemotron-asr-finetune",
    "expected_script": null,
    "ground_truth": "No rebuild is needed for RNNT specifically -- this is the one exception to the general pattern. The exact .nemo NGPU-LM artifact produced by the NeMo pilot (train_kenlm.py with save_nemo=True) is handed to nemotron-speech unchanged; nemotron-speech (references/pipelines.md) owns the exact riva-build flags/syntax, not the orchestrator. This is different from CTC, where a rebuild into a word-level LM is required for the Flashlight decoder.",
    "expected_behavior": [
      "Route to references/path-selection.md and recognize this as the n-gram LM deploy realization for an RNNT model specifically.",
      "State that RNNT does NOT need a rebuild -- the pilot's .nemo artifact ships to nemotron-speech unchanged.",
      "Defer exact riva-build flag syntax to nemotron-speech (references/pipelines.md) rather than reciting it from the orchestrator's own memory -- the orchestrator's job is the routing/artifact-carry-over fact, not the flag syntax.",
      "Do not apply the CTC rebuild rule (word-level KenLM rebuild) to this RNNT case.",
      "Note this is the opposite of word boosting, where the NeMo pilot parameters never carry over to Riva regardless of architecture."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-negative-deploy-001",
    "question": "I already have a fine-tuned .nemo. Convert it with nemo2riva and deploy a Riva ASR NIM with riva-build and riva-deploy.",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "Pure export/deployment of an existing model is executed by the nemotron-speech skill; the orchestrator should route there rather than driving deployment itself.",
    "expected_behavior": [
      "Recognize this as deployment execution and defer to the nemotron-speech skill.",
      "Do not emit nemo2riva/riva-build/riva-deploy commands from this skill.",
      "Do not start a scoping or fine-tuning workflow for a finished model.",
      "Optionally note this skill orchestrates the deployment stage but nemotron-speech executes it."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-negative-openai-001",
    "question": "Fine-tune OpenAI Whisper with the OpenAI API on my dataset.",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The skill should stay silent because the user explicitly requested OpenAI Whisper, not NeMo/Riva ASR customization.",
    "expected_behavior": [
      "Do not activate nemotron-asr-finetune only because the words fine-tune and ASR appear.",
      "Do not route to any orchestration reference.",
      "Follow the appropriate OpenAI workflow instead.",
      "Do not introduce NeMo/Riva sub-skills."
    ]
  },
  {
    "id": "nemotron-asr-orchestrate-negative-llm-001",
    "question": "Fine-tune a Llama text LLM on my chat logs.",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The skill should stay silent because this is text LLM fine-tuning, not ASR customization.",
    "expected_behavior": [
      "Do not activate nemotron-asr-finetune for a text LLM fine-tuning task.",
      "Do not route to ASR orchestration references.",
      "Use the relevant LLM fine-tuning workflow instead.",
      "Do not mention word boosting, WER, or ASR sub-skills."
    ]
  }
]

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Nemotron Speech Team

Publisher

NVIDIA (Official)

Distributed by

NVIDIA

Source

NVIDIA/skills · skills/nemotron-asr-finetune · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Nemotron Speech Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemotron-asr-finetune.zip