AutoGineer
Generated banner for Nemotron Speech
Generated logo for Nemotron Speech

Nemotron Speech

NVIDIAOfficialCreated by Nemotron Speech TeamDistributed by NVIDIA

  • Skill
  • Tool integration
  • Data and AI
  • Apache-2.0
  • Medium risk
  • Has scripts
Download ZIP

From the author

Routes NVIDIA Nemotron Speech (Formerly Riva) NIM tasks — deploys, runs, and tests ASR, TTS, and NMT NIMs on build.nvidia.com or self-hosted.
— the description by Nemotron Speech Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Nemotron Speech acts as a central command interface for managing NVIDIA's speech AI NIMs. It provides a structured workflow for users to navigate deployment options, whether using cloud-hosted inference via build.nvidia.com or self-hosted Docker containers. The skill handles the routing of tasks across ASR, TTS, and NMT modalities, ensuring users follow the correct procedures for model selection, pipeline configuration, and system readiness checks.

What it does

  • Routes users to specific workflows for ASR, TTS, and NMT deployment.
  • Provides guidance on self-hosted Docker deployment and cloud-hosted inference.
  • Assists with custom model builds using riva-build and riva-deploy.
  • Facilitates ASR pipeline tuning including VAD, diarization, and language models.
  • Supports TTS customization such as SSML, zero-shot voice cloning, and pronunciation dictionaries.
  • Directs users to authoritative documentation for version-specific model and hardware requirements.

When to use it

  • When you need to deploy or test NVIDIA Riva/Nemotron Speech NIMs.
  • When configuring custom ASR or TTS pipelines for specific accuracy or vocabulary needs.
  • When troubleshooting deployment readiness, GPU compatibility, or container health.
  • When integrating speech-to-text, text-to-speech, or translation into an application.

Advantages

  • Centralizes complex speech AI workflows into a single, navigable interface.
  • Ensures adherence to official NVIDIA documentation and best practices.
  • Simplifies the transition between cloud-hosted and self-hosted environments.
  • Provides clear, step-by-step procedures for custom model deployment.

Usage examples

“Deploy a Parakeet ASR NIM”
The agent loads the ASR reference file and guides the user through the self-hosted deployment steps.
“How do I fix the pronunciation of NIM in Riva TTS?”
The agent loads the TTS pronunciation reference, suggests IPA candidates, and provides the necessary gRPC snippet.
“Can my GPU run this?”
The agent initiates the deployment readiness check workflow to verify system requirements.
“Translate English to German”
The agent loads the NMT reference file and walks the user through the translation deployment flow.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/nvidia/nemotron-speech/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip nemotron-speech.zip -d .agents/skills

Documentation by Nemotron Speech Team

Nemotron Speech Skills

Note: "Nemotron Speech" is the public-facing name for what NVIDIA documents today as Riva / Riva NIM. All commands, container images, gRPC APIs, Python imports, and documentation URLs still use "Riva" — the rename is brand-only. Do not rename commands, images, or doc URLs.

Agent: When walking the user through a multi-step workflow, announce each step before presenting it: Step N/M — Step Title (e.g., "Step 1/4 — Deploy the Container").

Purpose

Single entry point for all NVIDIA Nemotron Speech (Riva) NIM workflows: ASR (speech-to-text), TTS (text-to-speech), and NMT (translation). Covers cloud-hosted inference via build.nvidia.com, self-hosted Docker deployment, client-protocol choice for ASR (gRPC, HTTP, WebSocket), custom NeMo model deployment via riva-build, ASR pipeline tuning (VAD, diarization, language models), and the prerequisite Docker / NGC / driver setup.

When to Use This Skill

Use this skill for any Nemotron Speech / Riva NIM task — deployment, testing, custom model build, system requirements check, or model selection across ASR / TTS / NMT modalities.

Workflow

Identify the user's task type, then load the corresponding reference file from references/. The reference files contain the detailed per-workflow content; this SKILL.md is a routing surface. Load only the reference relevant to the task at hand.

Prerequisites

  • For self-hosted deployment: NVIDIA AI Enterprise (NVAIE) entitlement, then complete the environment setup — NVIDIA drivers, Docker, Container Toolkit, NGC API key, Riva Python client. See references/setup.md.

  • For cloud-hosted inference: pip install -U nvidia-riva-client and a valid NVIDIA_API_KEY from https://build.nvidia.com.

  • Treat NVIDIA_API_KEY and NGC_API_KEY as secrets: never print, paste, commit, or log real key values. Prefer --password-stdin for Docker login and store persistent keys in a credential manager or a chmod 600 env file rather than world-readable shell startup files.

  • For self-hosted Docker model caching: host directories mounted at /opt/nim/.cache must be writable by the container user (the NIM container runs as nvs:1000 internally), not just the host user. Run sudo chown 1000:1000 $LOCAL_NIM_CACHE after creating the directory so the container can write to it. Avoid world-writable modes — they let any local user replace cached model artifacts. Also avoid -u "$(id -u):$(id -g)" on the docker run — /opt/nim/workspace inside the container isn't writable to arbitrary UIDs. If you see I/O error Permission denied (os error 13) during model download, the host directory ownership is the issue.

Instructions

  • Match the user's task to one reference file and load only that file; the references are detailed, so progressive disclosure keeps context tight.

  • Route setup requests for drivers, Docker, Container Toolkit, and NGC to references/setup.md.

  • Route GPU compatibility, deployment readiness, and container health checks to references/deployment-readiness-checks.md.

  • Route model choice across ASR, TTS, and NMT to references/model-selection.md.

  • Route ASR deployment or inference for Parakeet, Canary, Whisper, and Nemotron ASR Streaming to references/asr.md.

  • Route custom-trained NeMo ASR deployment (.nemo → RMIR → NIM) to references/asr-custom.md.

  • Route ASR pipeline configuration for VAD, diarization, language models, and chunk size to references/pipelines.md.

  • Route TTS deployment or inference for Magpie to references/tts.md.

  • Route custom or fine-tuned TTS model deployment (.nemo → RMIR → NIM) to references/tts-custom.md.

  • Route TTS synthesis pipeline configuration — SSML, zero-shot voice cloning, applying an existing pronunciation dictionary, audio encoding, sample rate, custom_configuration keys — to references/tts-pipelines.md. (For discovering and constructing the pronunciation itself, use tts-pronunciation.md.)

  • Route TTS pronunciation discovery — finding, testing, and applying IPA pronunciations for specific words or phrases — to references/tts-pronunciation.md.

  • Route NMT deployment or inference for Riva Translate, language pairs, and DNT tags to references/nmt.md.

Source of truth

For per-release detail — current model catalog, container IDs, function IDs, voice lists, VRAM minimums, per-model feature support — fetch or open the canonical NVIDIA doc rather than relying on text in this SKILL.md or the references. Each reference file includes its own routing table to the relevant doc pages.

Top-level landing pages:

TopicURL
ASR support matrixhttps://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html
TTS support matrixhttps://docs.nvidia.com/nim/speech/latest/reference/support-matrix/tts.html
NMT support matrixhttps://docs.nvidia.com/nim/speech/latest/reference/support-matrix/nmt.html
Prerequisites (driver / GPU / OS)https://docs.nvidia.com/nim/speech/latest/get-started/prerequisites.html
ASR pipeline configurationhttps://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html
ASR runtime customizationhttps://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html
TTS custom deployment (.nemo / .riva, riva-build, RMIR)https://docs.nvidia.com/nim/speech/latest/tts/custom-deployment.html
TTS request-time customization (SSML, pronunciation dictionaries, custom_configuration)https://docs.nvidia.com/nim/speech/latest/tts/customization.html
TTS voices and emotional styleshttps://docs.nvidia.com/nim/speech/latest/tts/voices.html
TTS zero-shot voice cloninghttps://docs.nvidia.com/nim/speech/latest/tts/voice-cloning.html
TTS IPA phone sethttps://docs.nvidia.com/nim/speech/latest/tts/phoneme-support.html
Cloud function IDs (per model)https://build.nvidia.com/<org>/<model>/api
NGC model cataloghttps://catalog.ngc.nvidia.com/models

Examples

"Deploy a Parakeet ASR NIM" → load references/asr.md, follow Option B (self-hosted), Steps 1–4.

"Synthesize speech with Magpie" → load references/tts.md, follow Option A (cloud) or Option B (self-hosted).

"Translate English to German" → load references/nmt.md, follow the 4-step flow.

"Convert my fine-tuned .nemo to a NIM" → load references/asr-custom.md for the 4-phase pipeline and references/pipelines.md for build-time config.

"Deploy a custom fine-tuned TTS voice as a NIM" → load references/tts-custom.md for the 4-phase pipeline.

"Use zero-shot voice cloning with Magpie" → load references/tts-pipelines.md.

"Add SSML emphasis tags to my TTS request" → load references/tts-pipelines.md.

"'NVIDIA' sounds wrong in my Magpie TTS output — suggest a few IPA options to test" → load references/tts-pronunciation.md, generate IPA candidates, synthesize variants, then output all three delivery formats.

"How do I fix the pronunciation of 'NIM' in Riva TTS with a custom_dictionary in gRPC Python?" → load references/tts-pronunciation.md, propose IPA for 'NIM', show wire format and gRPC snippet.

"Can my GPU run this?" → load references/deployment-readiness-checks.md and run the 6-step system check.

"Which Riva model should I use?" → load references/model-selection.md, apply the decision framework, then fetch the support matrix for the specific current model name.

Naming & Terminology

  • Skill brand: Nemotron Speech (public-facing name).

  • Internal naming preserved: commands (riva-build, riva-deploy, riva_streaming_asr_client), Python client (riva.client), gRPC namespace (nvidia.riva.asr.*), container registry (nvcr.io/nim/nvidia/*), and all NVIDIA documentation URLs still use "Riva". Do not rename these in code, commands, or docs.

Troubleshooting

For task-specific runtime or modality issues, use the relevant reference file (references/<task>.md). Cross-cutting readiness checks:

Limitations

  • x86_64 architecture only — WSL2 on Windows requires Podman and supports a subset of NIMs (see references/setup.md)

  • Self-hosted deployment requires an NVIDIA AI Enterprise license

  • Cloud-hosted inference requires an active NVIDIA_API_KEY and internet access

  • Public skill branding is "Nemotron Speech"; commands, container images, Python imports (riva.client), gRPC services (nvidia.riva.*), and NVIDIA documentation URLs still use "Riva" — follow official docs and catalogs for naming, do not rename these in commands or code

Next Steps

Files

references/setup.md

Riva NIM Setup

Agent: Announce each step before presenting it: Step N/7 — Step Title (e.g., "Step 1/7 — Install NVIDIA Drivers").

Source of truth. This skill describes the install workflow and command shapes, which are stable. For per-release minimums — driver version, supported OS list, WSL2 supported models, glibc minimum — fetch or open the canonical doc page and answer from that. See Looking up current information below.

Purpose

Prepare a Linux x86_64 system to run NVIDIA Riva Speech NIM containers. Covers NVIDIA driver installation, Docker setup, NVIDIA Container Toolkit, NGC authentication, and the Riva Python client. Follow the 7 steps in order — this setup only needs to be done once per machine.

Looking up current information

Question typeFetch this page
Minimum driver version, supported GPUs, glibc minimum, supported OSes, WSL2 driver / OS minimums and supported model subsethttps://docs.nvidia.com/nim/speech/latest/get-started/prerequisites.html
Per-model VRAM requirementshttps://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html (TTS / NMT analogs)
Container Toolkit install (latest steps)https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
Docker engine install (per distro)https://docs.docker.com/engine/install/
CUDA install guide for Linux (driver-only on host)https://docs.nvidia.com/cuda/cuda-installation-guide-linux
Generate NGC API keyhttps://org.ngc.nvidia.com/setup/api-keys

Do not infer driver-version or OS minimums from this skill's text. The prerequisites page is the contract.

Hardware Requirements

Key invariants (stable):

  • CPU: x86_64 only

  • NVIDIA AI Enterprise license required for self-hosting

  • Install driver only — CUDA toolkit is bundled inside the NIM container

For minimum driver / OS / glibc / GPU compute capability — fetch the prerequisites page. These rotate per release.

Instructions

Follow the 7 steps below in order. Steps 1–3 require root/sudo. Steps 4–7 run as a normal user. Complete all steps before attempting to pull or run any Riva NIM container.

Cache directory ownership

When a Riva NIM command exports model artifacts to a mounted host directory, create the directory and run sudo chown 1000:1000 <directory> because the NIM container runs as nvs:1000 inside and needs write access to the mount. Avoid world-writable modes; they let any local user replace exported model artifacts. Avoid -u $(id -u):$(id -g) on the docker run; /opt/nim/workspace inside the container is not writable to arbitrary UIDs.

Step 1 — Install NVIDIA Drivers

Install drivers via package manager. Skip the CUDA toolkit — it is bundled inside the NIM container.

# Verify installed driver version
nvidia-smi

Check the minimum required driver version on the prerequisites page (cited above) before installing. See the CUDA installation guide for Linux for package manager install steps.

Step 2 — Install Docker

Install Docker Engine for your distro: https://docs.docker.com/engine/install/

After install, allow your user to run Docker without sudo:

sudo usermod -aG docker $USER
# Log out and back in for this to take effect

Step 3 — Install NVIDIA Container Toolkit

The Container Toolkit lets Docker containers access the host GPU.

# Install (see full guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access inside a container:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

The output must show your driver version and GPU(s). If it does, the environment is ready.

Step 4 — NGC API Key

  1. Open https://org.ngc.nvidia.com/setup/api-keys

  2. Create a key with at least NGC Catalog under Services Included

  3. Export it in your terminal:

export NGC_API_KEY=${your-key-value}

To persist across sessions, keep the key in a private file rather than in your shell startup file:

umask 077
printf 'export NGC_API_KEY=%s\n' "${your-key-value}" > ~/.nvidia-ngc-env
chmod 600 ~/.nvidia-ngc-env

Then add this line to ~/.bashrc or ~/.zshrc with your editor:

[ -f ~/.nvidia-ngc-env ] && . ~/.nvidia-ngc-env

Security note: Do not write the key itself into ~/.bashrc or ~/.zshrc. Those files are often world-readable, so any process running as you can extract the key. Holding it in a chmod 600 file that your shell sources keeps the credential out of the RC file and off the default-readable path. For production, use a credential manager (for example HashiCorp Vault) instead of an on-disk file.

Step 5 — Docker Login to nvcr.io

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

  • Username is the literal string $oauthtoken (not your NGC username)

  • Password is the value of NGC_API_KEY

After this, docker pull nvcr.io/nim/nvidia/<image>:<tag> will succeed.

Step 6 — Install Riva Python Client

Required to run the sample inference scripts from python-clients/.

pip install nvidia-riva-client

Verify:

python3 -c "import riva.client; print('Riva client OK')"

Step 7 — Clone Client Repos (Optional)

Sample scripts live in the public repos. Clone whichever you need:

# Python clients and sample scripts
git clone https://github.com/nvidia-riva/python-clients

# C++ clients (requires Bazel)
git clone https://github.com/nvidia-riva/cpp-clients

# WebSocket bridge (AudioCodes / telephony)
git clone https://github.com/nvidia-riva/websocket-bridge

Examples

Verify GPU access inside a container (after Step 3):

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Verify Riva Python client (after Step 6):

python3 -c "import riva.client; print('Riva client OK')"

Log in to nvcr.io (Step 5):

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Troubleshooting

  • Username must be $oauthtoken literally — not your NGC username or email.

  • Driver-only install — do NOT install the CUDA toolkit separately; the NIM container brings its own.

  • Group change requires logout — usermod -aG docker only takes effect after re-login.

  • glibc check — run ld -v and compare against the minimum on the prerequisites page (older Ubuntu releases may not meet the requirement).

  • WSL2 on Windows — use Podman instead of Docker; the supported driver / Ubuntu version / model subset rotate per release. Fetch the prerequisites page for current minimums.

Limitations

  • x86_64 architecture only — WSL2 on Windows requires Podman instead of Docker and supports only a subset of NIMs (verify on prerequisites page)

  • NVIDIA AI Enterprise license required for self-hosting Riva NIMs

  • Do not install the CUDA toolkit separately — it is bundled inside the NIM container

  • Group membership changes (docker group) require logout/login to take effect

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Nemotron Speech Team
Publisher
NVIDIA (Official)
Distributed by
NVIDIA
Source
NVIDIA/skills · skills/nemotron-speech · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Nemotron Speech Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemotron-speech.zip