RAG Perf

NVIDIAOfficialCreated by NVIDIA RAG

Skill
Tool integration
Software development
Apache-2.0
Medium risk

From the author

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

— the description by NVIDIA RAG

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

rag-perf is a command-line tool designed to benchmark the performance of deployed NVIDIA RAG Blueprint servers. It executes a two-phase process: a server-side profiling pass to analyze stage-specific timing and bottlenecks, followed by an optional aiperf-based load test to measure throughput, latency, and error rates. The tool is entirely driven by YAML configuration files, allowing users to define concurrency levels, retrieval parameters, and synthetic query generation. It provides comprehensive reporting, including per-point analysis and aggregate sweep comparisons, making it suitable for regression testing and performance tuning of RAG pipelines.

What it does

Performs server-side profiling to identify bottlenecks and stage-specific latency.
Executes aiperf load tests to measure TTFT, throughput, and error rates.
Supports multi-axis sweeps for concurrency, vdb_top_k, and reranker_top_k parameters.
Generates unified reports including summary tables and side-by-side comparisons.
Validates configurations and provides reproducible output via resolved YAML dumps.
Integrates with synthetic query generation for automated load testing.

When to use it

When you need to profile the latency and performance of a deployed RAG server.
When conducting regression checks on RAG pipeline configurations.
When performing concurrency sweeps to determine optimal throughput.
When tuning retrieval and reranker parameters for performance optimization.

Advantages

Config-driven approach ensures reproducible benchmarking runs.
Provides granular stage-by-stage timing breakdowns.
Automates complex multi-axis performance sweeps.
Includes built-in support for synthetic query generation.
Produces clear, actionable reports with bottleneck identification.

Usage examples

“Run a quick performance profile of my RAG server.”: The agent executes the quick_profile.yaml preset, generating a profile report without running a full load test.
“Benchmark my RAG server with a concurrency sweep.”: The agent runs the sweep.yaml configuration, producing a nested directory structure with aggregate results comparing different concurrency and top-k settings.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/rag-perf/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip rag-perf.zip -d .agents/skills
```

Documentation by NVIDIA RAG

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
Python 3.11+ and uv on PATH.
Install rag-perf into its own uv-managed venv: uv sync --project scripts/rag-perf.
For unit tests: install dev extras as well — uv sync --project scripts/rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./scripts/rag-perf registers it via the aiperf.plugins entry point.
For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/v1/chat/completions).
rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

Pick a preset. The three under scripts/rag-perf/configs/ are:
- quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
- single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
- sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).
Edit the preset. Required: replace rag.collection_names: ["<collection_name>"] with a real collection on the deployed ingestor server. Verify the collection exists via GET /v1/collections on the ingestor. The placeholder <collection_name> validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.
Run. From repo root:
```
uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml
```
Same form for the other presets. The CLI accepts only -c / --config (required), --help, --version.
Read stdout. Every invocation prints, in order: a startup banner, a one-line summary, the fully resolved config as YAML (so the run is reproducible from terminal output), per-grid-point progress with the shlex-joined aiperf command in copy-pastable form, a rich per-point summary table (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a side-by-side comparison table auto-labelled by whichever axis varied. See references/output-and-analysis.md.
Inspect artifacts. Layout depends on run shape — flat for single-point + iterations=1, nested under iter_<i>/<point>/... otherwise. See references/output-and-analysis.md for the full directory tree, file purposes, and how to parse results.json / results.csv / report.md.
Summarise for the user. When reporting back, follow the playbook in references/output-and-analysis.md#summarising-results-to-the-user: pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, always flag zero citations / non-zero error rate / suspect llm_ttft_ms / small-sample p99, and propose a concrete next-experiment YAML.
Tune. Schema is fully documented in docs/performance-benchmarking.md and the deeper-dive references below. Common knobs: turn aiperf.enabled: false for profile-only mode, increase load.iterations for variance estimation, set load.sleep_between_points_s: 60 for overnight Cartesian sweeps.

Examples

Profile-only (quickest signal on retrieval / reranker tuning):

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/quick_profile.yaml

Output: rag-perf-results/quick_profile/run_<ts>/{profile_report.md, profile_results.json, profiling/}. The aiperf_rag_on/ directory is omitted. Filenames are profile_* because aiperf.enabled: false.

Single benchmark point with full report:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml

Output: flat run_<ts>/{report.md, results.json, results.csv, profiling/, aiperf_rag_on/}.

Concurrency sweep:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/sweep.yaml

Output: nested run_<ts>/iter_1/<CR:_VDB-K:_RERANKER-K:_…>/{profiling,aiperf_rag_on}/ per point, plus aggregate report.md / results.json / results.csv at the run root.

Run unit tests:

uv sync --project scripts/rag-perf --extra dev   # one-time, installs pytest-asyncio
uv run --project scripts/rag-perf python -m pytest tests/unit/test_rag_perf/

Limitations

The CLI is config-only: author or copy YAML to vary a parameter.
load.concurrency / rag.vdb_top_k / rag.reranker_top_k accept int | list[int]; the validator requires unique list values because each value names a unique point dir.
input.file and input.synthetic follow an XOR rule — both set fails validation. When neither is set, synthetic auto-fills with defaults so a bare config still validates.
File-based input format is inferred from extension only (.jsonl or .csv); other extensions are rejected.
Synthetic generation streams each query to disk as it completes (failure-resilient) but fails fast on the first LLM error — partial JSONL is preserved. Re-run after fixing the endpoint.
Reasoning models (Nemotron Omni, Qwen-Reasoning) require synthetic.disable_thinking: true (the default). Without it the model exhausts the token budget on chain-of-thought and content returns empty — the generator now raises with a clear message instead of substituting reasoning_content for the answer.
aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing AiperfRunner._base_aiperf_cmd in scripts/rag-perf/rag_perf/runner.py.
Procedural detail lives under references/ to keep this file concise.

Troubleshooting

Error / signal	Likely cause	What to do
`Configuration errors in <yaml>: • input — ... XOR rule`	Both `input.file` and `input.synthetic` set	Pick one. The XOR validator runs at YAML load time.
`input.file must end in .jsonl or .csv`	Extension other than `.jsonl` / `.csv`	Rename or convert.
`load.concurrency has duplicate values`	e.g. `[2, 2, 4]`	Each concurrency maps to a unique point dir; dedupe.
`warmup_requests must be >= 1`	YAML had `warmup_requests: 0`	aiperf rejects warmup=0; minimum is 1.
`LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true).`	Reasoning model used CoT and ran out of tokens	Set `synthetic.disable_thinking: true` (the default) or raise `min_query_tokens`.
`✗ All N profiling requests failed across M point(s).` + exit 1	Bad URL, server down, wrong collection	Verify `target.url`, `rag.collection_names` (the `<collection_name>` placeholder will hit this).
Per-iteration `⚠ N profiling requests failed` warning, run continues	Some requests timed out / errored mid-run	Check rag-server logs, raise `target.timeout_s`, drop concurrency.
`RuntimeError: Random synthetic query generation failed at query N: ...`	LLM endpoint rejected a request mid-generation	Partial JSONL is at `synthetic.jsonl_output_path`; fix endpoint and re-run with reduced `num_queries`, or point `input.file` at the partial file.
`Citation count (mean): 0` and `Citation relevance score: N/A` for a non-empty deployment	Collection mismatch between `rag.collection_names` and what's actually ingested	Run `curl -s http://<ingestor>:8082/v1/collections` to list real collections.
Tests error with `ModuleNotFoundError: No module named 'pytest_asyncio'`	Dev extras missing	`uv sync --project scripts/rag-perf --extra dev`.
CI: `ModuleNotFoundError: No module named 'ruamel'` from `tests/unit/test_rag_perf/`	rag-perf package missing from CI venv	Add `uv pip install -e ./scripts/rag-perf` after the top-level install in the unit-tests job.

Gotchas

Run from repo root. Preset configs reference scripts/rag-perf/examples/queries.jsonl and scripts/rag-perf/prompts/default_prompts.yaml with repo-root-relative paths. Running from inside scripts/rag-perf/ will fail those file lookups.
CLI is config-only. Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.
Always edit rag.collection_names before the first run. The presets ship with ["<collection_name>"] as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as Citation count (mean): 0 everywhere.
load.concurrency_list, rag.vdb_top_k_list, rag.reranker_top_k_list are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.
aiperf.enabled: false changes filenames. The top-level outputs become profile_report.md / profile_results.json / profile_results.csv. The aggregate sweep table also suppresses load-test rows and the "Optimal throughput" footer.
Resolved-config dump is verbose (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.
The aiperf shell command is logged before each subprocess. Look for \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ... in stdout — copy-paste runnable for reproducing a single point outside rag-perf.
--endpoint-type nvidia_rag comes from the bundled plugin at scripts/rag-perf/rag_perf/plugin/nvidia_rag.py. It teaches aiperf about the RAG /v1/generate request shape and parses citations + per-stage metrics out of the SSE stream. If aiperf can't resolve nvidia_rag, rag-perf needs editable installation in the venv — re-run uv sync --project scripts/rag-perf (or uv pip install -e ./scripts/rag-perf).
Sweep-mode point-name collision. When two points differ only in concurrency (e.g. [1, 4] × single vdb_top_k), the dir name encodes everything: CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:.... Cluster / GPU / experiment_name (output.cluster, output.gpu, output.experiment_name) are appended too — useful for diff-friendly artifact paths across machines.
load.iterations > 1 repeats the entire grid. Each repetition writes to its own iter_<i>/. Aggregate CSV row count = n_points × iterations.

Source of truth

Piece	Location
Driver	`scripts/rag-perf/rag_perf/cli.py` (`main` is the single Click command)
Schema	`scripts/rag-perf/rag_perf/config.py` (`RunConfig` and sub-models)
Orchestrator	`scripts/rag-perf/rag_perf/runner.py` (`BenchmarkRunner.run`, `RagProfiler`, `AiperfRunner`)
aiperf plugin	`scripts/rag-perf/rag_perf/plugin/nvidia_rag.py`
User-facing doc	`docs/performance-benchmarking.md`
Presets	`scripts/rag-perf/configs/{quick_profile,single_run,sweep}.yaml`
Sample queries	`scripts/rag-perf/examples/queries.jsonl`
Synthetic prompts	`scripts/rag-perf/prompts/default_prompts.yaml`
Config schema details	`references/config-schema.md`
Synthetic-query generation	`references/synthetic-generation.md`
Output layout & metric semantics	`references/output-and-analysis.md`

Agent playbook

Sync deps: uv sync --project scripts/rag-perf (one-time per checkout).
Pick & customise a preset: copy scripts/rag-perf/configs/<preset>.yaml if you want a variant; always set rag.collection_names to a real collection.
Run: uv run --project scripts/rag-perf rag-perf -c <config> from repo root.
Read the per-point + aggregate tables on stdout. Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table.
Parse artifacts under output.dir/run_<ts>/ — see references/output-and-analysis.md. For multi-point runs, results.csv has one row per (point × iteration).
Summarise for the user using the playbook in references/output-and-analysis.md#summarising-results-to-the-user — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations / non-zero errors / suspect llm_ttft_ms / low sample size, and a concrete next-experiment YAML.
Tune retrieval / reranker: flip to quick_profile.yaml or aiperf.enabled: false for fast iteration, then return to single_run.yaml / sweep.yaml when characterising under load.
Triage failures: see Troubleshooting above and references/output-and-analysis.md for empty-citation / bottleneck=N/A patterns.

Files

ATTRIBUTION.mdAdded by AutoGineer886 B
BENCHMARK.md2.6 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md16.0 KiB
eval/h100.json2.4 KiB
eval/nvidia_hosted.json1.8 KiB
references/config-schema.md7.6 KiB
references/output-and-analysis.md12.8 KiB
references/synthetic-generation.md7.2 KiB
skill-card.md2.7 KiB
skill.oms.sig5.4 KiB

references/synthetic-generation.md

Synthetic query generation

Load this when input.synthetic is in play, when reasoning-model query leakage is suspected, when generation fails midway, or when the user wants to reproduce a query set across runs.

Implementation lives in scripts/rag-perf/rag_perf/query.py (SyntheticQueryGenerator). Default prompts are in scripts/rag-perf/prompts/default_prompts.yaml.

Pipeline

When input.synthetic is set, rag-perf — before the benchmark phase even starts — does this:

Resolves the LLM model (synthetic.llm_model, or auto-discover via GET /v1/models).
Loads prompt templates (synthetic.prompts_file, or bundled defaults).
For mode: dataset_based, loads reference questions from synthetic.dataset_file or synthetic.dataset_name (auto-lookup under ./datasets/<name>/{train,data}.json). For mode: random, no reference material.
Builds N per-query user messages.
Fans out concurrent LLM calls (bounded by synthetic.generation_concurrency, default 8) using asyncio.gather over asyncio.to_thread wrappers around the sync httpx.post.
Streams each successful query to disk as it completes — under an asyncio.Lock, with flush() after every line. The file at synthetic.jsonl_output_path is opened in "w" mode and written line-by-line.
Returns the in-memory list (also persisted on disk).
Hands off to QueryLoader._load_jsonl and the benchmark runs from the now-static file.

The key consequence: a mid-generation failure preserves all queries that completed before it. The exception still propagates (asyncio.gather cancels remaining tasks on first failure) and the run aborts — no automatic retry.

All synthetic knobs

Field	Default	Purpose
`mode`	`random`	`random` (no seed) or `dataset_based` (seeded by reference questions).
`num_queries`	`50`	Distinct queries to generate. The list is cycled if `total_requests` exceeds it.
`min_query_tokens`	`50`	Approximate minimum word count target (multiplied by 0.75 to derive `word_target` for the prompt). Combined with `generation.min_tokens == max_tokens` and `generation.ignore_eos: true`, pins exact ISL × OSL.
`generation_concurrency`	`8` (`>= 1`)	Bounded parallel LLM calls. Raise on fast endpoints, lower for rate-limited ones.
`temperature`	`0.9`	Sampling temperature for the generator LLM.
`disable_thinking`	`true`	Inject `chat_template_kwargs: {enable_thinking: false}` into the request body. Critical for reasoning models.
`extra_body`	`null`	Escape hatch — arbitrary keys merged into the LLM request body. Merged after `disable_thinking`, so explicit keys here win.
`llm_url`	`http://localhost:8999/v1/chat/completions`	OpenAI-compatible endpoint. Often the same NIM the RAG server proxies, but can be any.
`llm_model`	`""`	Empty string → auto-discover via `GET <llm_url base>/v1/models`.
`prompts_file`	`null`	Custom YAML; `null` → bundled defaults.
`jsonl_output_path`	`./rag-perf-synthetic-queries.jsonl`	Where streamed queries land. Re-running with the same path overwrites it.
`dataset_file`	`null`	Required for `dataset_based` (or use `dataset_name`).
`dataset_name`	`null`	Auto-lookup — searches `./datasets/<name>/train.json`, `./datasets/<name>.json`, `./datasets/<name>/data.json` in order.

For dataset_based, validation requires either dataset_file or dataset_name. Both unset → ValidationError.

Reasoning-model gotcha (read this if generation looks corrupted)

Symptom: the synthetic JSONL contains entries like:

{"query": "We need to output a single question, at least 384 words long. Must be specific and self-contained. Only the question, no extra text. So we need a long question (384+ words). Must be a question that could be answered..."}

The LLM's chain-of-thought is leaking into the query text.

Cause: Nemotron Omni / Qwen-Reasoning / similar models, in reasoning mode, put their final answer in message.content and the deliberation in message.reasoning_content. With min_tokens near the model's reasoning budget, content can come back empty — the model exhausted the budget on CoT.

Why rag-perf used to leak this: an old version of _call_llm fell back to reasoning_content when content was empty. We removed that fallback — _call_llm now reads only message.content and raises if empty, with a clear hint:

LLM returned empty content (reasoning_content was populated — model exhausted its
budget on chain-of-thought; raise min_query_tokens or set
synthetic.disable_thinking=true).

Fix paths:

Default already correct: synthetic.disable_thinking: true injects chat_template_kwargs: {enable_thinking: false}. The model skips reasoning and writes the answer directly to content.
For non-reasoning endpoints: set disable_thinking: false to avoid sending the unsupported kwarg.
Last resort: raise min_query_tokens substantially so the model has budget for both reasoning and answer.

Failure recovery (partial JSONL)

If generation fails at query 47 of 100:

Queries 1–46 (or however many had completed; order is completion-order, not request-order, since calls are concurrent) are on disk at synthetic.jsonl_output_path.
The exception in stdout looks like: RuntimeError: Random synthetic query generation failed at query N: <root cause>.

Recovery options:

Fix the LLM endpoint and re-run: the file is overwritten ("w" mode) — old partial is lost.
Use the partial directly: swap input.synthetic for input.file: <jsonl_output_path> and the benchmark runs from whatever made it to disk.
Lower num_queries so the new total stays under what you previously generated; combine with input.file pointing at the partial.

Prompt templates

Default templates (prompts/default_prompts.yaml) are deliberately strict to keep content clean: forbid markdown, numbering, "Question:" / "Here is" / "Sure," prefixes, planning/thinking text, restating instructions. They require exactly one ? at the end.

If swapping in custom prompts via synthetic.prompts_file, preserve the same output discipline or expect leaked planning text in the JSONL — the rag-perf side does only minimal cleanup (q.lstrip("0123456789.). ").strip() to drop leading numbering).

Variables interpolated into the templates:

{word_target} — int(min_query_tokens * 0.75), lower-bound 10.
{index} — 1-based query index (for "make this unique" hints).
{ref} — reference question (dataset_based mode only).

Reproducibility

synthetic.jsonl_output_path is the canonical artefact. Commit it to a known location and switch to input.file: <that path> for subsequent runs to keep the load identical while iterating on retrieval / reranker config.
Generation is concurrent → completion order is non-deterministic. The dataset is reproducible across runs only if you pin and reuse the JSONL — not by re-running generation with the same config. (Even with the same seed, async scheduling is non-deterministic.)

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs network access, Needs an API key, Needs Python

The author says: Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/rag-perf (perf deps live in scripts/rag-perf/pyproject.toml); reachable RAG server (default http://localhost:8081); for synthetic queries an OpenAI-compatible chat-completions endpoint is required (default http://localhost:8999/v1/chat/completions); aiperf load-test phase uses the bundled nvidia_rag endpoint plugin, registered automatically when rag-perf is installed editable.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA RAG

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/rag-perf · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by NVIDIA RAG or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download rag-perf.zip