Tao Analyze Gaps Vlm Bcq

NVIDIAOfficialCreated by NVIDIA Corporation

Skill
Review and audit
Software development
Apache-2.0
Medium risk
Has scripts

From the author

Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.

— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill automates the evaluation of Vision Language Model (VLM) binary classification tasks. By comparing model responses against ground truth data, it identifies specific failure cases—false positives and false negatives—and generates structured reports. These outputs are designed to facilitate root-cause analysis and support iterative model improvement workflows.

What it does

Parses VLM prediction JSON files for binary yes/no classification results.
Compares model responses against provided ground truth labels.
Identifies and categorizes false-positive and false-negative prediction errors.
Generates a JSONL file containing detailed records of each identified failure case.
Produces a human-readable summary report detailing total error counts.
Supports relative path resolution for video assets referenced in predictions.

When to use it

When you need to audit VLM performance on binary classification tasks.
When preparing data for root-cause analysis (RCCA) in a DEFT iteration workflow.
When you need to extract specific failure samples from a large set of model predictions.
When you need to generate KPI reports for model evaluation cycles.

Advantages

Automates the tedious process of manual error identification in VLM outputs.
Produces machine-readable output for seamless integration with downstream analysis tools.
Provides clear, human-readable summary reports for quick performance assessment.
Standardizes the evaluation process for binary classification workflows.

Usage examples

“Analyze my VLM BCQ gaps using the results.json file.”: The skill will process the predictions, compare them against ground truth, and generate a kpi_gaps.jsonl file and a kpi_gaps_report.txt summary.
“Extract false positives and false negatives from my model predictions.”: The agent will identify the failure cases based on the provided predictions JSON and output the specific samples that were misclassified.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/tao-analyze-gaps-vlm-bcq/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip tao-analyze-gaps-vlm-bcq.zip -d .agents/skills

Documentation by NVIDIA Corporation

VLM Binary Classification Gap Analysis

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report. Run it with a TAO Data Services spec file; the data-services entrypoint requires -e <spec>.

Purpose

After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.

Usage

Generate a vlm_bcq_spec.yaml with the bundled helper:

python3 skills/data/tao-analyze-gaps-vlm-bcq/scripts/prepare_vlm_bcq_spec.py \
  --predictions-json /path/to/results.json \
  --videos-dir /path/to/videos/root \
  --results-dir /path/to/output/gaps \
  --output-spec /path/to/output/gaps/vlm_bcq_spec.yaml

Omit --videos-dir when prediction video_id values are already absolute. The generated spec has this shape:

predictions_json: /path/to/results.json
videos_dir: ""
results_dir: /path/to/output/gaps

Set videos_dir when video_id values in the predictions are relative paths:

predictions_json: /path/to/results.json
videos_dir: /path/to/videos/root
results_dir: /path/to/output/gaps

Invoke the vlm_bcq action inside the TAO Toolkit data services container with -e <spec>:

gap_analysis vlm_bcq -e /path/to/vlm_bcq_spec.yaml

Request exactly one GPU from the selected platform (compute_shape.gpus: 1, compute_shape.nodes: 1). VLM BCQ gap analysis does not perform GPU compute, but the Data Services image always calls nvidia-smi and fails when no GPU is visible. One is a GPU count, not a device ID; the platform selects the device.

After the run, surface the FP/FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.

Inputs

config spec: YAML file passed with -e. Template: assets/default_vlm_bcq.yaml.
predictions_json: Path to predictions JSON file. Must be a JSON array where each item has video_id, response, and gt fields. response and gt are parsed with word-boundary matching — 'yes' or 'no' anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.
videos_dir (optional): Base directory for resolving relative video_id paths. If omitted, video_id values are used as absolute paths.
results_dir: Output directory for gap-analysis artifacts.

Predictions JSON format:

[
  {
    "video_id": "/path/to/video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]

Outputs

kpi_gaps.jsonl: One JSON object per line for each FP/FN case. Fields: video_id (absolute path), error_type (FP or FN), question, ground_truth, response.
kpi_gaps_report.txt: Human-readable table with total FP/FN counts.

If no gaps are found, no files are written and a message is logged.

Spec Fields

Parameter	Required	Description
predictions_json	Yes	Path to predictions JSON file
results_dir	Yes	Output directory; created if it does not exist
videos_dir	No	Base directory for resolving relative `video_id` paths

Keep the spec file and every path it references under the bind-mounted workspace so they resolve inside the container. Pass -e <spec> even if you also add Hydra overrides; current TAO Data Services entrypoints hard-require an experiment spec file before processing overrides.

Error Patterns

Error	Cause	Fix
`FileNotFoundError`	`predictions_json` does not exist	Check the path
`requires the following argument: -e/--experiment_spec_file`	The container was launched without a spec file	Write `vlm_bcq_spec.yaml` and pass `gap_analysis vlm_bcq -e <spec>`
`ValueError: must be a JSON array`	Predictions file is not a list	Wrap predictions in `[...]`
`ValueError: missing 'gt'/'response'/'video_id'`	A prediction item is missing a required field	Inspect and fix the predictions JSON
Samples silently skipped	`response` or `gt` contains both or neither 'yes'/'no'	Check logs for warnings; inspect those samples

Files

ATTRIBUTION.mdAdded by AutoGineer942 B
BENCHMARK.md6.9 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md5.1 KiB
assets/default_vlm_bcq.yaml54 B
config/skillspector-baseline.yaml1.1 KiB
evals/evals.json873 B
references/skill_info.yaml871 B
scripts/prepare_vlm_bcq_spec.pyScript2.4 KiB
skill-card.md4.7 KiB
skill.oms.sig5.4 KiB

BENCHMARK.md

Skill Benchmark: tao-analyze-gaps-vlm-bcq

✅ Overall verdict: PASS — Recommended for publication

Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

Evaluation Metadata

Skill: tao-analyze-gaps-vlm-bcq
Evaluation date: 2026-10-16
Evaluator version: 1.5.6
Agents: Claude Code (aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5)
Tasks: 1 evaluation tasks (1 positive)
Dataset digest: sha256:e140184632b0fe19bd5db4852af88c561135630f4b93228a40a301a1db96580d (skill-evaluator-dataset-snapshot/1)
Attempts per task: 1
Environment: k8s-sandbox
Tier 2 evidence: required for publication
Tier 3 evidence: required for publication

Each task attempt ran in its own isolated sandbox pod.

What This Report Answers

The three-tier evaluation checks whether the skill:

is safe to use;
produces correct answers;
is discovered and activated when needed;
helps the agent complete the user's goal and expected workflow; and
avoids wasted skill and tool usage.

Results at a Glance

Measure	Claude Code (Baseline → Skill Uplift)	Codex (Baseline → Skill Uplift)
Overall	94.6% — baseline ran, but no comparable score was available; uplift unavailable	95.4% — baseline ran, but no comparable score was available; uplift unavailable
Security	100.0% → 100.0% (±0.0 points)	100.0% → 100.0% (±0.0 points)
Correctness	0.0% → 100.0% (+100.0 points)	40.0% → 100.0% (+60.0 points)
Discoverability	100.0% — baseline ran, but no comparable score was available; uplift unavailable	95.0% — baseline ran, but no comparable score was available; uplift unavailable
Effectiveness	0.0% → 100.0% (+100.0 points)	26.7% → 83.3% (+56.6 points)
Efficiency	73.1% — baseline ran, but no comparable score was available; uplift unavailable	98.7% — baseline ran, but no comparable score was available; uplift unavailable

How to read this table: baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Example: 47.0% → 92.0% (+45.0 points) means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.

Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

Agent	Dataset case	With skill	Without skill	Delta	Change	Coverage
claude-code	All cases	94,947	534,588	-439,641	-82.24%	skill 1/1; base 1/1
claude-code	tao-analyze-gaps-vlm-bcq-basic	94,947	534,588	-439,641	-82.24%	skill 1/1; base 1/1
codex	All cases	29,057	14,117	+14,940	+105.83%	skill 1/1; base 1/1
codex	tao-analyze-gaps-vlm-bcq-basic	29,057	14,117	+14,940	+105.83%	skill 1/1; base 1/1
ALL AGENTS	Dataset aggregate	124,004	548,705	-424,701	-77.40%	skill 2/2; base 2/2

Prompt tokens include cached reads, so total tokens are prompt + completion (cached is not added twice). The Efficiency score uses (prompt - cached) + completion. N/A means the relevant trajectory counters were not available; coverage is never estimated.

Tier Status

Tier	Purpose	Status	Evidence
Tier 1	Static validation	PASSED WITH OBSERVATIONS	11 validator(s); 17 finding(s)
Tier 2	Semantic deduplication	PASSED	2 validator(s); 0 finding(s)
Tier 3	Live agent evaluation	PASS	2 agent(s); 1 task(s)

Findings and Observations

<details> <summary>Show detailed findings and successful checks</summary>

MEDIUM QUALITY/quality_correctness: No documented scripts in table format (skills/data/tao-analyze-gaps-vlm-bcq/SKILL.md)
MEDIUM QUALITY/quality_correctness: Instructions don't mention 'run_script' (skills/data/tao-analyze-gaps-vlm-bcq/SKILL.md)
MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (skills/data/tao-analyze-gaps-vlm-bcq/SKILL.md)
MEDIUM SCHEMA/frontmatter_field_placement: Root field 'tags' is ignored; use 'metadata.tags' (skills/data/tao-analyze-gaps-vlm-bcq/SKILL.md)
MEDIUM SCHEMA/folder_hierarchy: Unexpected nesting depth for general skill (skills/data/tao-analyze-gaps-vlm-bcq)
12 additional finding(s) are available in the full evaluation artifacts.

</details>

Scoring Methodology

<details> <summary>Show dimension definitions, source signals, and thresholds</summary>

Dimension	Question	Scored signals
Security	Is it safe to use?	`security` (100%)
Correctness	Is the answer correct?	`accuracy` (100%)
Discoverability	Was the right skill loaded when needed?	`skill_execution` (100%)
Effectiveness	Did the skill help complete the task?	`goal_accuracy` (50%) + `behavior_check` (50%)
Efficiency	Did it avoid wasted tool calls and token usage?	`skill_efficiency` (50%) + `token_efficiency` (50%)

Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
Effectiveness is the equal-weight mean of goal completion (goal_accuracy) and expected workflow adherence (behavior_check).
Efficiency is 50% tool-call productivity (the backward-compatible skill_efficiency wire id) and 50% token_efficiency. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.

Signals present in this run:

security (Security): unsafe operations, secret leakage, and unauthorized access.
skill_execution (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
skill_efficiency (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
accuracy (Accuracy): final-answer correctness against the reference answer.
goal_accuracy (Goal Accuracy): whether the user's goal was achieved.
behavior_check (Behavior Check): whether the expected workflow behavior was followed.
token_efficiency (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).

</details>

Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python

The author says: Requires Docker, NVIDIA Container Toolkit, and one visible GPU.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Corporation

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/tao-analyze-gaps-vlm-bcq · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-analyze-gaps-vlm-bcq.zip