I4h Workflow Dataset Replay

NVIDIAOfficialCreated by Isaac for Healthcare TeamDistributed by NVIDIA

Skill
Workflow
Automation and integrations
Apache-2.0
Medium risk

From the author

Replay a workflow HDF5 episode through its original Scene. Use for visual trajectory and recording verification; do not use for policy evaluation or LeRobot data.

— the description by Isaac for Healthcare Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured procedure for replaying recorded action sequences from HDF5 files through their corresponding workflow scenes. It is designed for visual verification of trajectories and recording integrity, ensuring that the recorded actions align with the expected workflow contracts. The process involves resolving the necessary repository assets, inspecting the recording metadata for compatibility, and executing the replay in a visible simulation environment.

What it does

Resolves and clones the required workflow repository based on environment configuration.
Inspects HDF5 recordings to verify segment data and action-contract compatibility.
Executes workflow replays in a foreground simulation environment.
Provides a framework for observing trajectories and identifying potential motion divergences.
Reports final status, including frame counts and action width verification.

When to use it

When you need to visually verify the accuracy of a recorded workflow trajectory.
When checking if a specific HDF5 recording matches the action-contract of a workflow scene.
When debugging or auditing the integrity of recorded episode data.
When you need to confirm that a recorded sequence executes as expected in the original environment.

Advantages

Ensures strict adherence to the original workflow and scene context.
Provides clear visibility into the execution of stored action sequences.
Facilitates systematic identification of mismatches between recorded data and scene contracts.
Standardizes the process for inspecting and replaying dataset episodes.

Usage examples

“Replay the second episode.”: The agent selects the current recording, maps the second episode to index 1, and initiates the visible replay in the simulator.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/i4h-workflow-dataset-replay/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip i4h-workflow-dataset-replay.zip -d .agents/skills

Documentation by Isaac for Healthcare Team

Replay a Workflow Recording

Purpose

Replay the exact recorded action sequence through the matching workflow and inspect it visibly.

Instructions

Run the checkout resolver and select the exact HDF5.
Read the original workflow and requested episode.
Inspect action-contract compatibility.
Run the replay visibly in the foreground through completion.

Resolve the recording

export I4H_WORKFLOWS_REPO_URL="${I4H_WORKFLOWS_REPO_URL:-https://github.com/isaac-for-healthcare/i4h-workflows}"
I4H_REPO_DIR_NAME="${I4H_WORKFLOWS_REPO_URL%/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*:}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME%.git}"
[ -n "$I4H_REPO_DIR_NAME" ] || { echo "Cannot derive a checkout name from I4H_WORKFLOWS_REPO_URL" >&2; exit 2; }
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/i4h_workflows" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/$I4H_REPO_DIR_NAME}"
  [ -d "$ROOT/workflows/i4h_workflows" ] || git clone "$I4H_WORKFLOWS_REPO_URL" "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"
cd "$ROOT"
find runs -name '*.hdf5' -type f -printf '%T@ %p\n' | sort -nr | head

Treat the resolver above as part of the skill contract: a hosted copy may run outside the base repository, so never assume the current checkout contains workflows/i4h_workflows. I4H_WORKFLOWS_REPO_URL selects the clone source. When I4H_WORKFLOWS is unset, derive the fallback directory from that URL; set I4H_WORKFLOWS only to reuse or choose a specific destination. Never replace an existing checkout.

Use an explicit or current-chain recording first. Otherwise select the newest plausible HDF5 and state that choice. Never substitute an older file after a failed recording.

Interpret natural ordinals as zero-based indices: first is 0, second is 1.

Inspect before launch

HDF5_PATH=/absolute/path/to/recording.hdf5
uv run --project tools/dataset i4h-dataset inspect "$HDF5_PATH" --segments

Resolve the original workflow from recording metadata and conversation context. Confirm the requested episode exists and its action width matches the workflow's replay Scene contract.

Replay

./run.sh <workflow> --replay "$HDF5_PATH" --episode <zero-based-index>

Keep the visible simulator and command in the foreground. Poll yielded execution until run.sh exits; do not detach or return while replay is still running.

Verify

Observe the complete trajectory, relevant objects, camera views, segment boundaries, and final status. If motion diverges, report the first mismatching segment or action-contract error. Do not change workflow, episode, or recording silently.

Troubleshooting

On launch failure, verify workflow metadata, episode existence, and action width. On divergence, report the first mismatching segment.

Prerequisites

Require the original workflow assets and a readable episode whose action width matches replay mode.

Limitations

Replay verifies stored actions in one Scene; it does not evaluate a learned policy or guarantee transfer to another Scene.

Examples

Replay the second episode. → select the current recording, map “second” to index 1, and observe the visible replay.

Completion gate

Report workflow, HDF5 path, episode index, frame/segment count, action width, exit/final status, and visible agreement or first mismatch.

Files

ATTRIBUTION.mdAdded by AutoGineer958 B
BENCHMARK.md6.8 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md3.8 KiB
evals/evals.json1.1 KiB
skill-card.md3.9 KiB
skill.oms.sig4.5 KiB

BENCHMARK.md

Skill Benchmark: i4h-workflow-dataset-replay

✅ Overall verdict: PASS — Recommended for publication

Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

Evaluation Metadata

Skill: i4h-workflow-dataset-replay
Evaluation date: 2026-09-21
Evaluator version: 1.5.6
Agents: Claude Code (aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5)
Tasks: 2 evaluation tasks (2 positive)
Dataset digest: sha256:bb1ca36274d90171e49695a74ff3163120054f3fe62cc2897af88e299c50455b (skill-evaluator-dataset-snapshot/1)
Attempts per task: 3
Environment: k8s-sandbox
Tier 2 evidence: required for publication
Tier 3 evidence: required for publication

Each task attempt ran in its own isolated sandbox pod.

What This Report Answers

The three-tier evaluation checks whether the skill:

is safe to use;
produces correct answers;
is discovered and activated when needed;
helps the agent complete the user's goal and expected workflow; and
avoids wasted skill and tool usage.

Results at a Glance

Measure	Claude Code (Baseline → Skill Uplift)	Codex (Baseline → Skill Uplift)
Overall	86.2% — baseline ran, but no comparable score was available; uplift unavailable	74.2% — baseline ran, but no comparable score was available; uplift unavailable
Security	100.0% → 100.0% (±0.0 points)	100.0% → 75.0% (-25.0 points)
Correctness	25.0% → 100.0% (+75.0 points)	13.3% → 90.0% (+76.7 points)
Discoverability	92.5% — baseline ran, but no comparable score was available; uplift unavailable	91.5% — baseline ran, but no comparable score was available; uplift unavailable
Effectiveness	3.1% → 50.0% (+46.9 points)	6.3% → 37.5% (+31.2 points)
Efficiency	88.5% — baseline ran, but no comparable score was available; uplift unavailable	77.1% — baseline ran, but no comparable score was available; uplift unavailable

How to read this table: baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Example: 47.0% → 92.0% (+45.0 points) means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.

Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

Agent	Dataset case	With skill	Without skill	Delta	Change	Coverage
claude-code	All cases	423,820	490,877	N/A	N/A	skill 2/2; base 4/4
claude-code	i4h-workflow-dataset-replay-readme-quick-second-episode	192,716	269,660	N/A	N/A	skill 1/1; base 3/3
claude-code	i4h-workflow-dataset-replay-table-episode-zero	231,104	221,217	+9,887	+4.47%	skill 1/1; base 1/1
codex	All cases	431,055	351,965	N/A	N/A	skill 2/2; base 6/6
codex	i4h-workflow-dataset-replay-readme-quick-second-episode	229,075	81,889	N/A	N/A	skill 1/1; base 3/3
codex	i4h-workflow-dataset-replay-table-episode-zero	201,980	270,076	N/A	N/A	skill 1/1; base 3/3
ALL AGENTS	Dataset aggregate	854,875	842,842	N/A	N/A	skill 4/4; base 10/10

Prompt tokens include cached reads, so total tokens are prompt + completion (cached is not added twice). The Efficiency score uses (prompt - cached) + completion. N/A means the relevant trajectory counters were not available; coverage is never estimated.

Tier Status

Tier	Purpose	Status	Evidence
Tier 1	Static validation	PASSED WITH OBSERVATIONS	11 validator(s); 2 finding(s)
Tier 2	Semantic deduplication	PASSED	2 validator(s); 0 finding(s)
Tier 3	Live agent evaluation	PASS	2 agent(s); 2 task(s)

Findings and Observations

<details> <summary>Show detailed findings and successful checks</summary>

MEDIUM SECURITY/Unknown (SDI-2): The skill clones a remote Git repository using a URL derived from the environment variable I4H_WORKFLOWS_REPO_URL. If (SKILL.md:33)
LOW SECURITY/Unknown (SQP-2): The resolver block silently performs a git clone of a remote URL into $HOME/<repo-name> when the checkout is not fou (SKILL.md:32)

</details>

Scoring Methodology

<details> <summary>Show dimension definitions, source signals, and thresholds</summary>

Dimension	Question	Scored signals
Security	Is it safe to use?	`security` (100%)
Correctness	Is the answer correct?	`accuracy` (100%)
Discoverability	Was the right skill loaded when needed?	`skill_execution` (100%)
Effectiveness	Did the skill help complete the task?	`goal_accuracy` (50%) + `behavior_check` (50%)
Efficiency	Did it avoid wasted tool calls and token usage?	`skill_efficiency` (50%) + `token_efficiency` (50%)

Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
Effectiveness is the equal-weight mean of goal completion (goal_accuracy) and expected workflow adherence (behavior_check).
Efficiency is 50% tool-call productivity (the backward-compatible skill_efficiency wire id) and 50% token_efficiency. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.

Signals present in this run:

security (Security): unsafe operations, secret leakage, and unauthorized access.
skill_execution (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
skill_efficiency (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
accuracy (Accuracy): final-answer correctness against the reference answer.
goal_accuracy (Goal Accuracy): whether the user's goal was achieved.
behavior_check (Behavior Check): whether the expected workflow behavior was followed.
token_efficiency (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).

</details>

Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Isaac for Healthcare Team

Publisher

NVIDIA (Official)

Distributed by

NVIDIA

Source

NVIDIA/skills · skills/i4h-workflow-dataset-replay · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Isaac for Healthcare Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download i4h-workflow-dataset-replay.zip