Physical AI Neural Reconstruction

OpenAIOfficialCreated by NVIDIA Physical AIDistributed by OpenAI

Skill
Meta (authoring, orchestration, planning, memory)
Data and AI
Apache-2.0
Medium risk

From the author

Router for NVIDIA NuRec/NRE: USDZ rendering, NCore conversion, 3DGS, gRPC sensor sim, PhysicalAI HF datasets. Do NOT use for SimReady or infra setup.

— the description by NVIDIA Physical AI

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The Physical AI Neural Reconstruction (NuRec) Router acts as a central navigation point for NVIDIA's NuRec ecosystem. It does not perform reconstruction tasks itself but identifies the correct upstream sibling skill to handle specific stages of the pipeline, such as data conversion, training, rendering, or asset harvesting. It ensures users follow the correct sequence of operations—such as converting raw sensor data to NCore V4 before training—and provides guidance on managing dependencies like NGC API keys and gated Hugging Face datasets.

What it does

Routes user requests to the appropriate upstream NuRec sibling skill.
Provides a standardized workflow for multi-stage neural reconstruction projects.
Facilitates the cloning and updating of the canonical nurec-skills repository.
Helps users identify the correct tool for sensor simulation, rendering, and dataset management.
Ensures proper handling of gated Hugging Face licenses and NGC container authentication.
Maintains order-of-operations for complex tasks like asset harvesting and frame harmonization.

When to use it

When you need to identify which NuRec skill to use for a specific task.
Before starting any neural reconstruction project to ensure the correct workflow sequence.
When you need to download or locate NVIDIA-published NuRec datasets.
When you are unsure whether to use the built-in NRE fixer or the standalone DiffusionHarmonizer.
When setting up a new environment for training or rendering 3D scenes from sensor recordings.

Advantages

Prevents misuse of incorrect tools by clearly defining the scope of each sibling skill.
Centralizes the complex dependency management for NVIDIA's neural reconstruction stack.
Provides a clear, authoritative path for multi-step workflows like data-to-render pipelines.
Reduces errors by enforcing correct execution order for conversion, training, and rendering.

Usage examples

“I have a raw LiDAR recording. How do I turn it into a 3D scene?”: The router identifies the ncore skill for conversion and the nre skill for training, directing the user to follow the ncore -> nre workflow.
“Where can I find the NVIDIA driving datasets for NuRec?”: The router points the user to the physical-ai-datasets skill and provides instructions on accepting the necessary gated licenses on Hugging Face.
“How do I clean up ghosting in my rendered frames?”: The router explains the difference between the built-in --enable-difix flag in nre and the standalone nurec-fixer skill, guiding the user to the best choice.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/openai/physical-ai-neural-reconstruction/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip physical-ai-neural-reconstruction.zip -d .agents/skills

Documentation by NVIDIA Physical AI

Physical AI Neural Reconstruction (NuRec) Router

Purpose

This is a thin router for NVIDIA Neural Reconstruction (NuRec) requests. It points at the upstream nurec-index skill at https://github.com/NVIDIA/nurec-skills and its five sibling skills (physical-ai-datasets, ncore, nre, asset-harvester, nurec-fixer). Use this skill to:

Identify which upstream sibling skill answers a NuRec question.
Locate, clone, or refresh the canonical nurec-skills checkout.
Order multi-step NuRec workflows (data → conversion → train → render → cleanup) before opening the upstream recipe.

The canonical recipes (training, rendering, data conversion, dataset downloads, object harvesting, frame cleanup) live in the upstream sibling skills. Never copy or reconstruct their commands here.

Do NOT use this skill for:

SimReady packaging of CAD or source meshes → use omniverse-cad-to-simready.
Generic USD performance tuning unrelated to NuRec → use omniverse-usd-performance-tuning.
AKS / OSMO / NIM Operator infrastructure setup → use physical-ai-infrastructure-setup-and-resilient-scaling.

When to Use

Read this skill first whenever a user mentions any of:

nurec, nurec router, nurec index, neural reconstruction, neural reconstruction engine, NRE, 3DGUT, 3DGRT, USDZ, NCore V4, sensor sim, novel view synthesis, PhysicalAI-Autonomous-Vehicles-NuRec, PhysicalAI-NuRec-PPISP, Cosmos-Drive-Dreams, asset harvester, nurec fixer, DiffusionHarmonizer, harmonizer, difix, difix3d, serve-grpc, render-grpc, warm serve-grpc, nre thin client, batch_render_rgb, nurec teardown, "where do I start with NuRec", "which NuRec skill should I use for X?".

Decide which upstream sibling skill answers the question, fetch it (see Locate and fetch the upstream skills), then follow that skill's body.

Prerequisites

Router skill itself has no runtime prerequisites beyond git for fetching the upstream. Downstream sibling skills require:

Docker + NVIDIA Container Toolkit + GPU — for nre, nre-tools, and nurec-fixer containers (nvcr.io/nvidia/nre/nre, nvcr.io/nvidia/nre/nre-tools, nvcr.io/nvidia/cosmos/cosmos-predict2-container:1.2).
NGC API key (NGC_API_KEY) — for pulling NGC containers.
Hugging Face token (HF_TOKEN) with the nvidia/PhysicalAI-*, nvidia/DiffusionHarmonizer, and nvidia/asset-harvester gated licenses accepted in advance on Hugging Face.
Python 3.10+ with huggingface_hub installed.
(Optional) CARLA, Isaac Sim 5.1, or AlpaSim for simulator integration over serve-grpc.

Verify secrets safely (do not echo values):

hf auth whoami
[ -n "${HF_TOKEN:-}" ]      && echo "HF_TOKEN length=${#HF_TOKEN}"      || echo "HF_TOKEN unset"
[ -n "${NGC_API_KEY:-}" ]   && echo "NGC_API_KEY length=${#NGC_API_KEY}" || echo "NGC_API_KEY unset"

See references/secrets-handling.md for the bash anti-patterns to avoid.

What is NuRec?

NuRec (NVIDIA Omniverse Neural Reconstruction) takes camera, LiDAR, radar, or stereo recordings — typically from a self-driving car or a robot — and turns them into a 3D scene you can re-render from any viewpoint. Names that come up a lot:

NRE — "Neural Reconstruction Engine". NuRec is the product; NRE is the engine that trains and renders. Both route to the upstream nre skill.
USDZ — the file format of a trained scene. A zip archive that Omniverse, Isaac Sim, and CARLA can open.
NCore V4 — the input format NRE consumes. Raw recordings must be converted to NCore V4 before training.
3DGUT / 3DGRT — the two 3D Gaussian Splatting flavours used internally by NRE. The default Hydra recipe picks one; most users never set it manually.

A typical NuRec project has three stages:

Get the input — convert your own recording to NCore V4 (ncore), or download a pre-converted dataset (physical-ai-datasets).
Train the reconstruction — feed NCore V4 to NRE; out comes a USDZ (nre).
Render new views — render images, videos, or LiDAR sweeps from the USDZ (nre).

Projects that just want to use an existing NVIDIA-published scene skip step 2.

Pick a skill

Match the user's goal in the left column and open the named upstream skill on the right. Arrows mean "do these in order".

I want to…	Upstream skill
Find or download a NuRec dataset NVIDIA has published	`physical-ai-datasets`
Convert my own camera / LiDAR / radar / depth / stereo recording into NCore V4	`ncore`
Write a new converter for an unsupported sensor setup (drone, RGB-D, ROS 2 bag, COLMAP, ScanNet++)	`ncore`
Train a 3D reconstruction from an NCore clip	`ncore` → `nre`
Generate the extra inputs NRE needs (segmentation masks, depth, ego mask)	`nre` (uses the `nre-tools` container)
Render a USDZ along the original camera positions	`nre`
Render at full resolution / highest quality	`nre` (see "Quality presets")
Render along a shifted trajectory (e.g. car moved 3 m left)	`nre`
Render through a server so CARLA / Isaac Sim / AlpaSim / a custom simulator can ask for frames	`nre` (`serve-grpc`)
Render the same USDZ many times back-to-back from Python with minimal per-call latency	`nre` (warm `serve-grpc` + thin Python client / `batch_render_rgb`)
Render LiDAR sweeps (point clouds) from a USDZ	`nre` (`render-grpc --lidar`)
Skip training and just render a NuRec scene NVIDIA already built	`physical-ai-datasets` → `nre`
Extract individual 3D objects (cars, pedestrians) from a driving clip	`asset-harvester`
Add, remove, or replace cars / pedestrians in a NuRec scene	`asset-harvester` → `nre`
Clean up or harmonize rendered frames (ghosting, floaters, flicker, lighting/shadows)	`nurec-fixer`, or `--enable-difix` inside `nre` for inline rendering
Export the scene as a PLY, mesh, depth maps, ego mask, etc.	`nre`
Upgrade an old USDZ so newer NRE versions load it faster	`nre` (`upgrade-artifact`)
Open a USDZ or PLY in a browser viewer	`nre` (`viewer` / `ply_viewer`)
Measure rendering quality (PSNR, SSIM, LPIPS) against ground truth	`nre` (`eval-rendering-metrics`)
Benchmark different reconstruction methods on the same scenes	`physical-ai-datasets` (`PhysicalAI-NuRec-PPISP`) → `nre`
Train on multiple GPUs or on SLURM	`nre` (Workflow D)

Common workflows

Six end-to-end workflows are documented in references/workflows.md:

A. Make a NuRec scene from your own recording.
B. Use a NuRec scene NVIDIA has already trained.
C. Add, remove, or replace 3D objects in a scene.
D. Clean up rendered frames.
E. Benchmark reconstruction quality.
F. Connect NuRec to a simulator.

Open that file when the user's task spans more than one sibling skill.

Sibling skills (upstream)

Name	Upstream folder	What it does
`physical-ai-datasets`	`.agents/skills/physical-ai-datasets/`	Catalog and download recipes for every NVIDIA Physical AI dataset on Hugging Face (driving, robotics, manipulation, NuRec scenes, benchmarks).
`ncore`	`.agents/skills/ncore/`	Converts any sensor recording to NCore V4 (the format NRE needs). Also covers writing a new converter.
`nre`	`.agents/skills/nre/`	The Neural Reconstruction Engine itself. Trains, renders (locally, via warm `serve-grpc` + thin Python client / `batch_render_rgb`, or to an external simulator), exports meshes / point clouds / depth, edits actors, evaluates quality.
`asset-harvester`	`.agents/skills/asset-harvester/`	Open-source Apache-2.0 pipeline that extracts individual 3D objects from sparse views in a driving clip and saves them as `.ply` Gaussian splats with metadata.
`nurec-fixer`	`.agents/skills/nurec-fixer/`	Standalone NVIDIA DiffusionHarmonizer workflow — public successor to the older Fixer / Difix3D+ recipes — that cleans rendered frames, harmonizes inserted actors, evaluates PSNR/LPIPS, and optionally fine-tunes the model.

For naming overlaps (NRE vs Fixer, ncore vs nre, AV-NuRec vs Cosmos-Drive-Dreams, NuRec vs SimReady) see references/mix-ups.md.

Locate and fetch the upstream skills

Quick recipe (full version in references/upstream-fetch.md):

UPSTREAM_ROOT="${NUREC_SKILLS_UPSTREAM_ROOT:-${PHYSICAL_AI_SKILL_HUB_UPSTREAM_ROOT:-$HOME/.physical-ai-skill-hub/upstreams}}"
mkdir -p "$UPSTREAM_ROOT"
if [ -d "$UPSTREAM_ROOT/nurec-skills/.git" ]; then
  git -C "$UPSTREAM_ROOT/nurec-skills" fetch --tags
  git -C "$UPSTREAM_ROOT/nurec-skills" checkout main
  git -C "$UPSTREAM_ROOT/nurec-skills" pull --ff-only
else
  git clone --depth 1 https://github.com/NVIDIA/nurec-skills.git \
    "$UPSTREAM_ROOT/nurec-skills"
fi
test -f "$UPSTREAM_ROOT/nurec-skills/.agents/skills/SKILL.md"

Then read the upstream skill before running any mutating command:

cat "$UPSTREAM_ROOT/nurec-skills/.agents/skills/SKILL.md"          # router
cat "$UPSTREAM_ROOT/nurec-skills/.agents/skills/<folder>/SKILL.md" # sibling

Local lookup order (try in order before the upstream clone):

.agents/skills/<name>/SKILL.md (Cursor, Codex, NemoClaw)
.claude/skills/<name>/SKILL.md (Claude Code)
.cursor/skills/<name>/SKILL.md (project-scoped)
~/.cursor/skills/<name>/SKILL.md (personal skills)

Hard Rules

Router only — do not duplicate upstream NuRec recipes here. Read the upstream sibling skill body before running any mutating command.
Refer to sibling skills by their name: (e.g. nre), not by repo path. Folder layouts can change; the name is portable.
Clone or refresh https://github.com/NVIDIA/nurec-skills under the shared upstream root (${NUREC_SKILLS_UPSTREAM_ROOT:-${PHYSICAL_AI_SKILL_HUB_UPSTREAM_ROOT:-$HOME/.physical-ai-skill-hub/upstreams}}/nurec-skills). Do not scan broad developer workspaces such as ~/Codes or reuse unrelated old clones.
physical-ai-datasets covers gated Hugging Face datasets. Do not bypass dataset license terms; the user must accept the PhysicalAI-* gated licenses on Hugging Face and provide a token before downloading.
Asset Harvester runs before packaging into a USDZ. Do not call nre's export-external-assets on hand-rolled .ply files unless the user explicitly asks to skip Asset Harvester.
For artifact cleanup, prefer the built-in --enable-difix path in nre. Route to the standalone nurec-fixer only when the user needs the public code/model card, paired evaluation, fine-tuning, or fixes on previously rendered frames.
Do not invent NRE / NCore / DiffusionHarmonizer commands from memory. Re-read the upstream sibling skill — versions move fast (NRE release_26.04 is the current pinned tag).
This router does not deploy infrastructure. Route AKS / OSMO / NIM Operator setup to physical-ai-infrastructure-setup-and-resilient-scaling.

Limitations

Router only. This skill never executes mutating NuRec commands. All training, rendering, conversion, and harmonization happens in upstream sibling skills.
Upstream-pinned. Recipes live in https://github.com/NVIDIA/nurec-skills, which evolves outside this repo. Stale clones can drift; always git pull the upstream before relying on a sibling skill.
Gated content. nvidia/PhysicalAI-*, nvidia/DiffusionHarmonizer, and nvidia/asset-harvester require the user to accept license terms on Hugging Face first. The router cannot bypass this.
Heavy footprint. A complete NuRec workflow can leave 150 GB+ on disk. See references/teardown.md.
NVIDIA-only stack. Requires an NVIDIA GPU plus the NVIDIA Container Toolkit. AMD / Intel / Apple Silicon are not supported.
Not a SimReady pipeline. NuRec produces a renderable USDZ from a recording; SimReady packaging of CAD or source meshes is a different pipeline (see omniverse-cad-to-simready).

Troubleshooting

Error / symptom	Likely cause	Solution
`nurec-skills` clone missing or empty	Upstream not fetched yet	Run the clone block in Locate and fetch the upstream skills
`403`/`401` pulling `nvidia/PhysicalAI-*` from HF	Gated license not accepted, or `HF_TOKEN` unset / wrong scope	Accept the gated license on Hugging Face, then `hf auth login` with a token that has `read` access
`denied: requested access to the resource is denied` from `nvcr.io/nvidia/nre/*`	Missing or expired `NGC_API_KEY`	`docker login nvcr.io` with `$oauthtoken` / `NGC_API_KEY`; rotate the key at `org.ngc.nvidia.com/setup/api-key` if needed
NRE refuses to load a clip ("not valid NCore V4")	Recording was not converted	Run the `ncore` skill before invoking `nre`
`serve-grpc` cold-start latency dominates a Python loop	One-shot Docker invocation per render	Use the `nre` warm `serve-grpc` + thin Python client (`batch_render_rgb`) recipe
Output files are owned by `root` after a `docker run`	`-u $(id -u):$(id -g)` was missing	`sudo chown -R "$(id -u):$(id -g)" <output_dir>`; add the `-u` flag next time
Frames have ghosting / floaters / flicker after rendering	Inline cleanup not enabled	Re-render with `nre --enable-difix`, or post-process with `nurec-fixer` (DiffusionHarmonizer)
Stale skill names (`ncore-data-conversion`, old `nvidia/Fixer`) in agent output	Out-of-date cached skill	Update references to `ncore` and `nurec-fixer` (DiffusionHarmonizer); see `references/maintenance.md`
Bash anti-pattern `${HF_TOKEN:+yes}${HF_TOKEN:-no}` echoed token value	Misuse of bash parameter expansion	Rotate the token; use `hf auth whoami` or length-only checks (see `references/secrets-handling.md`)

Cross-skill teardown

A complete NuRec workflow can leave 150 GB+ on disk between container images, model weights, code clones, conda envs, and output directories. Each sibling skill has its own dedicated Teardown section — read them in the order documented in references/teardown.md when the user no longer needs the workflow.

Keeping this router up to date

Procedure for adding new sibling skills, renames, or upstream URL changes lives in references/maintenance.md. Treat the upstream nurec-index at https://github.com/NVIDIA/nurec-skills/blob/main/.agents/skills/SKILL.md as authoritative; this skill mirrors only the picker tables, the workflow ordering, and the upstream fetch recipe.

Files

ATTRIBUTION.mdAdded by AutoGineer1.1 KiB
BENCHMARK.md3.0 KiB
LICENSEAdded by AutoGineer10.0 KiB
SKILL.md16.3 KiB
agents/openai.yaml236 B
evals/evals.json4.9 KiB
references/maintenance.md1.2 KiB
references/mix-ups.md2.1 KiB
references/secrets-handling.md1.0 KiB
references/teardown.md1.6 KiB
references/upstream-fetch.md2.3 KiB
references/workflows.md4.2 KiB
skill-card.md2.9 KiB
skill.oms.sig5.7 KiB

evals/evals.json

[
  {
    "id": "physical-ai-neural-reconstruction-001",
    "question": "I need help with the physical-ai-neural-reconstruction skill. Which sibling skill should I use to download the PhysicalAI-Autonomous-Vehicles-NuRec dataset from Hugging Face?",
    "expected_skill": "physical-ai-neural-reconstruction",
    "expected_script": null,
    "ground_truth": "The agent used the physical-ai-neural-reconstruction router skill to identify that the physical-ai-datasets sibling skill handles Hugging Face dataset downloads, and directed the user to that upstream skill with instructions on how to fetch it.",
    "expected_behavior": [
      "The agent read the physical-ai-neural-reconstruction SKILL.md to understand the routing structure",
      "The agent identified physical-ai-datasets as the correct sibling skill for downloading PhysicalAI-Autonomous-Vehicles-NuRec",
      "The agent provided guidance on locating or cloning the upstream nurec-skills repository to access the physical-ai-datasets skill",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "physical-ai-neural-reconstruction-002",
    "question": "I have a set of captured driving scenes and I want to convert them to NCore V4 format, then train a 3D Gaussian Splatting model, and finally render novel views as USDZ. Can you help me figure out the right workflow order?",
    "expected_skill": "physical-ai-neural-reconstruction",
    "expected_script": null,
    "ground_truth": "The agent used the physical-ai-neural-reconstruction router to decompose the multi-step NuRec workflow into the correct order (data → ncore conversion → 3DGS training → USDZ rendering) and identified which sibling skills (physical-ai-datasets, ncore, nre) handle each step.",
    "expected_behavior": [
      "The agent read the physical-ai-neural-reconstruction SKILL.md to understand the multi-step workflow ordering",
      "The agent outlined the workflow sequence: data acquisition, NCore conversion, training, and rendering",
      "The agent mapped each workflow step to the appropriate sibling skill (physical-ai-datasets, ncore, nre)",
      "The agent explained how to fetch the upstream nurec-skills repository for the detailed recipes",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "physical-ai-neural-reconstruction-003",
    "question": "We're building a sensor simulation pipeline for our autonomous vehicle stack. We need to serve neural reconstructions over gRPC so that our CARLA-based test harness can query novel viewpoints in real time. How do I set up the serve-grpc endpoint and what NuRec components are involved?",
    "expected_skill": "physical-ai-neural-reconstruction",
    "expected_script": null,
    "ground_truth": "The agent used the physical-ai-neural-reconstruction router to identify that serve-grpc / render-grpc functionality belongs to the nre sibling skill, explained the relationship to CARLA simulator integration, and directed the user to the upstream nre skill for the canonical setup recipe.",
    "expected_behavior": [
      "The agent read the physical-ai-neural-reconstruction SKILL.md and recognized serve-grpc as a trigger keyword",
      "The agent identified the nre sibling skill as responsible for gRPC-based sensor simulation serving",
      "The agent mentioned the requirement for Docker, NVIDIA Container Toolkit, and GPU for the nre container",
      "The agent directed the user to clone or locate the upstream nurec-skills repository for the detailed serve-grpc recipe",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  },
  {
    "id": "physical-ai-neural-reconstruction-004",
    "question": "I have a CAD assembly exported from SolidWorks and I need to convert it into a SimReady USD asset with proper physics and material annotations for use in Isaac Sim. What's the best approach?",
    "expected_skill": null,
    "expected_script": null,
    "ground_truth": "The agent correctly recognized that SimReady packaging of CAD meshes is explicitly excluded from the physical-ai-neural-reconstruction skill and routed the user to the omniverse-cad-to-simready skill instead.",
    "expected_behavior": [
      "The agent did not invoke or route through the physical-ai-neural-reconstruction skill for this request",
      "The agent identified that SimReady CAD conversion is handled by a different skill such as omniverse-cad-to-simready",
      "The agent provided guidance appropriate to CAD-to-SimReady workflows rather than neural reconstruction",
      "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
    ]
  }
]

Requirements and compatibility

Written for: OpenAI Codex

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs a command-line tool

The author says: Router skill; downstream sibling skills require Docker, NVIDIA Container Toolkit, GPU, NGC API key, Hugging Face token with PhysicalAI gated licenses accepted, Python 3.10+, and `huggingface_hub`. Optional: CARLA / Isaac Sim 5.1 / AlpaSim for simulator integration over `serve-grpc`.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Physical AI

Publisher

OpenAI (Official)

Distributed by

OpenAI

Source

openai/plugins · plugins/nvidia/skills/physical-ai-neural-reconstruction · commit d420d13

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (written by AutoGineer)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by NVIDIA Physical AI or OpenAI.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download physical-ai-neural-reconstruction.zip