Omniverse Realtime Viewer
OpenAIOfficialCreated by NVIDIA OmniverseDistributed by OpenAI
- Skill
- Meta (authoring, orchestration, planning, memory)
- Software development
- Apache-2.0
- Medium risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The Omniverse Realtime Viewer skill acts as a central orchestrator for developing USD-based viewer applications. It provides a structured approach to selecting implementation recipes, enforcing architectural conventions, and managing dependencies. By utilizing a set of focused reference documents, it guides developers through the lifecycle of building, configuring, and validating viewer applications, ensuring consistent behavior across different delivery paths such as local workstations or streaming environments.
What it does
- Routes development requests to specific implementation recipes based on the target delivery path.
- Enforces architectural rules for USD rendering, stage management, and UI integration.
- Provides guidance on camera selection, input handling, and viewport interaction.
- Standardizes the integration of AOV switching and render product configuration.
- Facilitates the capture of validation evidence for viewer readiness.
- Maintains separation between user-provided USD files and application-specific session state.
When to use it
- When initiating the development of a new USD viewer application within the Omniverse ecosystem.
- When determining the appropriate delivery path for a viewer, such as local native or streaming.
- When needing to implement consistent camera, selection, or UI behavior across viewer projects.
- When integrating advanced rendering features like AOV switching into an existing viewer.
Advantages
- Ensures architectural consistency across diverse viewer implementations.
- Provides a clear, modular path for adding specific capabilities like input routing or data display.
- Promotes clean separation of concerns between stage data and viewer-owned state.
- Standardizes the validation process for consistent delivery.
Usage examples
- “I need to build a browser-based viewer for USD files.”
- The agent will route you to the streaming viewer recipe, providing guidance on WebRTC stream setup, UI integration, and viewport interaction.
- “How do I implement AOV switching in my local USD viewer?”
- The agent will provide the AOV switching reference, detailing how to map render vars and update the display state on the server.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/openai/omniverse-realtime-viewer/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip omniverse-realtime-viewer.zip -d .agents/skills
Documentation by NVIDIA Omniverse
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->
Omniverse Realtime Viewer
This is the top-level entry point for the Omniverse Realtime Viewer skill package. It is self-contained: all required routing, conventions, and validation guidance live in the selected references.
Use the focused reference documents as implementation recipes. This file chooses the right recipes and preserves the architectural rules that must hold across all generated viewer apps.
Instructions
Start by classifying the requested viewer, then read only the references needed
for that delivery path and feature set. Implement the render path first, layer
interaction and UI behavior on top of it, and finish by capturing validation
evidence from references/validation.md.
Read Order
Read
references/routing.mdto choose the delivery path and focused references.Read
references/conventions.mdbefore implementing camera, input, selection, viewport, streaming protocol, scene loading, or environment behavior.For broad viewer requests, read
references/usd-viewer-app/README.md.If the delivery path is unclear, read
references/streaming-vs-local/README.md.If the prompt includes layout, panels, controls, inspectors, status, or UX, read
references/viewer-ux-workflow/README.mdand then the focused viewer UI references. This applies to React/WebRTC, Tauri, Electron,ovui,ovwidgets, and Dear ImGui apps; "frontend" means user-facing UI, not only browser UI.For viewport interaction, read
references/viewer-input-routing/README.mdbeforereferences/camera-controls/README.md,references/native-picking-selection/README.md, orreferences/object-selection/README.md.Read only the focused capability references needed for the requested app.
Use
references/validation.mdto capture review evidence before handoff.
Non-Negotiables
Use
ovrtxfor all USD and 3D rendering.Browser apps display an
ovstreamWebRTC video stream plus UI. The browser does not render USD geometry.Do not substitute WebGL, Three.js, Babylon.js, PlayCanvas, A-Frame, model-viewer, react-three-fiber, glTF browser viewers, or other client-side 3D renderers.
If local validation cannot run because the GPU/runtime environment is absent, scaffold the
ovrtxpath and document the runtime requirement. Do not add a browser-renderer fallback.Keep user USD files unmodified. Viewer cameras, render products, render vars, settings, selection metadata, and runtime state belong in session/composite layers or app state.
Keep one owner for
renderer.step(), stage mutation, native picking, selection writes, and live attribute writes.Keep dependency acquisition in
references/dependencies/README.mdand deployment choices inreferences/cloud-deployment/README.md; do not duplicate package locations or deployment setup.
Focused Reference Families
Entry points and recipes:
references/usd-viewer-app/README.md,references/streaming-viewer-recipe/README.md,references/ovui-local-viewer-recipe/README.md,references/streaming-vs-local/README.md,references/electron-shm-viewer/README.md,references/ovwidgets-editor-shell/README.md.Rendering and stage:
references/ovrtx-rendering/README.md,references/stage-loading/README.md,references/stage-management/README.md,references/render-settings/README.md,references/aov-switching/README.md,references/stage-hierarchy/README.md,references/stage-queries/README.md,references/stage-attribute-reads/README.md,references/prim-transform-safety/README.md,references/usd-sample-data/README.md.Delivery and runtime:
references/streaming-server/README.md,references/streaming-client/README.md,references/streaming-messages/README.md,references/streaming-lifecycle/README.md,references/local-viewer/README.md,references/tauri-local-viewer/README.md,references/cpp-native-viewer/README.md,references/headless-shm-cli/README.md,references/viewer-backend-interface/README.md,references/webgl-shm-transport/README.md.Viewer UI/UX:
references/viewer-ux-workflow/README.md,references/viewer-layout-patterns/README.md,references/viewer-control-patterns/README.md,references/viewer-data-view-patterns/README.md,references/viewer-feedback-status/README.md.Interaction:
references/viewer-input-routing/README.md,references/camera-controls/README.md,references/object-selection/README.md,references/native-picking-selection/README.md,references/selection-feedback/README.md,references/selection-animation/README.md,references/transform-manipulator/README.md,references/gl-viewport-overlay/README.md,references/ovui-library/README.md,references/prim-pick-effects/README.md,references/prim-info-display/README.md,references/viewport-overlays/README.md.Infrastructure:
references/dependencies/README.md,references/windows-native-setup/README.md,references/cloud-assets/README.md,references/cloud-deployment/README.md,references/troubleshooting/README.md.
Build Workflow
Classify the prompt by delivery path, target user, required capabilities, runtime environment, validation needs, and explicit constraints.
Select a small reference set. Start with the recipe or routing reference, then add focused capabilities such as camera, picking, hierarchy, properties, render settings, transform tools, cloud assets, or deployment.
Read selected references before writing app code. Follow their build order, import order, data-channel contracts, and renderer ownership rules.
Implement the core render path first, then input routing and camera, then selection and data panels, then scene/settings features, then packaging or deployment.
Treat the selected references as the behavior contract for API shape, compatibility, and generated project structure.
Capture validation evidence before calling the viewer ready.
Examples
For a browser viewer request, use the streaming recipe references plus camera, picking, hierarchy, properties, render settings, and stream-status references.
For a local workstation viewer request, use the local or native delivery references plus renderer setup, stage loading, viewport input, and validation.
Completion Checklist
Selected references match the user's intent and delivery path.
No code path uses a browser-side 3D renderer for USD.
The generated app has one clear owner for render stepping and stage mutation.
User USD files remain untouched by viewer-owned session data.
Camera, input, selection, scene loading, and stream behavior follow
references/conventions.md.Setup/build/run results and visual interaction evidence are captured with
references/validation.md.
Files
- ATTRIBUTION.mdAdded by AutoGineer1.0 KiB
- BENCHMARK.md5.5 KiB
- LICENSEAdded by AutoGineer10.0 KiB
- SKILL.md7.4 KiB
- agents/openai.yaml315 B
- evals/evals.json8.1 KiB
- references/aov-switching/README.md15.2 KiB
- references/camera-auto-select/README.md7.4 KiB
- references/camera-controls/README.md17.1 KiB
- references/camera-picker/README.md9.7 KiB
- references/cloud-assets/README.md10.4 KiB
- references/cloud-deployment/README.md18.0 KiB
- references/conventions.md5.3 KiB
- references/cpp-native-viewer/README.md2.4 KiB
- references/cpp-native-viewer/interaction-features.md19.2 KiB
- references/cpp-native-viewer/presentation-loop.md9.0 KiB
- references/cpp-native-viewer/project-build.md6.4 KiB
- references/cpp-native-viewer/renderer-session.md4.5 KiB
- references/cpp-native-viewer/validation.md3.3 KiB
- references/dependencies/README.md3.4 KiB
- references/dependencies/electron-shm.md1.5 KiB
- references/dependencies/environment-validation.md4.4 KiB
- references/dependencies/frontend.md2.3 KiB
- references/dependencies/local-openusd-gpu.md4.4 KiB
- references/dependencies/nvidia-runtime.md9.0 KiB
- references/dependencies/ovrtx.md2.4 KiB
- references/dependencies/ovstream.md2.5 KiB
- references/dependencies/quick-setup.md5.3 KiB
- references/electron-shm-viewer/README.md2.2 KiB
- references/electron-shm-viewer/architecture-project.md6.3 KiB
- references/electron-shm-viewer/electron-client.md5.6 KiB
- references/electron-shm-viewer/protocol-interaction-lifecycle.md6.9 KiB
- references/electron-shm-viewer/python-shm-server.md4.9 KiB
- references/electron-shm-viewer/validation.md2.8 KiB
- references/gl-viewport-overlay/README.md7.9 KiB
- references/gl-viewport-overlay/architecture.md4.1 KiB
- references/gl-viewport-overlay/example-gizmo.md5.5 KiB
- references/gl-viewport-overlay/interaction-pattern.md4.0 KiB
- references/gl-viewport-overlay/projection-alignment.md4.3 KiB
- references/headless-shm-cli/README.md12.2 KiB
- references/huggingface-usd/README.md7.7 KiB
- references/local-viewer/README.md15.0 KiB
- references/native-picking-selection/README.md10.4 KiB
- references/object-selection/README.md9.4 KiB
- references/ovrtx-rendering/README.md18.2 KiB
- references/ovui-library/README.md3.3 KiB
- references/ovui-library/api-reference.md4.3 KiB
- references/ovui-library/extending.md4.8 KiB
- references/ovui-local-viewer-recipe/README.md3.7 KiB
- references/ovui-local-viewer-recipe/interaction-features.md16.1 KiB
- references/ovui-local-viewer-recipe/project-structure.md6.2 KiB
- references/ovui-local-viewer-recipe/setup-shell-renderer.md14.3 KiB
- references/ovui-local-viewer-recipe/validation-build-order.md5.5 KiB
- references/ovwidgets-editor-shell/README.md8.6 KiB
- references/prim-info-display/README.md9.3 KiB
- references/prim-pick-effects/README.md5.9 KiB
- references/prim-transform-safety/README.md8.2 KiB
- references/render-settings/README.md9.9 KiB
- references/routing.md16.5 KiB
- references/seg-outline-highlight/README.md1.4 KiB
- references/selection-animation/README.md9.7 KiB
- references/selection-feedback/README.md6.6 KiB
- references/stage-attribute-reads/README.md5.5 KiB
- references/stage-hierarchy/README.md17.1 KiB
- references/stage-hierarchy/fallback-worker-protocol.md3.5 KiB
- references/stage-loading/README.md19.3 KiB
- references/stage-management/README.md10.1 KiB
- references/stage-queries/README.md5.3 KiB
- references/streaming-client/README.md17.4 KiB
- references/streaming-lifecycle/README.md8.8 KiB
- references/streaming-messages/README.md16.5 KiB
- references/streaming-messages/server-handler-map.md3.1 KiB
- references/streaming-server/README.md18.6 KiB
- references/streaming-server/frame-loop-and-continuity.md3.3 KiB
- references/streaming-viewer-recipe/README.md4.8 KiB
- references/streaming-viewer-recipe/client-protocol.md11.0 KiB
- references/streaming-viewer-recipe/interaction-features.md13.0 KiB
- references/streaming-viewer-recipe/project-structure.md7.0 KiB
- references/streaming-viewer-recipe/server-runtime.md16.8 KiB
- references/streaming-viewer-recipe/validation-build-order.md5.8 KiB
- references/streaming-vs-local/README.md8.1 KiB
- references/tauri-local-viewer/README.md18.2 KiB
- references/tauri-shm-transform-gizmo/README.md11.8 KiB
- references/transform-manipulator/README.md15.0 KiB
- references/troubleshooting/README.md7.2 KiB
- references/troubleshooting/scenario-playbooks.md18.8 KiB
- references/usd-sample-data/README.md11.7 KiB
- references/usd-viewer-app/README.md7.5 KiB
- references/validation.md3.0 KiB
- references/viewer-backend-interface/README.md17.7 KiB
- references/viewer-control-patterns/README.md4.8 KiB
- references/viewer-data-view-patterns/README.md4.8 KiB
- references/viewer-feedback-status/README.md4.9 KiB
- references/viewer-input-routing/README.md7.8 KiB
- references/viewer-layout-patterns/README.md4.7 KiB
- references/viewer-ux-workflow/README.md4.6 KiB
- references/viewport-overlays/README.md4.8 KiB
- references/viewport-resize/README.md8.5 KiB
- references/webgl-shm-transport/README.md14.5 KiB
- references/windows-native-setup/README.md5.7 KiB
- skill-card.md2.9 KiB
- skill.oms.sig28.6 KiB
references/troubleshooting/README.md
Troubleshooting
Triggers
Use this skill for troubleshoot, debug Omniverse Realtime Viewer, server won't start, no video, data channel not working, wrong colors, black frame, scene won't load, camera doesn't move, picking broken, or WebRTC internals.
Use this when an Omniverse Realtime Viewer fails during startup, streaming, scene loading, rendering, input, selection, or UI state sync. Start by classifying the first broken boundary instead of chasing downstream symptoms.
Triage Flowchart
Server process does not start
-> Server startup diagnostics
Server starts but browser cannot connect
-> WebRTC signaling and port diagnostics
Browser connects but video is blank or frozen
-> Video streaming and renderer step diagnostics
Electron SHM viewer opens but viewport is black, disconnected, or stutters
-> Electron SHM Viewer diagnostics
Video streams but colors/materials/frame are wrong
-> Render var, BGRA conversion, camera, MDL diagnostics
UI connects but buttons/tree/settings do nothing
-> Data-channel diagnostics
Stage open fails or loads as empty
-> Scene loading and asset path diagnostics
Scene loads but camera does not move
-> Input callback and live camera write diagnostics
Camera works but click selection is wrong or empty
-> Picking, coordinate, and segmentation diagnostics
Selection works but highlight/info/tree is stale
-> Derived state reset and message routing diagnostics
Scene switch crashes or hangs
-> Renderer ownership and stage reset diagnosticsFast Rules
Debug one boundary at a time: process startup, WebRTC connection, rendered frame, JSON message, USD query, then feature state.
Keep one render thread as the owner of
renderer.step(),reset_stage(),open_usd(),open_usd_from_string(), reference mutation, native pick queries, selection outline writes, and livewrite_attribute()calls.In WebRTC streaming apps, mouse, wheel, keyboard, and touch input arrive through NVST/ovstream
InputEventcallbacks; app state commands use the JSON data channel.A frontend "loading forever" state is usually either a missing
openStageResult, a missed proactive state push, or a message-name mismatch.A local Omniverse Realtime Viewer skips WebRTC entirely. If the same renderer and scene code works locally but not in a browser, focus on ovstream, frame conversion, and the standalone ovstream Direct AppStreamer config.
Scenario Playbooks
Detailed startup, streaming, scene, input, selection, hierarchy, and recovery playbooks live in scenario-playbooks.md.
Common Error Map
| Message or symptom | Actual cause | Usual fix |
|---|---|---|
ModuleNotFoundError: ovstream | Python binding missing or native lib path missing | Install ovstream and set OVSTREAM_LIB_PATH |
CRenderApi not found | ovrtx plugin tree not resolved | Set OVRTX_BIN_PATH and plugin library path |
usd-core detected | ovrtx USD check found another USD package | Set OVRTX_SKIP_USD_CHECK=1 before ovrtx work |
multiple debug symbol definitions for SDF_ASSET | Two USD registries loaded | Put ovrtx bundled libs first or split pxr into worker |
_tf import failure | USD DLL/shared library conflict | Fix import order or use subprocess queries |
Default.mdl parse crash | Renderer initialized after wrong USD registry | Fix import/construction order |
| Magenta materials | MDL resolver path missing | Set OVRTX_BIN_PATH and library path |
Unable to find RenderProduct prim | Inline/session render path missing or mismatched | Create the render pipeline and pass the exact path |
| Black frame, no exception | Camera, RenderProduct, resolution, or RenderVar invalid | Validate stage-loading data and camera relation |
USD parse error near RenderVar inline braces | ovrtx parser rejected one-line def RenderVar "X" { ... } syntax | Use multi-line RenderVar definitions from stage-loading |
RenderProductSetOutputs has no attribute get | renderer.step() output was treated as a dict | Use with products as ctx: and index with ctx[render_product_path] |
| Invalid output handle | Frame or render var view outlived its step result | Copy buffers before leaving the RenderProductSetOutputs context |
First renderer.step() exceeds normal test timeout | Cold RTX shader or pipeline compilation | Use a 300s+ first-run timeout and inspect ovrtx logs |
| Red/blue swapped | RGBA submitted as BGRA | Convert ovrtx LdrColor before ovstream |
| Stage load reports success but first frame fails | load operation status or RenderProduct path was not checked | wait/check load status, then step the exact RenderProduct |
TypeError: a coroutine was expected from ui.run | ovui run loop received a callback/function instead of an awaitable | Pass an async render loop coroutine and yield with await asyncio.sleep(0) |
VIEWPORT_CAMERA_POSE_SOURCE import failure | stale data adapters installed with newer local UI packages | Install local UI packages from the same package set |
ovui-data-adapters is not installable | selected package set lacks matching package metadata | Use a compatible package set from references/dependencies |
| Native UI package requires a compiler toolchain | package/build instructions require local tools | Follow the current ovui dependency guidance |
Previous session is already running | Old WebRTC client/session still active | Close old tab, reduce reconnect storm, restart server if stuck |
Server sees messageType but no event_type | AppStreamer envelope not unwrapped | Parse nested data payload before dispatch |
POST /sign_in returns HTTP 501 | Frontend used a Kit/OVC/NVCF/GFN client profile or injected auth/session fields into standalone ovstream Direct config | Rebuild the frontend from streaming-client using @nvidia/ov-web-rtc Direct mode with only the exposed server and signalingPort |
UI waits on getChildrenResponse | Protocol name mismatch | Use active getChildrenResult route |
| Picks return old prims | pending pick/selectable state survived scene reload | clear pick state and refresh native pickability |
| Highlight visible on load | native selection outline groups not cleared | write group 0 for stale selected paths and clear runtime selection |
| Textures missing only after composition | cache path or sublayer path broke relative references | preserve cache layout and quote the original asset path correctly |
Streaming And Local Omniverse Realtime Viewer Paths
Streaming path:
Use
streaming-serverfor ovstream lifecycle, ports, input callbacks, and frame submission.Use
streaming-clientfor standalone ovstream Direct config, video element setup, browser diagnostics, and guarded sends.Use
streaming-lifecyclewhen the connection exists but state, messages, or reconnects are wrong.Use
streaming-messagesto verify exact JSON event names and payload shapes.
Local Omniverse Realtime Viewer path:
Use
local-viewerfor ovui shell, image display, UI-thread rules, and coordinate mapping.Use
ovrtx-renderingfor renderer construction, frame extraction, and live attribute writes.Use
stage-loadingfor render prim injection and RenderProduct failures.Use
viewer-input-routing,camera-controls,object-selection,stage-hierarchy, andprim-info-displayfor feature-specific debugging once the frame renders correctly.
Requirements and compatibility
Written for: OpenAI Codex
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
The author says: Orchestrator skill. Downstream focused references may require NVIDIA GPUs, ovrtx, ovstream, ovui, OpenUSD, Python, Node/React, Tauri, Electron, C++, or cloud GPU deployment access depending on the selected viewer path.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
- Instructions that use the network
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA Omniverse
- Publisher
- OpenAI (Official)
- Distributed by
- OpenAI
- Source
- openai/plugins · plugins/nvidia/skills/omniverse-realtime-viewer · commit d420d13
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (written by AutoGineer)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA Omniverse or OpenAI.
Related skills
Setup TS Deep Modules
Matt Pocock· Community
Configures dependency-cruiser to enforce deep module boundaries in TypeScript repositories.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository317.5k
To Spec
Matt Pocock· Community
Synthesizes conversation context into a structured technical specification and publishes it to a project issue tracker.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Improve Codebase Architecture
Matt Pocock· Community
A systematic process to identify, visualize, and refine architectural deepening opportunities in a codebase.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Domain Modeling
Matt Pocock· Community
An expert assistant for defining project terminology, maintaining a glossary, and documenting architectural decisions.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Codebase Design
Matt Pocock· Community
A shared vocabulary and set of principles for designing deep, testable, and maintainable software modules.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Prototype
Matt Pocock· Community
Build a throwaway prototype to answer a design question. Use when the user wants to sanity-check whether a state model or logic feels right, or explore what a UI should look like.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download omniverse-realtime-viewer.zip