Omniverse Realtime Viewer
OpenAIOfficialCreated by NVIDIA OmniverseDistributed by OpenAI
- Skill
- Meta (authoring, orchestration, planning, memory)
- Software development
- Apache-2.0
- Medium risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The Omniverse Realtime Viewer skill acts as a central orchestrator for developing USD-based viewer applications. It provides a structured approach to selecting implementation recipes, enforcing architectural conventions, and managing dependencies. By utilizing a set of focused reference documents, it guides developers through the lifecycle of building, configuring, and validating viewer applications, ensuring consistent behavior across different delivery paths such as local workstations or streaming environments.
What it does
- Routes development requests to specific implementation recipes based on the target delivery path.
- Enforces architectural rules for USD rendering, stage management, and UI integration.
- Provides guidance on camera selection, input handling, and viewport interaction.
- Standardizes the integration of AOV switching and render product configuration.
- Facilitates the capture of validation evidence for viewer readiness.
- Maintains separation between user-provided USD files and application-specific session state.
When to use it
- When initiating the development of a new USD viewer application within the Omniverse ecosystem.
- When determining the appropriate delivery path for a viewer, such as local native or streaming.
- When needing to implement consistent camera, selection, or UI behavior across viewer projects.
- When integrating advanced rendering features like AOV switching into an existing viewer.
Advantages
- Ensures architectural consistency across diverse viewer implementations.
- Provides a clear, modular path for adding specific capabilities like input routing or data display.
- Promotes clean separation of concerns between stage data and viewer-owned state.
- Standardizes the validation process for consistent delivery.
Usage examples
- “I need to build a browser-based viewer for USD files.”
- The agent will route you to the streaming viewer recipe, providing guidance on WebRTC stream setup, UI integration, and viewport interaction.
- “How do I implement AOV switching in my local USD viewer?”
- The agent will provide the AOV switching reference, detailing how to map render vars and update the display state on the server.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/openai/omniverse-realtime-viewer/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip omniverse-realtime-viewer.zip -d .agents/skills
Documentation by NVIDIA Omniverse
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->
Omniverse Realtime Viewer
This is the top-level entry point for the Omniverse Realtime Viewer skill package. It is self-contained: all required routing, conventions, and validation guidance live in the selected references.
Use the focused reference documents as implementation recipes. This file chooses the right recipes and preserves the architectural rules that must hold across all generated viewer apps.
Instructions
Start by classifying the requested viewer, then read only the references needed
for that delivery path and feature set. Implement the render path first, layer
interaction and UI behavior on top of it, and finish by capturing validation
evidence from references/validation.md.
Read Order
Read
references/routing.mdto choose the delivery path and focused references.Read
references/conventions.mdbefore implementing camera, input, selection, viewport, streaming protocol, scene loading, or environment behavior.For broad viewer requests, read
references/usd-viewer-app/README.md.If the delivery path is unclear, read
references/streaming-vs-local/README.md.If the prompt includes layout, panels, controls, inspectors, status, or UX, read
references/viewer-ux-workflow/README.mdand then the focused viewer UI references. This applies to React/WebRTC, Tauri, Electron,ovui,ovwidgets, and Dear ImGui apps; "frontend" means user-facing UI, not only browser UI.For viewport interaction, read
references/viewer-input-routing/README.mdbeforereferences/camera-controls/README.md,references/native-picking-selection/README.md, orreferences/object-selection/README.md.Read only the focused capability references needed for the requested app.
Use
references/validation.mdto capture review evidence before handoff.
Non-Negotiables
Use
ovrtxfor all USD and 3D rendering.Browser apps display an
ovstreamWebRTC video stream plus UI. The browser does not render USD geometry.Do not substitute WebGL, Three.js, Babylon.js, PlayCanvas, A-Frame, model-viewer, react-three-fiber, glTF browser viewers, or other client-side 3D renderers.
If local validation cannot run because the GPU/runtime environment is absent, scaffold the
ovrtxpath and document the runtime requirement. Do not add a browser-renderer fallback.Keep user USD files unmodified. Viewer cameras, render products, render vars, settings, selection metadata, and runtime state belong in session/composite layers or app state.
Keep one owner for
renderer.step(), stage mutation, native picking, selection writes, and live attribute writes.Keep dependency acquisition in
references/dependencies/README.mdand deployment choices inreferences/cloud-deployment/README.md; do not duplicate package locations or deployment setup.
Focused Reference Families
Entry points and recipes:
references/usd-viewer-app/README.md,references/streaming-viewer-recipe/README.md,references/ovui-local-viewer-recipe/README.md,references/streaming-vs-local/README.md,references/electron-shm-viewer/README.md,references/ovwidgets-editor-shell/README.md.Rendering and stage:
references/ovrtx-rendering/README.md,references/stage-loading/README.md,references/stage-management/README.md,references/render-settings/README.md,references/aov-switching/README.md,references/stage-hierarchy/README.md,references/stage-queries/README.md,references/stage-attribute-reads/README.md,references/prim-transform-safety/README.md,references/usd-sample-data/README.md.Delivery and runtime:
references/streaming-server/README.md,references/streaming-client/README.md,references/streaming-messages/README.md,references/streaming-lifecycle/README.md,references/local-viewer/README.md,references/tauri-local-viewer/README.md,references/cpp-native-viewer/README.md,references/headless-shm-cli/README.md,references/viewer-backend-interface/README.md,references/webgl-shm-transport/README.md.Viewer UI/UX:
references/viewer-ux-workflow/README.md,references/viewer-layout-patterns/README.md,references/viewer-control-patterns/README.md,references/viewer-data-view-patterns/README.md,references/viewer-feedback-status/README.md.Interaction:
references/viewer-input-routing/README.md,references/camera-controls/README.md,references/object-selection/README.md,references/native-picking-selection/README.md,references/selection-feedback/README.md,references/selection-animation/README.md,references/transform-manipulator/README.md,references/gl-viewport-overlay/README.md,references/ovui-library/README.md,references/prim-pick-effects/README.md,references/prim-info-display/README.md,references/viewport-overlays/README.md.Infrastructure:
references/dependencies/README.md,references/windows-native-setup/README.md,references/cloud-assets/README.md,references/cloud-deployment/README.md,references/troubleshooting/README.md.
Build Workflow
Classify the prompt by delivery path, target user, required capabilities, runtime environment, validation needs, and explicit constraints.
Select a small reference set. Start with the recipe or routing reference, then add focused capabilities such as camera, picking, hierarchy, properties, render settings, transform tools, cloud assets, or deployment.
Read selected references before writing app code. Follow their build order, import order, data-channel contracts, and renderer ownership rules.
Implement the core render path first, then input routing and camera, then selection and data panels, then scene/settings features, then packaging or deployment.
Treat the selected references as the behavior contract for API shape, compatibility, and generated project structure.
Capture validation evidence before calling the viewer ready.
Examples
For a browser viewer request, use the streaming recipe references plus camera, picking, hierarchy, properties, render settings, and stream-status references.
For a local workstation viewer request, use the local or native delivery references plus renderer setup, stage loading, viewport input, and validation.
Completion Checklist
Selected references match the user's intent and delivery path.
No code path uses a browser-side 3D renderer for USD.
The generated app has one clear owner for render stepping and stage mutation.
User USD files remain untouched by viewer-owned session data.
Camera, input, selection, scene loading, and stream behavior follow
references/conventions.md.Setup/build/run results and visual interaction evidence are captured with
references/validation.md.
Files
- ATTRIBUTION.mdAdded by AutoGineer1.0 KiB
- BENCHMARK.md5.5 KiB
- LICENSEAdded by AutoGineer10.0 KiB
- SKILL.md7.4 KiB
- agents/openai.yaml315 B
- evals/evals.json8.1 KiB
- references/aov-switching/README.md15.2 KiB
- references/camera-auto-select/README.md7.4 KiB
- references/camera-controls/README.md17.1 KiB
- references/camera-picker/README.md9.7 KiB
- references/cloud-assets/README.md10.4 KiB
- references/cloud-deployment/README.md18.0 KiB
- references/conventions.md5.3 KiB
- references/cpp-native-viewer/README.md2.4 KiB
- references/cpp-native-viewer/interaction-features.md19.2 KiB
- references/cpp-native-viewer/presentation-loop.md9.0 KiB
- references/cpp-native-viewer/project-build.md6.4 KiB
- references/cpp-native-viewer/renderer-session.md4.5 KiB
- references/cpp-native-viewer/validation.md3.3 KiB
- references/dependencies/README.md3.4 KiB
- references/dependencies/electron-shm.md1.5 KiB
- references/dependencies/environment-validation.md4.4 KiB
- references/dependencies/frontend.md2.3 KiB
- references/dependencies/local-openusd-gpu.md4.4 KiB
- references/dependencies/nvidia-runtime.md9.0 KiB
- references/dependencies/ovrtx.md2.4 KiB
- references/dependencies/ovstream.md2.5 KiB
- references/dependencies/quick-setup.md5.3 KiB
- references/electron-shm-viewer/README.md2.2 KiB
- references/electron-shm-viewer/architecture-project.md6.3 KiB
- references/electron-shm-viewer/electron-client.md5.6 KiB
- references/electron-shm-viewer/protocol-interaction-lifecycle.md6.9 KiB
- references/electron-shm-viewer/python-shm-server.md4.9 KiB
- references/electron-shm-viewer/validation.md2.8 KiB
- references/gl-viewport-overlay/README.md7.9 KiB
- references/gl-viewport-overlay/architecture.md4.1 KiB
- references/gl-viewport-overlay/example-gizmo.md5.5 KiB
- references/gl-viewport-overlay/interaction-pattern.md4.0 KiB
- references/gl-viewport-overlay/projection-alignment.md4.3 KiB
- references/headless-shm-cli/README.md12.2 KiB
- references/huggingface-usd/README.md7.7 KiB
- references/local-viewer/README.md15.0 KiB
- references/native-picking-selection/README.md10.4 KiB
- references/object-selection/README.md9.4 KiB
- references/ovrtx-rendering/README.md18.2 KiB
- references/ovui-library/README.md3.3 KiB
- references/ovui-library/api-reference.md4.3 KiB
- references/ovui-library/extending.md4.8 KiB
- references/ovui-local-viewer-recipe/README.md3.7 KiB
- references/ovui-local-viewer-recipe/interaction-features.md16.1 KiB
- references/ovui-local-viewer-recipe/project-structure.md6.2 KiB
- references/ovui-local-viewer-recipe/setup-shell-renderer.md14.3 KiB
- references/ovui-local-viewer-recipe/validation-build-order.md5.5 KiB
- references/ovwidgets-editor-shell/README.md8.6 KiB
- references/prim-info-display/README.md9.3 KiB
- references/prim-pick-effects/README.md5.9 KiB
- references/prim-transform-safety/README.md8.2 KiB
- references/render-settings/README.md9.9 KiB
- references/routing.md16.5 KiB
- references/seg-outline-highlight/README.md1.4 KiB
- references/selection-animation/README.md9.7 KiB
- references/selection-feedback/README.md6.6 KiB
- references/stage-attribute-reads/README.md5.5 KiB
- references/stage-hierarchy/README.md17.1 KiB
- references/stage-hierarchy/fallback-worker-protocol.md3.5 KiB
- references/stage-loading/README.md19.3 KiB
- references/stage-management/README.md10.1 KiB
- references/stage-queries/README.md5.3 KiB
- references/streaming-client/README.md17.4 KiB
- references/streaming-lifecycle/README.md8.8 KiB
- references/streaming-messages/README.md16.5 KiB
- references/streaming-messages/server-handler-map.md3.1 KiB
- references/streaming-server/README.md18.6 KiB
- references/streaming-server/frame-loop-and-continuity.md3.3 KiB
- references/streaming-viewer-recipe/README.md4.8 KiB
- references/streaming-viewer-recipe/client-protocol.md11.0 KiB
- references/streaming-viewer-recipe/interaction-features.md13.0 KiB
- references/streaming-viewer-recipe/project-structure.md7.0 KiB
- references/streaming-viewer-recipe/server-runtime.md16.8 KiB
- references/streaming-viewer-recipe/validation-build-order.md5.8 KiB
- references/streaming-vs-local/README.md8.1 KiB
- references/tauri-local-viewer/README.md18.2 KiB
- references/tauri-shm-transform-gizmo/README.md11.8 KiB
- references/transform-manipulator/README.md15.0 KiB
- references/troubleshooting/README.md7.2 KiB
- references/troubleshooting/scenario-playbooks.md18.8 KiB
- references/usd-sample-data/README.md11.7 KiB
- references/usd-viewer-app/README.md7.5 KiB
- references/validation.md3.0 KiB
- references/viewer-backend-interface/README.md17.7 KiB
- references/viewer-control-patterns/README.md4.8 KiB
- references/viewer-data-view-patterns/README.md4.8 KiB
- references/viewer-feedback-status/README.md4.9 KiB
- references/viewer-input-routing/README.md7.8 KiB
- references/viewer-layout-patterns/README.md4.7 KiB
- references/viewer-ux-workflow/README.md4.6 KiB
- references/viewport-overlays/README.md4.8 KiB
- references/viewport-resize/README.md8.5 KiB
- references/webgl-shm-transport/README.md14.5 KiB
- references/windows-native-setup/README.md5.7 KiB
- skill-card.md2.9 KiB
- skill.oms.sig28.6 KiB
references/viewer-backend-interface/README.md
Viewer Backend Interface
Triggers
Use this skill for ViewerBackend, local shared UI, backend-agnostic React
viewer UI, reusable viewer components, viewer interface adapters, or
cross-transport UI.
Use this when a frontend should share the same panels, inspectors, selection controls, asset list, and viewer widgets across WebRTC streaming, Electron SHM, Tauri SHM/native IPC, or a future transport.
Purpose
The local viewer UI module is generated React component code plus a TypeScript
interface contract. The contract lets UI components depend on one
ViewerBackend object instead of transport-specific APIs such as AppStreamer
messages, Electron preload calls, Tauri commands, or future IPC layers.
The shared UI does not render USD or 3D content. All USD rendering still uses ovrtx in the appropriate server or native process. The UI displays a video, canvas, or pixel surface owned by the transport and overlays reusable React controls that call ViewerBackend.
Read These Skills
| Need | Read |
|---|---|
| Browser/WebRTC transport | streaming-client, streaming-messages, streaming-lifecycle |
| Electron + SHM local transport | electron-shm-viewer, webgl-shm-transport |
| Tauri/native desktop transport | tauri-local-viewer |
| Camera input behavior | viewer-input-routing, camera-controls |
| Picking and selection | viewer-input-routing, object-selection, selection-feedback |
| Hierarchy and property data | stage-hierarchy, stage-attribute-reads |
| AOV and render settings messages | aov-switching, render-settings |
Local Module Setup
Generate these files inside the frontend app unless the user requests a different local structure:
frontend/src/viewer-ui/
ViewerBackend.ts
types.ts
StageTree.tsx
PropertyPanel.tsx
Inspector.tsx
PrimIconSprite.tsx
USDAsset.tsx
index.tsUse index.ts as the local barrel export. Do not add an external package
dependency for shared viewer UI.
Import components and types from the local module:
import {
Inspector,
PrimIconSprite,
PropertyPanel,
StageTree,
USDAsset,
type FrameData,
type PrimNode,
type PrimProperty,
type USDAssetItem,
type ViewerBackend,
} from './viewer-ui';The local index.ts exports Inspector, InspectorProps, PrimIconSprite,
PropertyPanel, StageTree, StageTreeProps, USDAsset, and all shared
viewer types.
ViewerBackend Interface
Implement this interface at the transport boundary. Components should call this interface, not raw WebRTC, Electron, Tauri, or server-message APIs.
export interface ViewerBackend {
connect(): Promise<void>;
disconnect(): void;
loadStage(path: string): Promise<void>;
resize?(width: number, height: number): Promise<void>;
sendRenderScale?(scale: number): Promise<void>;
setCamera(params: CameraParams): Promise<void>;
cameraMouseButton(input: PointerInput): Promise<boolean>;
cameraMouseMove(x: number, y: number): Promise<void>;
cameraWheel(delta: number): Promise<void>;
onFrame(callback: (frame: FrameData) => void): () => void;
onStats?(callback: (stats: FrameBudgetStats) => void): () => void;
onLoadProgress?(callback: (progress: LoadProgressEvent) => void): () => void;
onAOVStateChanged?(callback: (active: string, available: string[]) => void): () => void;
changeAOV?(aov: string): Promise<void>;
onSelectionChanged(callback: (paths: string[]) => void): () => void;
pick(x: number, y: number): Promise<string | null>;
getStageTree(rootPath?: string): Promise<PrimNode[]>;
selectPrims(paths: string[]): Promise<void>;
getProperties(path: string): Promise<PrimProperty[]>;
}Method contract:
| Method | Contract |
|---|---|
connect() | Establish the transport, register event/frame listeners, and resolve when commands can be sent. Guard duplicate calls from React remounts. |
disconnect() | Remove listeners, stop frame pumps, close transport handles, and reject or clear pending resolvers. Make cleanup safe to call more than once. |
loadStage(path) | Ask the backend to load a USD stage by URL/path. Clear stale tree, property, selection, progress, AOV, and pending-pick state. Resolve after the backend accepts or completes the load according to that transport's normal semantics. |
resize?(width, height) | Resize a backend-owned dynamic render target when supported. Fixed-resolution video and fixed render-product transports should implement a no-op or omit this method. |
sendRenderScale?(scale) | Request a render-scale change when the backend supports adaptive scaling. Clamp to supported values and publish the effective scale through frame or stats updates when possible. |
setCamera(params) | Set camera state directly for backends that own explicit camera parameters. Native-input transports may resolve without action if camera motion is driven by pointer messages. |
cameraMouseButton(input) | Send a button press/release to the camera controller. Return true when the backend classifies the gesture as a click so callers can perform selection picking. |
cameraMouseMove(x, y) | Send pointer motion in viewport-local CSS pixels. Continuous input should be fire-and-forget internally but the Promise should settle quickly. |
cameraWheel(delta) | Send wheel zoom input. Preserve the backend's existing wheel sign convention from viewer-input-routing and camera-controls. |
onFrame(callback) | Subscribe to frame arrivals or frame timing. Return an unsubscribe function. FrameData.pixels is optional because WebRTC video transports may expose only timing. |
onStats?(callback) | Subscribe to transport/render performance stats such as FPS, queue depth, dropped frames, latency, and WebRTC inbound RTP stats when available. |
onLoadProgress?(callback) | Subscribe to stage-load progress phases. Use this for progress bars and disabled UI states during load. |
onAOVStateChanged?(callback) | Subscribe to active AOV and available AOV list changes. Call immediately with cached state when available so controls populate after reconnect. |
changeAOV?(aov) | Request an AOV/render-var switch. Reject unsupported AOV names or no-op only when the backend intentionally has no AOV support. |
onSelectionChanged(callback) | Subscribe to canonical selected prim paths. Return an unsubscribe function and fan out both local UI selections and server/native selection events. |
pick(x, y) | Return the prim path under a viewport-element-local CSS pixel coordinate, or null. Never require callers to pass window coordinates. |
getStageTree(rootPath?) | Return normalized PrimNode[] for the root or requested prim path. It may use cached tree data, lazy queries, or a full hierarchy snapshot. |
selectPrims(paths) | Select the canonical prim paths in the backend and publish the same path list through onSelectionChanged. Also update native selection/highlight state when supported. |
getProperties(path) | Return displayable properties for a prim as PrimProperty[]. Normalize backend dictionaries, USD values, or typed command responses at this boundary. |
Critical Coordinate Contract
pick(x, y), cameraMouseButton(input), and cameraMouseMove(x, y) use viewport-element-local CSS pixel coordinates, not window coordinates.
Callers starting from DOM events must subtract the viewport element rect:
function toViewportPoint(event: React.PointerEvent, viewport: HTMLElement) {
const rect = viewport.getBoundingClientRect();
return {
x: event.clientX - rect.left,
y: event.clientY - rect.top,
};
}
const { x, y } = toViewportPoint(event, viewportElement);
const clicked = await backend.cameraMouseButton({ x, y, button: event.button, pressed: false });
if (clicked) {
const pickedPath = await backend.pick(x, y);
await backend.selectPrims(pickedPath ? [pickedPath] : []);
}If a backend needs render-product pixels, map from viewport-local CSS pixels to the contained image/canvas area inside the backend or viewport adapter. Do not pass raw PointerEvent.clientX/clientY into ViewerBackend.
Frame Delivery Contract
FrameData.pixels is optional. WebRTC/AppStreamer transports usually render into a <video> element and can only report frame timing, decoded dimensions, or stats through onFrame. SHM, Tauri, or future pixel transports may provide RGBA pixels.
Shared UI components must not require pixel data. Viewport display components may choose the transport-specific surface:
| Transport | Display surface | onFrame payload |
|---|---|---|
| WebRTC streaming | <video> from AppStreamer | Timing/dimensions; pixels usually absent |
| Electron SHM | Canvas/WebGL texture upload from SHM | RGBA pixels or transport-local buffer metadata adapted to FrameData |
| Tauri SHM/native IPC | Canvas/ImageData or texture upload | RGBA pixels when copied/decoded for the React layer |
| Future transport | Transport-owned surface | At least width, height, encoding, and timing when available |
Type Catalog
export interface CameraParams {
azimuth?: number;
elevation?: number;
distance?: number;
target?: [number, number, number];
}
export interface FrameData {
width: number;
height: number;
encoding: 'rgba';
pixels?: Uint8ClampedArray;
frameIndex?: number;
renderScale?: number;
}
export interface FrameBudgetStats {
timestampMs: number;
fps: number;
sourceFps: number;
frameIntervalMs: number;
deliveryMs: number;
queueDepth: number;
skippedFrames: number;
droppedFrames: number;
renderScale: number;
backpressure: boolean;
frame_time_ms?: number;
stream_time_ms?: number;
frames_rendered?: number;
gpu_encoder_active?: boolean;
roundTripTimeMs?: number;
packetsLost?: number;
jitter?: number;
framesDecoded?: number;
freezeCount?: number;
framesPerSecond?: number;
}
export type LoadProgressPhase =
| 'resolving_asset'
| 'loading_stage'
| 'compiling_shaders'
| 'ready';
export interface LoadProgressEvent {
phase: LoadProgressPhase;
stage_url?: string;
}
export type PrimType = 'xform' | 'scope' | 'geom' | 'light' | 'camera';
export interface PrimNode {
name?: string;
path: string;
children?: PrimNode[] | null;
hasChildren?: boolean;
type?: PrimType;
}
export type USDPrim = PrimNode;
export interface PrimProperty {
name: string;
type: string;
value: string;
}
export interface USDAssetItem {
name: string;
url: string;
}
export interface PointerInput {
x: number;
y: number;
button: number;
pressed: boolean;
}Normalize transport payloads to these types before data reaches shared components. In particular, convert server fields such as has_children or boolean children into hasChildren, and convert property dictionaries into PrimProperty[].
Shared Components
| Component | Use |
|---|---|
StageTree | Displays the USD hierarchy, selected-row state, expandable prims, and prim type icons. Wire row selection to backend.selectPrims() and expansion/lazy loading to backend.getStageTree(path) when the component props require callbacks. |
PropertyPanel | Displays PrimProperty[] for one prim. Feed it data from backend.getProperties(selectedPath). |
Inspector | Higher-level selected-prim inspector. Use it when the app wants shared selection/property behavior instead of composing PropertyPanel directly. |
PrimIconSprite | Defines the icon sprite used by hierarchy rows and inspector UI. Render once near the application root. |
USDAsset | Displays a loadable USD asset item. Feed it USDAssetItem values and call backend.loadStage(asset.url) on activation. |
When generating components, keep prop names small and stable. Do not bake
transport APIs into components; adapt the transport to ViewerBackend instead.
Minimum prop contracts:
export interface StageTreeProps {
backend: ViewerBackend;
selectedPaths: string[];
rootPath?: string;
}
export interface PropertyPanelProps {
properties: PrimProperty[];
}
export interface InspectorProps {
backend: ViewerBackend;
selectedPaths: string[];
}
export interface USDAssetProps {
asset: USDAssetItem;
backend?: ViewerBackend;
onLoad?: (asset: USDAssetItem) => void;
}Minimum behavior:
StageTreeloads root nodes withbackend.getStageTree(rootPath)on mount, loads children when an expandable row opens, highlightsselectedPaths, and callsbackend.selectPrims([path])on row activation.PropertyPanelrenders a compact name/type/value table and treats values as display strings unless another skill adds editing behavior.Inspectorsubscribes to selected paths through props, fetches properties for the primary selected path withbackend.getProperties(path), cancels stale responses on selection change, and renders an empty state for no selection.PrimIconSpritemay be a no-op component when the app uses text labels or CSS icons instead of an SVG sprite.USDAssetrenders one loadable asset row or button and callsonLoad(asset)when supplied; otherwise it callsbackend.loadStage(asset.url)when a backend prop is supplied.
Implementing a New Backend
Use the hook-per-transport pattern:
useWebRTCBackend.ts -> AppStreamer + data-channel messages + video timing
useShmBackend.ts -> Electron preload API + SHM frame pump + JSON control messages
useTauriBackend.ts -> Tauri commands/events/channels + native frame deliveryImplementation rules:
Return one stable
ViewerBackendobject from the hook withuseMemo.Keep transport objects, request resolvers, caches, and subscriber sets in refs.
Convert transport events into shared callbacks: frame, stats, load progress, AOV state, and selection.
Add timeouts to request/response promises for
loadStage,pick,getStageTree, andgetProperties.Resolve responses by request id when the protocol supports it; otherwise bucket by URL/path/message type.
Normalize all hierarchy nodes, property values, AOV names, and selected paths at the backend boundary.
Make
disconnect()clear subscribers only when the app is truly tearing down; ordinary event-handler cleanup should only unsubscribe that handler.Preserve the single render-thread/server ownership rules from the transport skill. The React backend is an adapter, not a renderer.
Resolver pattern:
type Resolver<T> = {
resolve: (value: T) => void;
reject: (error: Error) => void;
timer: number;
};
const selectionHandlers = useRef(new Set<(paths: string[]) => void>());
const frameHandlers = useRef(new Set<(frame: FrameData) => void>());
const treeResolvers = useRef(new Map<string, Resolver<PrimNode[]>[]>());
const propertyResolvers = useRef(new Map<string, Resolver<PrimProperty[]>[]>());
const treeCache = useRef(new Map<string, PrimNode[]>());
function emitSelection(paths: string[]) {
for (const handler of selectionHandlers.current) handler(paths);
}
function resolveBucket<T>(buckets: Map<string, Resolver<T>[]>, key: string, value: T) {
const bucket = buckets.get(key) || [];
buckets.delete(key);
for (const resolver of bucket) {
window.clearTimeout(resolver.timer);
resolver.resolve(value);
}
}Transport notes:
WebRTC/AppStreamer:
connect()owns one AppStreamer connection. RouteonCustomEventmessages into resolver buckets and subscriber sets.onFrameusually publishes timing from video playback quality or stats; leavepixelsundefined.Electron SHM:
connect()attaches preload/native SHM APIs and starts one frame pump. Convert SHM frames to the viewport's upload path and publish sharedFrameDataonly for data the shared UI needs.Tauri SHM/native IPC:
connect()registers Tauri event listeners and frame channels once. Decode binary RGBA frames when the React layer owns a canvas; otherwise publish timing/dimensions and let the viewport adapter own pixels.
Integration Recipe
Generate
frontend/src/viewer-ui/with theViewerBackendtypes and local React components required by the app.Implement a backend hook for the chosen transport:
useWebRTCBackend,useShmBackend,useTauriBackend, or a new hook.Mount the transport viewport surface:
<video>,<canvas>, or a native pixel-presenting component.Convert pointer events to viewport-local CSS pixels before calling camera or pick methods.
Subscribe to backend selection and frame/stat events in React effects.
Render shared UI components against the backend rather than raw transport APIs.
Keep transport-specific message names and native handles inside the backend hook.
Skeleton:
function ViewerApp() {
const backend = useWebRTCBackend(config);
const [selectedPaths, setSelectedPaths] = useState<string[]>([]);
useEffect(() => {
void backend.connect();
return () => backend.disconnect();
}, [backend]);
useEffect(() => backend.onSelectionChanged(setSelectedPaths), [backend]);
async function handlePointerUp(event: React.PointerEvent) {
const viewport = event.currentTarget as HTMLElement;
const rect = viewport.getBoundingClientRect();
const x = event.clientX - rect.left;
const y = event.clientY - rect.top;
const clicked = await backend.cameraMouseButton({ x, y, button: event.button, pressed: false });
if (!clicked) return;
const picked = await backend.pick(x, y);
await backend.selectPrims(picked ? [picked] : []);
}
return (
<>
<PrimIconSprite />
<main className="viewer-shell">
<section className="viewport" onPointerUp={handlePointerUp}>
<video id="remote-video" />
</section>
<aside className="sidebar">
<StageTree backend={backend} selectedPaths={selectedPaths} />
<Inspector backend={backend} selectedPaths={selectedPaths} />
</aside>
</main>
</>
);
}Keep the dependency direction the same: app UI calls local viewer UI
components, those components call ViewerBackend, and only backend hooks know
the transport.
Requirements and compatibility
Written for: OpenAI Codex
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
The author says: Orchestrator skill. Downstream focused references may require NVIDIA GPUs, ovrtx, ovstream, ovui, OpenUSD, Python, Node/React, Tauri, Electron, C++, or cloud GPU deployment access depending on the selected viewer path.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
- Instructions that use the network
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA Omniverse
- Publisher
- OpenAI (Official)
- Distributed by
- OpenAI
- Source
- openai/plugins · plugins/nvidia/skills/omniverse-realtime-viewer · commit d420d13
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (written by AutoGineer)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA Omniverse or OpenAI.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download omniverse-realtime-viewer.zip