Omniverse Realtime Viewer

Tauri + Rust OVRTX Omniverse Realtime Viewer

Triggers

Use this skill for Tauri 2, Rust desktop Omniverse Realtime Viewer, OVRTX through C FFI, Tauri Channel IPC, ViewerBackend interface, native desktop Omniverse Realtime Viewer, web-technology UI, no Python, or no ovui runtime.

Use this pattern when the app should be a native desktop binary with a React frontend, no Python runtime, and direct Rust FFI to OVRTX. The Tauri app shares its ViewerBackend interface with the WebRTC streaming frontend so UI components can be reused across both paths.

For ovrtx C API behavior, FFI behavior, renderer lifecycle guidance, or release-specific behavior not covered here, read references/dependencies for acquisition guidance and supplemental dependency documentation.

This is not a GPU zero-copy renderer. The active display path is:

OVRTX step -> CPU map LdrColor -> raw RGBA payload -> Tauri Channel -> canvas putImageData

Do not add wgpu, CUDA/Vulkan external-memory import, NVENC, or WebCodecs as part of the active local display path unless the app is explicitly redesigned around a new presentation contract.

When to Use This vs Other Paths

Architecture Overview

React WebView (Vite)
  |-- ViewerBackend interface (useTauriBackend.ts)
  |-- Tauri invoke / Channel IPC
  |
Rust backend
  |-- commands.rs       -> Tauri commands, events, shared-state reads
  |-- render_loop.rs    -> single render thread owns mutable OVRTX state
  |     |-- drain commands
  |     |-- write camera (ovrtx_write_attribute)
  |     |-- step renderer (ovrtx_step)
  |     |-- CPU map "LdrColor"
  |     `-- push binary frame via Tauri Channel
  |-- ovrtx_bridge.rs   -> OVRTX C API FFI, loading, mapping, native picking
  |-- camera.rs         -> OrbitCamera math and input handling
  |-- session_layer.rs  -> inline root/session USDA generation
  |-- ovrtx_env.rs      -> Windows DLL/path/runtime discovery
  `-- picking.rs        -> native pick requests and UI selection state

Generated project paths may be tauri-app/src-tauri/src/ and tauri-app/src/, or a flattened layout such as tauri-src/ and ts-src/. Treat the module names above as the stable structure.

Critical Invariants

1. Single Render Thread

One Rust thread owns all mutable render state:

• OvrtxBridge renderer and stage handles

• OrbitCamera

• InstanceTable

• selection state

• frame counters and render error counters

Tauri commands enqueue RenderCommand messages through mpsc when they need to touch OVRTX or camera state. Never call OVRTX from the IPC thread. Continuous input is fire-and-forget; discrete actions use reply channels or events.

2. Camera Write Contract

ovrtx_write_attribute:
  attribute:    "omni:xform"
  semantic:     XFORM_MAT4X4
  binding dtype: float64, lanes=16
  input tensor:  (1,4,4) float64 lanes=1, ndim=3, strides=NULL, CPU
  prim_mode:    CREATE_NEW
  access:       SYNC
  path:         /Session/Cameras/Main

Write the camera every frame before step(). Validate finite values and skip the write on NaN. Do not use bind+map or a different lane layout; OVRTX failures here can look like a camera that silently never moves.

3. Session Layer Paths

These paths must match the Python local Omniverse Realtime Viewer and streaming server camera setup:

/Session/Cameras/Main         camera
/Session/Render/Viewport      LdrColor display render product
/Session/Render/PickBuffer    InstanceSeg pick render product
/Session/Render/Vars/...      render vars
/Session/OVRenderSettings     render settings

Non-matching paths can cause OVRTX to use a default camera or fail to expose the expected render outputs.

4. Fixed Render Resolution

Resolution is baked into the generated session USDA. Changing it requires a session reload and wipes the camera xform. Keep the backend render size fixed and use UI-side letterboxing plus pointer-coordinate mapping.

resize(width, height) is advisory in the current backend; it does not resize OVRTX render products.

5. Channel-Based Frame Delivery

Primary frame transport is a Tauri Channel<InvokeResponseBody> registered by subscribe_frame_events(on_frame). The payload is:

[u32 width LE][u32 height LE][u64 sequence LE][RGBA8 pixels]

The frontend decodes ArrayBuffer payloads, drops duplicate or older sequence numbers, creates ImageData, and paints with canvas.putImageData.

Do not use a named binary ovrtx-frame event for frames. Named events are used for lifecycle, errors, picking, and selection, not for the main frame stream.

take_frame_bytes() is a pull fallback for reconnect/debug paths. Channel push is the primary display path. New Channel subscribers should receive the latest cached good frame immediately when one exists.

6. Stage Load Sequence

The stage load flow should:

1. Resolve the input path.

2. Read/cache a lightweight stage tree before the render-thread load.

3. Reset any existing OVRTX stage.

4. Add the user USD at the root.

5. Add the generated session USDA under /Session.

6. Clear selection and render error state.

7. Reset the camera to the default view.

8. Rebuild InstanceTable from the cached tree.

9. Mark the stage loaded and emit stage-loaded.

Effect-fader writes are currently inactive. The load path calls write_effect_faders with an empty shader-path list, so the helper returns without writing anything. If real shader paths are supplied in a future feature, the helper must keep using CREATE_NEW, not EXISTING_ONLY.

7. Render Loop Timing

The render thread drains all pending commands before rendering a frame. When no stage is loaded it waits for commands instead of busy-spinning.

The active loop targets about 16 ms per frame. Render delta time is clamped to a small lower bound and to about 0.1 seconds on the high end so long stalls do not explode camera or animation updates.

Do not call renderer.step() concurrently with stage reset/load.

8. Error Model

• FrameNotReady is not a fatal render error.

• Ten consecutive non-FrameNotReady render failures disable rendering.

• Disabling rendering emits ovrtx-render-stopped.

• The disabled state clears only on the next LoadStage.

• Keep serving the latest good frame while rendering is disabled.

• Display-frame map/result cleanup uses RAII guards.

• Pick-buffer mapping uses explicit cleanup; do not assume all mapped-output paths are RAII protected.

IPC Commands and Events

Expected Tauri command inventory:

Expected emitted event names:

Do not add a frame event named ovrtx-frame; use the Channel command for frame payloads.

ViewerBackend Interface

The shared contract between Tauri and WebRTC frontends should look like this for the current local Omniverse Realtime Viewer:

interface ViewerBackend {
  connect(): Promise<void>;
  disconnect(): void;
  loadStage(path: string): Promise<{ path: string; tree: PrimNode[] } | void>;
  resize(width: number, height: number): Promise<void>;
  setCamera(camera: CameraState): Promise<void>;
  cameraMouseButton(input: PointerInput): Promise<boolean>;
  cameraMouseMove(x: number, y: number): Promise<void>;
  cameraWheel(delta: number): Promise<void>;
  onFrame(callback: (frame: FrameData) => void): () => void;
  onSelectionChanged(callback: (paths: string[]) => void): () => void;
  pick(x: number, y: number): Promise<string | null>;
  getStageTree(rootPath?: string): Promise<PrimNode[]>;
  selectPrims(paths: string[]): Promise<void>;
  getProperties(path: string): Promise<PrimProperty[]>;
}

Use viewer-input-routing for the shared click-vs-drag and button semantics behind cameraMouseButton, cameraMouseMove, cameraWheel, and pick.

getProperties is currently a frontend stub that returns only the path property. There is no Rust/USD property-query command yet. get_camera exists on the Rust side but may not be exposed by useTauriBackend.

To add a feature, extend ViewerBackend first. Use a Tauri command plus RenderCommand only when the feature must touch OVRTX, camera, or render-thread state. Shared-state reads such as cached tree access and frame-channel subscription do not need to enqueue render-thread work.

Picking and Selection

Display and picking use the same render-product pixel coordinate space. A frontend click is mapped through the letterboxed image rect and sent to the Rust render thread, which enqueues an ovrtx native pick query for the active RenderProduct.

Picking is asynchronous from the frontend point of view:

1. The frontend invokes pick(x, y).

2. Rust returns a request_id immediately.

3. The render thread enqueues the native pick query and steps the active RenderProduct.

4. The render thread decodes ovrtx_pick_hit, resolves the picked path id, and emits ovrtx-pick with the request_id and optional path.

5. The frontend resolves the pending pick promise, currently with a short timeout.

Do not document or implement pick as a synchronous command returning the final path directly.

select_prims(paths) stores UI selection state and emits stage-selection-changed. When renderer-visible feedback is requested, OvrtxBridge::set_selection should write native selection outline groups: group 0 to clear previous paths and a non-zero styled group for selected paths.

Stage Tree and Properties

The current stage-tree reader is intentionally lightweight:

• It reads simple .usda text.

• It recognizes basic def and over prim declarations.

• It returns children under /World if present; otherwise it returns roots or a fallback /World node.

• It does not build complete nested children arrays for all USD constructs.

• get_stage_tree(rootPath?) ignores rootPath and returns the cached tree.

Do not present this as a full USD stage browser. Use stage-hierarchy when the app needs robust USD traversal, properties, variants, bounds, or composition queries.

Stage path resolution accepts an absolute existing path, then tries the current directory, parent, and grandparent before returning the original input path.

Camera and Input

The expected controls are:

• left drag: orbit

• middle drag: pan

• right drag: dolly

• wheel: zoom

• button release with movement under the click threshold: pick

OrbitCamera::current_xform recomputes the camera transform and the render loop writes it every frame. On stage load, reset the camera to the default view before rendering.

Frontend pointer handling should use pointer capture plus window-level pointermove, pointerup, and pointercancel listeners so drags survive WebView edge cases and leaving the canvas bounds.

Environment Setup and Loader Notes

Runtime setup should prove the app loaded the packaged ovrtx runtime that it will use at run time, not only that Rust compiled. Read the local ovrtx C/CMake headers or examples before assuming FFI names, library names, or config keys.

On Linux, packaged runtime layouts commonly expose bin/libovrtx-dynamic.so; do not assume libovrtx.so is the load target. On Windows, use the equivalent packaged DLL path from the same runtime root. In both cases, resolve absolute paths before launch and log them.

Expected runtime proof:

• OVRTX_LIB_PATH=/absolute/path/to/ovrtx/bin when the loader needs an explicit library directory.

• OVRTX_BINARY_PACKAGE_ROOT_PATH=/absolute/path/to/ovrtx when the runtime needs package-root discovery.

• Runtime layout contains the expected bin, plugins, usd_plugins, and rendering-data entries for the selected ovrtx package.

• Backend logs include the exact dynamic library path, ovrtx version, requested stage path, and stage-open result.

• /proc/<pid>/maps on Linux, or platform-equivalent loader inspection on Windows, confirms the running desktop process mapped the expected ovrtx library from the expected package root.

On Windows, runtime setup should:

• set OVRTX_SKIP_USD_CHECK=1 before OVRTX work

• honor OVRTX_BINARY_PACKAGE_ROOT when provided

• probe upward from the executable for the OVRTX SDK/package layout

• derive OVRTX_SDK_PATH and OVRTX_BIN_PATH

• prepend bin/plugins and bin to PATH

build.rs should add OVRTX_SDK_PATH as a native link search path. Loader setup may need to pass the package bin directory as the loader root when plugins live under bin/plugins.

OvrtxBridge::new should pass OVRTX config entries for binary package root, sync mode, and active CUDA GPUs according to the local SDK expectations.

Prefer one reproducible launch command or script that configures the runtime, stage, and log file together:

OVRTX_LIB_PATH=/path/to/ovrtx/bin \
OVRTX_BINARY_PACKAGE_ROOT_PATH=/path/to/ovrtx \
OVRTX_OPEN_STAGE=/path/to/stage.usd \
npm run dev:desktop 2>&1 | tee /tmp/ovrtx-tauri-viewer-dev.log

For Hugging Face-hosted USD datasets, read huggingface-usd, download or clone the dataset to the local filesystem, preserve relative dependency layout, and open the resolved local path. Validate that the root USD is real data before launching; for binary crate USD, head -c 8 <file>.usd should show PXR-USDC, not a Git LFS pointer.

FFI and Memory Safety

OVRTX FFI is layout-sensitive. Keep Rust structs synchronized with ovrtx_types.h; in particular, OvrtxConfigEntryT must not grow an extra field between key_type and key.

Keep strings, config arrays, tensors, and attribute buffers alive for the full duration of synchronous OVRTX calls. Do not pass pointers to temporaries that can be dropped before the C function returns.

Mapped output handling must respect DLPack layout:

• accept 2D, 3D, 4D, and known fallback dimensional layouts

• compute width, height, row stride, byte length, and data pointer from metadata

• pack padded RGBA rows into tightly packed bytes before sending to the WebView

• unmap and destroy result handles on every error path

Display mapping has RAII cleanup. Pick mapping currently uses explicit cleanup; keep that distinction in mind when editing pick_instance.

Behavioral Reference

When behavior differs between local Tauri code and the Python viewer, prefer the local app's established contracts for:

• camera write shape and session USDA paths

• fixed render resolution with UI letterboxing

• pointer coordinate mapping into the rendered image content rect

• render-thread ownership of OVRTX state

Common Mistakes

Definition Of Done

A Tauri ovrtx viewer is not done at compile time. Before handing it off, capture evidence for the real desktop runtime path:

• npm run check or equivalent TypeScript/Rust validation passes.

• npm run build:desktop or equivalent Tauri package build passes.

• The real Tauri desktop app launches through the reproducible launch script.

• The requested USD stage path is opened at startup, and backend logs show runtime loaded, ovrtx version, and stage opened or equivalent fields.

• The process stays alive after stage load and produces at least one displayed frame through the Tauri Channel path.

• Loader inspection confirms the process mapped the expected ovrtx dynamic library from the configured package root.

• nvidia-smi or the platform equivalent shows the app using the GPU when GPU rendering is expected and available.

• UI-visible runtime status agrees with backend logs; do not show "ready" when the runtime fell back, failed to load, or never opened the stage.