Omniverse Realtime Viewer

ovui Local Setup, Shell, And Renderer

2. Install Dependencies And Configure The Environment

Read references/dependencies FIRST. Its references/nvidia-runtime.md file is the source of truth for NVIDIA runtime locations. Do not guess or repeat ovui package URLs, wheel names, artifact locations, or fallback install commands in this recipe.

Do this:

• Install ovrtx using the package guidance in references/dependencies.

• Install ovui using the current PyPI package guidance in references/dependencies.

• Install any selected local UI companion packages from the same ovui package set as the base UI package.

• Install usd-core==24.11 when the local app needs pxr queries. Pin to 24.11 — newer versions cause TfType schema conflicts with ovrtx.

• Install numpy for matrices, camera math, and CPU frame handling (pip install numpy).

• Install warp-lang only when the local app needs CUDA-side display or image-processing utilities (pip install warp-lang).

• Do not install ovstream or frontend streaming packages for a local-only viewer.

Set these environment contracts before starting the app:

• OVRTX_SKIP_USD_CHECK=1 must be set before ovrtx is imported or the renderer is constructed.

• OVRTX_BIN_PATH must point at the ovrtx bin directory when materials or renderer plugins fail to resolve.

• The ovrtx plugin library path must be first in the dynamic library path if another USD build is present.

• PYTHONPATH must include any ovui import paths required by the selected package.

• A real display must be available. For headless validation, use a configured X display such as Xvfb.

Decision points:

• If the local app imports pxr in the main process, follow the local import discipline documented in references/local-viewer.

• If USD registry or DLL conflicts appear, isolate pxr queries into a subprocess and keep the main app focused on ovui plus ovrtx.

• If the user only needs rendering and camera navigation, defer usd-core until hierarchy or bounds queries are required.

• If running on Windows, read references/windows-native-setup before changing import order or subprocess strategy.

• If local UI imports report missing package metadata or missing VIEWPORT_CAMERA_POSE_SOURCE, read references/dependencies and use one compatible local UI package set.

Common failure modes:

• usd-core detected, duplicate USD debug symbols, _tf import failures, or MDL resolver crashes usually mean import order or library path is wrong.

• Magenta materials usually mean OVRTX_BIN_PATH or plugin library path is missing.

• ovui import failure usually means the selected package was not installed or its required import path is not on PYTHONPATH.

• A blank or crashing local window in CI usually means no real display is available.

Read for depth: see references/dependencies, references/local-viewer, references/ovrtx-rendering, and references/windows-native-setup for the full environment contracts.

3. Build The ovui Window Shell

Do this in local_app/app.py and local_app/viewport.py:

• Initialize ovui once with the requested title, width, height, and target FPS.

• Create one main window that fills the app window.

• Use a large central viewport, a compact header or toolbar, an optional scene tree or info sidebar, and an optional render settings panel.

• Put the rendered image inside a viewport frame that can resize with the window.

• Add a transparent hit surface above the image for mouse input.

• Route toolbar actions to runtime commands such as load scene, reload, fit camera, clear selection, toggle tree, and toggle render settings.

• For lightweight viewers, prefer a path field plus one LOAD button for scene loading. Add native file dialogs only after they are validated in the same display/session environment as the app.

• Wrap ovui callbacks so exceptions are logged and do not tear down the app loop.

Critical contracts:

• Use fill_app_window=True for the main ovui window so the UI frame tracks GLFW window resizes.

• The viewport widget must report its current size or provide a reliable fallback so letterbox math remains valid.

• The transparent mouse surface must be above the image and marked as receiving mouse events.

• Header, tree, info, and settings widgets must not intercept viewport gestures unless the pointer is actually over those controls.

• The app loop must call runtime tick/update work from one predictable place.

• If a selected-prim transform gizmo is visible, dragging it must write the selected prim's live omni:xform through the serialized renderer runtime; a visual-only handle is not complete.

Decision points:

• If the user wants a compact visual tool, keep a header plus viewport and place advanced panels in collapsible sidebars.

• If the user wants a dense inspection app, reserve width for a stage tree and prim info panel from the start.

• If the user wants a context menu, show it on right-button release only when the drag threshold was not exceeded.

• If the user wants a camera gizmo, build it as a local ovui scene overlay rather than using streaming overlay patterns.

• If the user wants selected-prim movement in a lightweight shell, either use a proven transform manipulator that emits transform deltas or add an app-owned fallback that treats presses near the selected pivot as transform drags.

Common failure modes:

• Without fill_app_window=True, the OS window resizes but the UI frame stays at the initial dimensions.

• If the hit surface is behind the image or not opaque to mouse events, orbit, pan, zoom, and picking never receive input.

• A separate OPEN button can be worse than no button if the native dialog is not implemented or cannot appear under the target display stack. Keep the path-field LOAD path reliable and make dialogs secondary.

• A selected-prim gizmo that appears but does not move the prim usually means handle input is being consumed by camera/orbit logic or the manipulator is not connected to renderer.write_attribute.

• Uncaught callback exceptions can stop the app loop or leave drag state stuck.

• DOM-style browser assumptions do not apply; input comes from ovui callbacks in the desktop process.

Read for depth: see references/local-viewer, references/viewer-input-routing, references/camera-controls, and references/viewport-overlays for ovui shell, input routing, context menu, and local overlay guidance.

4. Construct The ovrtx Renderer Runtime

Do this in local_app/renderer_runtime.py:

• Create the ovrtx renderer after environment variables are set and import order is settled.

• Use synchronous rendering first.

• Store the active render product path, render width, render height, current frame index, and whether a valid stage is loaded.

• Expose render-loop-only operations for loading a scene, resetting the stage, stepping a frame, mapping render vars, enqueueing and decoding native pick queries, writing native selection outline groups, and writing live attributes.

• Keep renderer mutations serialized with scene loading, scene reset, settings changes, and camera writes.

Critical contracts:

• The application calls renderer.step() explicitly. ovrtx does not run a hidden app loop for the viewer.

• Pass the exact viewer RenderProduct path to every step call.

• Extract LdrColor for local image display. It is RGBA8 from ovrtx.

• For local UI display, copy CPU-mapped pixel data inside the map context before returning it to the widget.

• Use write_attribute for live camera transforms and other live state. Write omni:xform, not authored xformOp:*, for interactive updates.

• Use the correct transform semantic and create-new prim mode for attributes that may not already exist in Fabric.

Decision points:

• If the app only needs basic local viewing, start with LdrColor only.

• Object selection uses native ovrtx pick queries. Do not add segmentation render vars just to make picking work.

• If the app needs high-FPS local display, profile CPU readback first before introducing a CUDA-to-UI path.

• If the renderer reports stale GPU hangs after a crash, inspect running Python GPU processes before changing code.

Common failure modes:

• Unable to find RenderProduct prim means scene setup did not create the path used by renderer.step().

• Black frame usually means camera relation, render product resolution, render var source, or camera transform is invalid.

• Live camera changes doing nothing usually means the app wrote xformOp:transform instead of omni:xform, or used existing-only prim mode.

• Crashes during scene switches usually mean renderer.step() overlapped a reset, load, or layer mutation.

Read for depth: see references/ovrtx-rendering for the full renderer construction, frame extraction, and live attribute contract.

5. Implement Scene Loading

Do this in local_app/scene_loader.py and call it only from the render loop or serialized runtime load path:

• Resolve the requested URL/path against the configured asset root, allowed schemes, and security policy.

• Create viewer-owned camera, RenderProduct, RenderVar, and RenderSettings data through one inline root/session USDA string when the user stage lacks viewer render config.

• Load the user stage without modifying it.

• Store the viewer camera path and render product path in runtime state.

• Reset selection, native selection outline groups, stage query caches, pending pick queries, and loading status for the new stage.

• Fit the camera to the stage bounds unless the user requested preserving the current camera or using an authored stage camera.

Critical contracts:

• Every loaded stage needs Camera -> RenderProduct -> RenderVar -> RenderSettings wiring that ovrtx can find.

• The viewer camera path must be the same path used by camera controls when writing omni:xform.

• Do not inject lights unless the user explicitly asks for viewer-controlled lighting.

• Include segmentation render vars only for explicit debug/AOV display modes, not for picking.

• Prefer renderer.open_usd_from_string() for inline roots that sublayer the user USD and author viewer render config.

• Do not call reference or layer-add APIs after a stage is already loaded unless the renderer has been reset to an empty stage and the operation is part of the serialized load path.

Decision points:

• Use a single inline root USDA string with subLayers = [@user_scene@] when the user file needs viewer camera/render-product/render-var data.

• If the user stage has an authored camera and the requested policy is stage-camera, copy its focal length, apertures, clipping range, and transform into the viewer camera.

• If the user requests persistent camera across scene switches, keep camera state but sanitize and refit only when the old state is invalid for the new bounds.

• If the user requests viewer lighting controls, add explicit viewer-owned light prims only with a verified live apply path or an explicit reload/profile workflow; otherwise leave lighting untouched and omit live lighting controls.

Common failure modes:

• Inline roots that omit or misquote the user sublayer path fail composition or break relative asset resolution.

• Camera path mismatch makes input appear connected but the view never moves.

• A stage-load operation that reports an error must not be treated as a successful load just because the enqueue call returned.

Read for depth: see references/stage-loading, references/render-settings, references/selection-feedback, and references/stage-hierarchy for the full scene setup contract.

6. Display The Rendered Image

Do this in local_app/viewport.py:

• Create an ImageBridge with the render width and height.

• Display the bridge provider through an ovui image widget using preserve-aspect fit.

• Before connecting ovrtx output, push a synthetic RGBA gradient through the same provider path and capture a window screenshot proving the native widget paints nonblank, non-solid pixels.

• On each rendered frame, update the bridge with copied RGBA pixels from LdrColor.

• Compute the visible image rectangle inside the viewport widget after every resize.

• Store the current widget size, visible image offset, visible image size, render width, and render height for input mapping.

• Show explicit idle, loading, and error states when no frame should be displayed.

Critical contracts:

• The ovrtx LdrColor buffer is RGBA8 and is suitable for local image display through ImageBridge.

• Copy CPU-mapped pixel data while the render var map context is still open.

• Preserve-aspect display creates letterboxing. All pick and camera coordinates must pass through the same letterbox transform.

• If render resolution changes, recreate or resize all dependent state: render product, ImageBridge, letterbox math, pending pick coordinate state, and viewport overlays.

• Do not run scene load/reset work while the image update is reading render output.

• A nonblank direct LdrColor artifact plus a blank ovui window points at presentation, not scene setup. Fix or switch the ovui presentation path before changing camera, render product, or USD composition.

• If ImageBridge or ByteImageProvider updates do not paint in the active ovui runtime, validate with another ovui-native presentation path such as a RasterImageProvider screenshot/frame fallback instead of continuing to debug renderer state.

Decision points:

• If the target is a simple viewer, keep render resolution fixed and let the image widget scale with letterboxing.

• If the target requires pixel-perfect viewport resolution, treat resize as an explicit render product reconfiguration path.

• If the app needs a screenshot command, read from the same copied LdrColor frame used by display unless a higher-quality render path is requested.

• If CPU readback is too slow, investigate a GPU-native UI path only after the basic Omniverse Realtime Viewer is correct.

Common failure modes:

• Reading pixels after the map context closes returns invalid or stale data.

• Ignoring letterboxing makes picks offset and camera drag speed inconsistent.

• Recreating the image bridge on every frame causes flicker, memory churn, or UI stalls.

• Changing render resolution without updating viewport math and pick query coordinates causes selection to drift.

• A solid red, brown, or unexpectedly light background can be a style-color byte order problem. ovui integer colors are 0xAARRGGBB.

Read for depth: see references/local-viewer, references/ovrtx-rendering, and references/object-selection for image bridge, frame extraction, and coordinate mapping details.