Omniverse Realtime Viewer

Streaming Server Runtime

2. Install Dependencies And Configure The Environment

Read references/dependencies FIRST. Its references/nvidia-runtime.md file is the source of truth for NVIDIA runtime locations. Do not guess install paths and do not repeat ovstream, ovui, ovrtx, or ov-web-rtc client acquisition details in this recipe.

Do this on the server side:

• Install ovrtx using the package guidance in references/dependencies.

• Install ovstream using the current package guidance in references/dependencies. If ovstream.initialize() reports missing native StreamSDK libraries, follow the current dependency guidance.

• Install warp-lang for CUDA-side channel conversion (pip install warp-lang).

• Install numpy for matrices and camera math (pip install numpy).

• Install usd-core==24.11 only when the server process needs direct pxr queries. Pin to 24.11 — newer versions cause TfType schema conflicts with ovrtx. Prefer subprocess query mode on Windows or when USD registry conflicts appear.

For generated browser viewers, create app-local setup and run wrappers rather than pointing users at repo-level scripts. The setup wrapper should install the server dependencies above, run the verification checks from references/dependencies/environment-validation.md, and fail with the exact command that did not pass. The run wrapper should set OVRTX_SKIP_USD_CHECK, derive OVRTX_BIN_PATH from the installed ovrtx package when needed, preserve the selected ovstream package layout, start the server, and write logs to a known project-local path.

Do this on the frontend side:

• Create a Vite React app (npm create vite@latest frontend -- --template react-ts).

• Configure and install @nvidia/ov-web-rtc using references/dependencies. Use only the standalone ovstream Direct connection pattern from streaming-client; do not use Kit, OVC, NVCF, or GFN client connection profiles in the browser WebRTC config. If OKAS or another orchestrator launches the container, convert its exposed endpoint into Direct server and signalingPort values.

• Add normal UI dependencies only when they are part of the requested UI. Keep the core connection path dependency-light.

Set these environment contracts before starting the server:

• OVRTX_SKIP_USD_CHECK=1 must be set before ovrtx is imported or the renderer is constructed.

• OVRTX_BIN_PATH must point at the ovrtx bin directory when materials or renderer plugins fail to resolve.

• The ovrtx plugin library path must be first in the dynamic library path if another USD build is present.

• OVSTREAM_LIB_PATH must point at the ovstream native library directory when the Python binding cannot find native libraries.

• If ovstream native libraries are not found, use the current runtime guidance from references/dependencies instead of copying stale fallback library paths from older recipes.

Decision points:

• If the target is local development, use WebRTC signaling port 49100, stream port defaulting through ovstream, and public IP 127.0.0.1.

• If the target is LAN access, expose signaling host and port through frontend URL parameters and environment variables.

• If the target is cloud deployment, do not invent a new deployment model in this recipe; read references/cloud-deployment and keep to supported paths.

• If running on Windows with pxr imports for advanced USD inspection, prefer a separate USD query subprocess rather than importing pxr in the ovrtx render process.

Common failure modes:

• usd-core detected, duplicate USD debug symbols, _tf import failures, or MDL resolver crashes usually mean import order or library path is wrong.

• Magenta materials usually mean OVRTX_BIN_PATH or plugin library path is missing.

• ovstream import or initialize failure usually means native StreamSDK libraries are missing, OVSTREAM_LIB_PATH points at the wrong layout, the platform build does not match, or GPU/driver support is missing.

• No matching distribution found for ovrtx: wrong package guidance, unsupported platform, or unsupported Python version; re-check references/dependencies.

• ovstream package not found: wrong package source, stale package metadata, wrong platform tag, or network/proxy issue; re-check references/dependencies.

Read for depth: see references/dependencies for install commands, references/ovrtx-rendering and references/streaming-server for the full environment and native library contracts.

3. Build The Server Runtime Shell

Do this:

• In server/ov_web_viewer_server.py, parse width, height, target FPS, signaling port, public IP, initial scene URL/path, asset root, and settings path.

• Construct a single application runtime object that owns the renderer runtime, scene manager, stream server, message router, input router, settings store, and command queue.

• Follow the canonical startup order: set env, construct an ovrtx renderer with native selection outline support when needed, load the initial stage through open_usd() or open_usd_from_string(), warm up the renderer, initialize ovstream, register callbacks, start readiness health, then start the render loop.

• Initialize ovstream once, create a WebRTC server, register on_connection, on_message, and on_input, then start the server.

• Enter one render loop that drains queued commands, updates camera/selection/settings state, steps ovrtx when a scene is loaded, converts the frame to BGRA, and submits video to ovstream.

• On shutdown, stop streaming, close the ovstream server, close renderer resources if exposed by the API, and call ovstream shutdown exactly once for each initialize call.

Canonical startup sequence:

set OVRTX_SKIP_USD_CHECK=1
import ovrtx and construct Renderer(RendererConfig(sync_mode=True, selection_outline_enabled=True))
load initial stage:
  build an inline root USDA that sublayers the user stage and authors viewer render config
  renderer.open_usd_from_string(inline_root_usda)
  bind /OVCamera omni:xform
  initialize native selection outline styles and clear outline groups
warm up renderer:
  write camera transform
  step /Render/OVServer/ViewportTexture0 several frames
  probe render vars and allocate persistent BGRA stream buffer
initialize ovstream and register callbacks:
  on_connection -> push_initial_state
  on_message -> JSON app protocol
  on_input -> raw mouse/keyboard input routed through viewer-input-routing
start /healthz:
  503 until first valid frame
  200 after first successful converted frame
start render loop thread

Critical contracts:

• Register callbacks before server.start().

• Keep callbacks fast. They may be called from StreamSDK internal threads.

• Do not call renderer load/reset/step/write APIs directly from ovstream callbacks. Enqueue work for the render loop.

• Guard send_message with the connected-client state.

• Maintain explicit runtime states: starting, idle/no scene, loading, streaming, error, shutting down.

• Keep the most recent loaded stage URL, root hierarchy summary, selection, render settings, and loading state in server memory so a newly connected browser can receive initial state.

• Readiness is not liveness: /healthz must return 503 not ready until the render loop has produced and copied the first valid frame, then 200 ok.

• Readiness must not depend on an active browser client or on stream_video() succeeding. Before any client connects, frame submission can no-op or raise transient no-client/disconnect errors depending on the selected ovstream build.

Decision points:

• If no initial scene is configured, start the server in idle state and let the frontend send openStageRequest.

• If an initial scene is configured, load it before or during the first render loop iteration, then push state when the client connects.

• If one client is already connected and a second browser connects, follow ovstream's one-client WebRTC constraint. Either reject the new client clearly or replace the old session intentionally.

• If the render loop falls behind target FPS, prefer dropping frames or reducing render quality over running concurrent renderer steps.

Common failure modes:

• Registering callbacks after start() causes missed connection and data-channel events.

• Calling renderer.step() during reset_stage(), open_usd*(), or reference mutation causes races, stale buffers, or crashes.

• Sending messages while no client is connected silently loses important state unless the runtime pushes initial state after connection.

Read for depth: see references/streaming-server and references/streaming-lifecycle for the full lifecycle contract.

4. Construct The ovrtx Renderer

Do this:

• Create the renderer in server/renderer_runtime.py after environment variables are set.

• Use synchronous rendering first. Add asynchronous rendering only after buffer lifetime, readiness, and stream pacing are explicitly handled.

• Store the active render product path, stream width, stream height, current frame index, and whether a valid stage is loaded.

• Expose render-loop-only operations for loading a scene, resetting the stage, stepping a frame, mapping render vars, enqueueing and decoding pick queries, setting selection outline groups, and writing live attributes.

Critical contracts:

• The application calls renderer.step() explicitly. ovrtx does not run a hidden app loop for the viewer.

• Pass the exact viewer RenderProduct path to every step call.

• Extract LdrColor from the returned frame. This is RGBA8 from ovrtx.

• For AOVs, handle both single-tensor and multi-tensor render var outputs. Single-tensor outputs are consumed directly through DLPack; multi-tensor outputs must select a named image tensor and read params separately. Image tensors are channel-last (H x W x C), not channel-first.

• Map CUDA buffers for streaming. Avoid CPU round trips in the normal streaming path.

• Keep mapped or converted frame buffers alive until stream_video() returns.

• Use write_attribute for live camera transforms and other live state. Write omni:xform, not authored xformOp:*, for interactive updates.

• Use the correct transform semantic and create-new prim mode for attributes that may not already exist in Fabric.

Decision points:

• If the app only needs basic video streaming, start with LdrColor only.

• Object selection uses native pick queries. Do not add segmentation render vars just to make picking work.

• If render settings include debug segmentation view, keep both render vars available and switch the streamed image source intentionally.

• If the renderer reports stale GPU hangs after a crash, inspect running Python GPU processes before changing code.

Common failure modes:

• Unable to find RenderProduct prim means scene setup did not create the path used by renderer.step().

• Black frame usually means camera relation, render product resolution, render var source, or camera transform is invalid.

• Red/blue color swap means ovrtx RGBA was submitted to ovstream without BGRA conversion.

• Live camera changes doing nothing usually means the app wrote xformOp:transform instead of omni:xform, or used existing-only prim mode.

Read for depth: see references/ovrtx-rendering for the full renderer construction, frame extraction, and live attribute contract.

5. Implement Scene Loading

Do this in server/scene_loader.py and call it only from the render loop:

• Resolve the requested URL/path against the configured asset root, allowed schemes, and security policy.

• Create viewer-owned camera, RenderProduct, RenderVar, and RenderSettings data in one inline root USDA string when the user stage lacks viewer render config.

• Load the user stage without modifying it.

• Store the viewer camera path and render product path in runtime state.

• Reset selection, native selection outline groups, hierarchy cache, pending pick queries, and loading progress for the new stage.

• Fit the camera to the stage bounds unless the user requested preserving the current camera or using an authored stage camera.

Critical contracts:

• Every loaded stage needs Camera -> RenderProduct -> RenderVar -> RenderSettings wiring that ovrtx can find.

• The viewer camera path must be the same path used by camera controls when writing omni:xform.

• Do not inject lights unless the user explicitly asks for viewer-controlled lighting. User stages usually own their lighting.

• Include segmentation render vars only for explicit debug/AOV display modes, not for picking.

• Prefer renderer.open_usd_from_string() for inline roots that sublayer the user USD and author viewer render config. This avoids temporary file lifetime issues while preserving relative asset resolution through the sublayer path.

• Do not call reference or layer-add APIs after a stage is already loaded unless the renderer has been reset to an empty stage and the operation is part of the serialized load path.

Decision points:

• Use a single inline root USDA string with subLayers = [@user_scene@] when the user file needs viewer camera/render-product/render-var data.

• If the user stage has an authored camera and the requested policy is stage-camera, copy its focal length, apertures, clipping range, and transform into the viewer camera.

• If the user requests persistent camera across scene switches, keep camera state but sanitize and refit only when the old state is invalid for the new bounds.

• If the user requests viewer lighting controls, add explicit viewer-owned light prims only with a verified live apply path or an explicit reload/profile workflow; otherwise leave lighting untouched and omit live lighting controls.

Common failure modes:

• Inline roots that omit or misquote the user sublayer path fail composition or break relative asset resolution.

• Camera path mismatch makes input appear connected but the view never moves.

• A stage-load operation that reports an error must not be treated as a successful load just because the enqueue call returned.

Read for depth: see references/stage-loading, references/render-settings, references/selection-feedback, and references/stage-hierarchy for the full contracts.

6. Build Frame Streaming

Do this in server/stream_server.py and server/frame_converter.py:

• Start ovstream in WebRTC mode.

• Configure width, height, target FPS, signaling port, optional public IP, and video codec policy.

• For each rendered frame, map LdrColor on CUDA, convert RGBA8 to BGRA8 on CUDA, wrap the CUDA buffer as an ovstream video frame, and submit it.

• Pace the render loop to target FPS while allowing command queue work to run between frames.

• Send loading and error state messages even when video frames are temporarily unavailable.

• Log the first valid converted BGRA frame and set readiness before depending on browser connection state. This lets validation distinguish renderer/frame-conversion failures from WebRTC negotiation failures.

Critical contracts:

• ovstream expects BGRA8 for raw CUDA frames; ovrtx LdrColor is RGBA8.

• Avoid CPU readback in the normal path. CPU readback is acceptable only for debugging screenshots or tests.

• stream_video() does not own a deep copy of the source buffer. Keep the buffer alive until the call returns.

• stream_video() can fail during disconnect races. Catch and debug-log transient failures; do not crash the render loop.

• First-frame readiness should be set after render-var mapping and RGBA-to-BGRA conversion into the persistent app-owned stream buffer. Do not wait for an attached browser client.

• Stream width, height, frame pitch, RenderProduct resolution, and camera aspect must agree for the fixed stream size.

• The WebRTC signaling port is not the media port. The browser discovers media endpoints through SDP.

• For local WebRTC development, set the server public IP to 127.0.0.1 when ICE discovery otherwise hangs.

Decision points:

• Use raw frames first. Add H264, H265, AV1, or custom encoded frames only when latency, bandwidth, or deployment constraints require it.

• If using encoded frames, make the encoder own color format conversion and frame lifetime explicitly.

• If no client is connected, still run scene loading and state updates, but consider pausing expensive frame submission work.

• If the user asks for RTSP, treat it as a separate delivery mode; the browser-streamed Omniverse Realtime Viewer path should remain WebRTC.

Common failure modes:

• Connected browser with no video often means frontend configured a media port manually, server ICE/public IP is wrong, or frame submission never happens.

• One frame then black usually means the frontend connected, received state, rerendered, and accidentally terminated/restarted the AppStreamer connection. Inspect server logs for connected=True followed by connected=False within about a second.

• Wrong colors mean missing RGBA-to-BGRA conversion.

• GPU memory growth can mean mapped frame resources or conversion buffers are retained beyond the intended frame lifetime.

Read for depth: see references/streaming-server and references/streaming-lifecycle for the full frame streaming contract.