Omniverse Realtime Viewer

Streaming Server

Triggers

Use this skill for stream to browser, WebRTC server, ovstream server, stream_video, on_input, VideoFrame, RGBA BGRA, fixed resolution, or no video.

Use this for the Python server that renders with ovrtx and streams frames through ovstream.

GPU Requirement

ovrtx requires an NVIDIA GPU. Build and document the ovrtx server path as the only rendering path; do not add CPU rendering, WebGL, Three.js, Babylon.js, glTF viewer, or other client-side rendering substitutes.

Native Library Setup

Read references/dependencies before installing or locating ovstream. references/dependencies/nvidia-runtime.md is the source of truth for the current package source. For ovstream server, SHM, native input, or release-specific behavior not covered here, read the supplemental dependency documentation referenced by references/dependencies.

For ovrtx Python/C API behavior or release-specific server integration details not covered here, read references/dependencies for acquisition guidance and supplemental dependency documentation.

If the installed runtime cannot locate native libraries automatically, set:

export OVSTREAM_LIB_PATH=/path/to/ovstream/lib/   # directory with .so/.dll files

Bundled Native Library

The ovstream Python wheel bundles its own libovstream.so. Do not override it by placing a separate libovstream.so on LD_LIBRARY_PATH or in OVSTREAM_LIB_PATH unless you are deliberately using a different version for a specific transport (e.g., a newer SHM transport build). Overriding with a mismatched .so causes symbol errors or silent protocol failures.

Display Requirement

ovrtx requires an X11 display for GPU rendering, even in headless deployments. Use Xvfb when no physical display is available:

Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99

Without a display, ovrtx initialization will fail with EGL/GLX errors.

Canonical Startup Sequence

The reference WebRTC server starts in this order. Keep the ordering when generating server/ov_web_viewer_server.py or equivalent runtime shells:

1. Set OVRTX_SKIP_USD_CHECK=1 before importing ovrtx or any module that can import pxr.

2. Import ovrtx, construct Renderer(RendererConfig(sync_mode=True, selection_outline_enabled=True)), then import sibling helpers. Use ovrtx stage queries for basic prim discovery; keep a pxr_worker.py subprocess only for USD features that still require OpenUSD.

3. Import and initialize CUDA helpers such as warp for frame conversion.

4. Load the initial stage if one is configured: build one inline root USDA string that sublayers the user file and authors viewer camera/render-product/render-var data, call renderer.open_usd_from_string(...), bind or write the camera omni:xform, initialize native selection outline styles, and cache current_stage_root_path.

5. Warm up the renderer before starting ovstream: step several frames against the canonical render product, update camera transforms, probe render vars, allocate the persistent BGRA stream buffer, and discover the currently available display AOVs.

6. Initialize ovstream, create ovstream.Server(ovstream.ServerType.WEBRTC), register on_connection, on_message, on_input, and on_unicode callbacks where needed, then call server.start(ServerConfig(...)).

7. Start the /healthz endpoint before or alongside server startup. It must return 503 not ready until the renderer has produced and copied one valid display frame into the app-owned stream buffer, then 200 ok after that.

8. Start exactly one render loop thread. That thread owns renderer.step(), frame conversion, native pick-query enqueue/result decoding, selection-outline state writes, animation updates, and stream_video().

Skeleton:

import os
os.environ["OVRTX_SKIP_USD_CHECK"] = "1"

from ovrtx import Renderer, RendererConfig, Device, PrimMode
import ovstream
import warp as wp

from healthz import HealthServer
from message_handler import MessageHandler

wp.init()
renderer = Renderer(config=RendererConfig(sync_mode=True, selection_outline_enabled=True))

health = HealthServer()
health.start()

load_initial_stage(renderer)
warm_up_renderer(renderer, render_product="/Render/OVServer/ViewportTexture0")

ovstream.initialize(log_fn=stream_log, log_min_severity=ovstream.LogLevel.VERBOSE)
stream = ovstream.Server(ovstream.ServerType.WEBRTC)
handler = MessageHandler(server_runtime)
stream.on_connection = handler.on_connection
stream.on_message = handler.on_message
stream.on_input = handler.on_input
if hasattr(handler, "on_unicode"):
    stream.on_unicode = handler.on_unicode
config = ovstream.ServerConfig(width=1920, height=1080, video_input=ovstream.VideoInput.CUDA)
config.webrtc_signal_port = 49100
config.webrtc_public_ip = public_ip or "127.0.0.1"
stream.start(config)

threading.Thread(target=render_loop, daemon=True).start()

Lifecycle

import ovstream
from ovstream import LogLevel, ServerType, ServerConfig, VideoFrame

# LogLevel enum values: DEFAULT, ERROR, INFO, NONE, VERBOSE, WARNING
# Note: there is no WARN variant — use WARNING.
ovstream.initialize(
    log_fn=lambda level, channel, msg, timestamp: print(f"[{level.name}] {channel}: {msg}"),
    log_min_severity=LogLevel.WARNING,
)
server = ovstream.Server(ServerType.WEBRTC)
server.on_connection = on_connection
server.on_input = on_input
server.on_message = on_message
# Optional for composed text input:
# server.on_unicode = on_unicode
server.start(ServerConfig(
    width=1920,
    height=1080,
    target_fps=60,
    stream_port=0,
    video_input=ovstream.VideoInput.CUDA,
    webrtc_signal_port=0,
))
try:
    while running:
        cuda_buffer = render_bgra8_cuda_frame()
        server.stream_video(VideoFrame(buffer=cuda_buffer, width=1920, height=1080, pitch_bytes=1920 * 4))
finally:
    server.stop()
    server.close()
    ovstream.shutdown()

initialize() is ref-counted; every call needs a matching shutdown(). Register callbacks before start() so initial connection/input/message events cannot race past handlers.

Guard server sends and frame submission against disconnect races. A client can disconnect between is_client_connected and send_message(), or during stream_video(). Those transient failures should not crash the render loop:

def send_event(server, event_type: str, payload: dict) -> None:
    if not server.is_client_connected:
        return
    try:
        server.send_message(json.dumps({"event_type": event_type, "payload": payload}, default=str))
    except Exception:
        logger.debug("Dropping event during disconnect: %s", event_type, exc_info=True)

def stream_frame(server, frame: ovstream.VideoFrame) -> None:
    try:
        server.stream_video(frame)
    except Exception:
        logger.debug("Dropping frame during disconnect", exc_info=True)

Frame Loop And Continuity

Detailed frame-source, fixed-resolution, and stage-load continuity guidance lives in frame-loop-and-continuity.md.

Readiness Health Gate

Expose readiness separately from process liveness. Orchestrators and load balancers must not send clients to the service until the renderer has produced and converted a valid frame.

from http.server import HTTPServer, BaseHTTPRequestHandler
from threading import Event, Thread

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_response(404); self.end_headers(); return
        if self.server.ready_event.is_set():
            self.send_response(200); self.end_headers(); self.wfile.write(b"ok")
        else:
            self.send_response(503); self.end_headers(); self.wfile.write(b"not ready")

ready = Event()
httpd = HTTPServer(("0.0.0.0", 8081), HealthHandler)
httpd.ready_event = ready
Thread(target=httpd.serve_forever, daemon=True).start()

# In the render loop, after the first successful render-var map, copy,
# and RGBA-to-BGRA conversion into the app-owned stream buffer:
ready.set()

Do not mark readiness when the process starts, when ovstream starts, or when a stage-load operation returns. Mark it only after a valid frame path has succeeded.

Readiness must not depend on an active browser client or on server.stream_video() succeeding. Before any client connects, stream_video() may be a no-op or may raise a transient no-client/disconnect error depending on the selected ovstream build. A server that has already rendered and converted a valid frame should report ready even while no client is attached.

Generated viewers should log the first converted frame before entering normal streaming, for example First BGRA frame ready: WIDTHxHEIGHT. Use that log line to separate renderer/camera/frame-conversion failures from browser/WebRTC negotiation failures.

RGBA To BGRA

ovrtx outputs RGBA8. ovstream expects BGRA8. Without conversion, red/blue channels swap.

import warp as wp

@wp.kernel
def swap_rb(img: wp.array3d(dtype=wp.uint8)):
    i, j, k = wp.tid()
    if k == 0 or k == 2:
        r = img[i, j, 0]
        b = img[i, j, 2]
        img[i, j, 0] = b
        img[i, j, 2] = r

with frame.render_vars["LdrColor"].map(device=ovrtx.Device.CUDA) as var:
    wp_array = wp.from_dlpack(var)
    wp.launch(swap_rb, dim=(h, w, 4), inputs=[wp_array])
    server.stream_video(ovstream.VideoFrame.from_cuda_array(wp_array))

No CPU round trip is needed.

Production AOV Conversion Before stream_video()

Every displayed render var must be converted into a persistent CUDA uint8 [H,W,4] BGRA buffer before creating ovstream.VideoFrame. Keep LdrColor as the fallback if the active AOV cannot be copied.

ovrtx 0.3 render vars can be single-tensor or multi-tensor outputs. For a single-tensor render var, consume the mapped object directly with DLPack. For a multi-tensor render var, choose the named tensor that represents the image payload and read params separately. Image tensors are channel-last: H x W, H x W x 1, H x W x 3, or H x W x 4. Do not assume C x H x W, and do not use old .tensor access in new generated code.

copied = copy_aov_to_stream_buffer(fout, active_aov)
if not copied and active_aov != "LdrColor":
    copied = copy_aov_to_stream_buffer(fout, "LdrColor")
if copied:
    video_frame = ovstream.VideoFrame.from_cuda_array(stream_bgra_buffer)
    stream_server.stream_video(video_frame)

Pick queries are independent of the displayed AOV. Do not update a segmentation-derived pick buffer in generated ovrtx 0.3 apps.

Native Picking And Selection Outlines

Use ovrtx native pick queries and native selection outline state:

1. Convert the ovstream input coordinate to render-product pixel space.

2. Enqueue renderer.enqueue_pick_query_async(...) with a 1x1 rectangle for click picking or a larger rectangle for marquee selection.

3. Step the same RenderProduct. The pick result appears as the synthetic render var ovrtx_pick_hit.

4. Map the pick-hit output, validate its params such as magic and version, read the named primPath tensor, and resolve each non-zero path id with renderer.resolve_prim_path_id(...).

5. Deduplicate resolved paths and publish stageSelectionChanged.

6. Clear previous outlines by writing selection group 0, then write group 1 or another styled group to selected prims through omni:selectionOutlineGroup / OVRTX_ATTR_NAME_SELECTION_OUTLINE_GROUP.

Configure selection outlines at renderer creation with RendererConfig(selection_outline_enabled=True, selection_outline_width=...). Configure per-group colors at runtime with Renderer.set_selection_group_styles(...). Changing global width or fill mode requires recreating the renderer; changing per-group colors does not.

Do not create legacy segmentation picker modules, CPU ray fallback picker modules, segmentation ID maps, isolation ID discovery, or Warp outline compositors for ovrtx 0.3 generated apps.

Operation Status And Errors

Treat stage loads, render steps, and pick queries as operations whose status must be checked:

• In Python, blocking helpers such as open_usd() may raise; async variants return operations that must be .wait()ed and fetched before the result is trusted.

• In C, load errors are reported through ovrtx_op_wait_result_t::error_op_ids; the enqueue return value only says whether the work was accepted.

• A failed or timed-out stage load must send openStageResult {result: "error"} plus viewerError, keep or restore the previous valid frame when possible, and avoid marking readiness.

• A failed render step should keep the last good frame, emit a bounded error event, and stop retrying only after repeated non-recoverable failures.

• A failed or empty pick query should clear hover state or return no path without mutating the current selection unless the user explicitly requested clear-on-miss.

Input Callback

Input is separate from JSON messages. For WebRTC, NVST forwards mouse/keyboard/gamepad input as binary InputEvent structs that arrive through server.on_input. For SHM Python clients, send the same native input struct path through ovstream.ShmClient.send_input_event(); C clients use ovstream_shm_client_send_input_event(). Do not send JSON mouseInput. Read viewer-input-routing before implementing this callback.

def on_input(event):
    if not viewport_input_active:
        camera.cancel_interaction()
        return
    if event.type == ovstream.InputEventType.MOUSE:
        mouse = event.mouse
        if mouse.type == ovstream.MouseEventType.MOVE:
            handle_mouse_move(mouse.x, mouse.y, mouse.modifiers)
        elif mouse.type == ovstream.MouseEventType.BUTTON:
            button = camera_button_from_ovstream(mouse.data, ovstream)
            if button is not None:
                handle_mouse_button(button, mouse.button_state, mouse.x, mouse.y)
        elif mouse.type == ovstream.MouseEventType.WHEEL:
            handle_scroll(mouse.scroll_y or mouse.data, mouse.x, mouse.y)
    elif event.type == ovstream.InputEventType.KEYBOARD:
        handle_key(event.keyboard.key_code, event.keyboard.key_state, event.keyboard.modifiers)
    elif event.type == ovstream.InputEventType.GAMEPAD:
        handle_gamepad(event.gamepad.control, event.gamepad.position, event.gamepad.gamepad_id)

Use this native input path for orbit/pan/zoom and viewport picking. In browser apps with DOM controls, maintain viewport_input_active from a lightweight app message such as setViewportInputActive {active}. Disable it when the pointer is over sidebars, trees, inspectors, menus, or top bars so UI clicks do not move the camera or trigger picks.

ovstream.MouseButton values are not DOM button ids: LEFT=1, MIDDLE=2, and RIGHT=3. Use viewer-input-routing to convert through the enum or an explicit mapping before passing buttons to shared camera helpers that use 0=left, 1=middle, 2=right.

When the WebRTC stream surface is the only source of native input, initialize viewport_input_active = True and let DOM panels turn it off with setViewportInputActive {active:false}. If the server starts inactive, the first mouse-down can race ahead of the React activation message, so a left-click release is seen without a matching press and click picking never queues.

ServerConfig Reference

ServerType.WEBRTC is for browser streaming, RTSP for VLC/ffplay, NATIVE for native clients, and SHM for same-machine shared-memory readers. WebRTC supports one connected client at a time; guard send_message with server.is_client_connected. Multiple network servers in one process need explicit unique ports; ovstream does not auto-increment conflicting defaults.

Ports

Signaling is TCP/WebSocket on 49100 by default. WebRTC media is UDP and negotiated by SDP; check the selected ovstream release notes for the current default media port. Do not conflate these in frontend config.

Generated Module Checklist - streaming server

• [ ] main() sets OVRTX_SKIP_USD_CHECK=1 before ovrtx imports.

• [ ] OVWebViewerServer.start() or equivalent constructs Renderer(RendererConfig(sync_mode=True, selection_outline_enabled=True)) when selection feedback is needed.

• [ ] PxrWorkerClient.start() is optional and used only for USD queries not covered by ovrtx native stage APIs.

• [ ] Initial stage load uses renderer.open_usd_from_string() with an inline root USDA when viewer render config must be injected.

• [ ] Renderer warmup steps before ovstream.Server.start().

• [ ] MessageHandler.on_message is registered for JSON app messages.

• [ ] MessageHandler.on_input is registered for raw ovstream input events.

• [ ] MessageHandler.on_unicode is registered when composed text input matters.

• [ ] /healthz returns 503 before first successful frame and 200 afterward.

• [ ] Render loop is the only owner of renderer.step().

• [ ] Render loop enqueues/decodes native pick queries and updates native selection outline groups.

• [ ] AOV conversion writes a persistent CUDA BGRA8 buffer before stream_video().

• [ ] Disconnect races around send_message() and stream_video() are caught and debug-logged.

See also: streaming-client, streaming-messages, streaming-lifecycle, ovrtx-rendering, stage-loading, viewer-input-routing, camera-controls, object-selection.