Omniverse Realtime Viewer

Streaming Messages

Triggers

Use this skill for custom streaming message, data channel, event_type, openStageRequest, getChildrenRequest, getPropertiesRequest, getPrimCountRequest, getStatsRequest, selectPrimsRequest, setViewportInputActive, changeAOVRequest, activeAOVState, availableAOVsResult, variants, or message protocol.

Application messages use this envelope in both directions:

{"event_type": "<MessageType>", "payload": {}}

Input events normally are not JSON messages. WebRTC input arrives through NVST's native input channel as binary InputEvent structs and reaches streaming-server on_input. SHM Python clients must use ovstream.Client(ovstream.ClientType.SHM, stream_name="...").send_input_event(event); C clients use ovstream_client_send_input_event(client, event). In-process clients should call the local Python/C++ APIs directly. A browser-only JSON mouseInput fallback is allowed only after viewer-input-routing's explicit native-callback failure diagnostic; it must be mutually exclusive with native input.

ovstream Callback Split

Keep the StreamSDK callback responsibilities separate:

The browser does not implement camera math. Normally the WebRTC streaming library forwards raw input through NVST/ovstream; the Python server handles orbit, pan, zoom, drag-threshold click detection, and picking. The verified fallback only serializes normalized pointer intent to that same server path.

Use the transport contract and the server adapter for encoding, dispatch, and replies. Put application correlation fields inside payload, and echo them there in both success and error responses. The browser SDK rebuilds the outer envelope, so custom top-level fields are not a reliable roundtrip contract:

{"event_type":"getPropertiesRequest","payload":{"request_id":"properties-17","prim_path":"/World/Cube"}}

Resolve pending work from event.payload.request_id; keep any legacy top-level ID support inside the app adapter. Preserve stage/session generation checks when settling the request so delayed results cannot update a replaced scene.

WebRTC server routing:

stream_server.on_connection = message_handler.on_connection
stream_server.on_app_message = message_handler.on_message
stream_server.on_input = message_handler.on_input
# Optional for composed text input:
# stream_server.on_unicode = message_handler.on_unicode

Route native input through viewer input routing to the runtime queue, where camera motion and picking share the displayed frame's coordinates. Keep this path separate from JSON app commands.

Message Reference

Exact event names matter. Current apps route getChildrenResult and getPropertiesResponse; older notes may say getChildrenResponse or getPropertiesResult. Accept old aliases when practical, but emit and document getPropertiesResponse for selected-prim properties.

mouseInput is not a general extension point. When the documented fallback is active, use kind: "mouse_move", "mouse_button", or "mouse_wheel"; x and y are render-product pixels, button uses the shared 0/1/2 convention, down is present for button events, and delta_y is present for wheel events. Reject malformed or out-of-bounds fallback messages, cancel state on mode changes, and never process both mouseInput and native on_input for one gesture.

getPropertiesResponse.prim_path is the response correlation key. Browser inspectors should compare it against the current selected prim path stored in a ref or resolver map, not against stale React state captured when the message handler was registered.

Feature-specific messages should follow the same envelope and compatibility rules, but they are opt-in. Advertise support through render/settings capabilities, backend feature flags, or an app-specific getCapabilities response before showing controls. Keep names semantic and transport-neutral: runtimeTransformRequest, pickEffectRequest, and physicsImpulseRequest are better extension points than commands named after one delivery path or one library call.

Physics commands target the configured simulator owner; see physics integration for the stage handoff.

Payload Shapes

{"event_type":"openStageResult","payload":{"url":"samples/samples_data/stage01.usd","result":"success","root_prim_path":"/World"}}

{"event_type":"getChildrenResult","payload":{"prim_path":"/World","children":[{"name":"Cube","path":"/World/Cube","type":"geom","children":true},{"name":"Light","path":"/World/Light","type":"light","children":false}]}}

{"event_type":"getPropertiesResponse","payload":{"prim_path":"/World/Cube","properties":{"typeName":"Mesh","visibility":"inherited","xformOp:translate":[0,1,0]},"truncated":false}}

{"event_type":"getVariantsResponse","payload":{"prim_path":"/World/Car","variants":{"color":{"options":["red","blue"],"selection":"red"}}}}

{"event_type":"changeAOVRequest","payload":{"aov":"NormalSD"}}

{"event_type":"activeAOVState","payload":{"active":"NormalSD","available":["LdrColor","HdrColor","NormalSD","InstanceSegmentationSD","SemanticSegmentationSD","DistanceToImagePlaneSD","DiffuseAlbedoSD"],"result":"success","previous":"LdrColor"}}

{"event_type":"availableAOVsResult","payload":{"aovs":["LdrColor","HdrColor","NormalSD","InstanceSegmentationSD","SemanticSegmentationSD","DistanceToImagePlaneSD","DiffuseAlbedoSD"],"available":["LdrColor","HdrColor","NormalSD","InstanceSegmentationSD","SemanticSegmentationSD","DistanceToImagePlaneSD","DiffuseAlbedoSD"]}}

paths: [] in selectPrimsRequest clears selection.

{"event_type":"setViewportInputActive","payload":{"active":false}}

Use setViewportInputActive only as an app UI ownership hint. In native mode, mouse, keyboard, wheel, and touch events travel through the native input channel; the verified mouseInput fallback replaces that path only after its diagnostic passes. The server should cancel active camera gestures when the flag changes to false.

Server Handler Map

Detailed server handler dispatch guidance lives in server-handler-map.md.

Selection Sync

Selection is bidirectional:

• Viewport clicks update server selection, highlight, animation, server-side overlays, and then broadcast stageSelectionChanged.

• Tree clicks must send selectPrimsRequest {paths} to the server; local React state alone does not update RTX highlight or animation.

• The server's selectPrimsRequest handler should perform the same selection side effects as the pick path before broadcasting canonical selection.

def _handle_select_prims(self, payload: Dict[str, Any]) -> None:
    paths = [p for p in payload.get("paths", []) if isinstance(p, str)]
    prev_selected = set(self.server.selected_prims)
    new_selected = set(paths)
    self.server.selected_prims = paths

    if self.server._highlight_mgr:
        self.server._highlight_mgr.update_selection(new_selected, prev_selected)
    if self.server._animator:
        for path in new_selected - prev_selected:
            self.server._animator.select(path)
        for path in prev_selected - new_selected:
            self.server._animator.deselect(path)

    self.send_message("stageSelectionChanged", {"prims": self.server.selected_prims})

On the frontend, StageTree row selection should call:

sendMessage({ event_type: 'selectPrimsRequest', payload: { paths: selectedPaths } });

Dynamic Root Prim

Do not assume every stage uses /World. The reference server asks the pxr worker for a hierarchy root after loading:

def cmd_get_root_prim_path() -> Dict[str, Any]:
    if not _stage:
        return {"ok": False, "error": "no stage loaded"}
    world = _stage.GetPrimAtPath("/World")
    if world.IsValid():
        return {"ok": True, "path": "/World"}

    default_prim = _stage.GetDefaultPrim()
    if default_prim and default_prim.IsValid():
        return {"ok": True, "path": str(default_prim.GetPath())}

    for child in _stage.GetPseudoRoot().GetChildren():
        return {"ok": True, "path": str(child.GetPath())}

    return {"ok": True, "path": "/"}

Cache the result on the server as current_stage_root_path, include it in openStageResult, and use it for the initial getChildrenResult push:

self.send_message("openStageResult", {
    "url": active_url,
    "result": "success",
    "root_prim_path": root_path,
})
children = server._pxr.get_children(root_path)
self.send_message("getChildrenResult", {"prim_path": root_path, "children": children})

Stage Reload Semantics

Open-stage requests should be idempotent. Normalize paths before deciding whether a request points at the already loaded stage:

if not force and self.current_stage_url:
    requested_key = os.path.normcase(os.path.abspath(url))
    current_key = os.path.normcase(os.path.abspath(self.current_stage_url))
    if requested_key == current_key:
        logger.info("Stage already loaded, skipping reload: %s", url)
        return True

For explicit reset/reload messages, call the load path with force=True so a same-path reload is not optimized away:

def _handle_reset_stage(self, payload: Dict[str, Any]) -> None:
    if server.current_stage_url:
        server._load_stage(server.current_stage_url, force=True)

AOV Messages

AOV state is synchronized through normal data-channel messages; the video stream itself stays unchanged.

def _send_aov_state(self, extra: Optional[Dict[str, Any]] = None) -> None:
    available = server.get_available_aovs()
    active = getattr(server, "_active_aov", "LdrColor")
    payload = {"active": active, "available": available}
    if extra:
        payload.update(extra)
    self.send_message("activeAOVState", payload)
    self.send_message("availableAOVsResult", {"aovs": available, "available": available})

def _handle_change_aov(self, payload: Dict[str, Any]) -> None:
    requested = payload.get("aov") or payload.get("name")
    if not isinstance(requested, str) or not requested:
        self._send_aov_state({"result": "error", "reason": "Missing AOV name"})
        return

    previous = getattr(server, "_active_aov", "LdrColor")
    if server.set_active_aov(requested):
        self._send_aov_state({"result": "success", "previous": previous})
        return

    self._send_aov_state({
        "result": "error",
        "requested": requested,
        "reason": "AOV is not available for the current render product",
    })

toggleSegView should remain as a compatibility shim. Map it to InstanceSegmentationSD when enabled and LdrColor when disabled, then send normal AOV state.

On the frontend, accept both AOV event shapes:

case 'activeAOVState': {
  const payload = event.payload as ActiveAOVStatePayload;
  if (Array.isArray(payload.available) && payload.available.length > 0) {
    setAvailableAOVs(payload.available);
  }
  setActiveAOV(payload.active || 'LdrColor');
  break;
}
case 'availableAOVsResult': {
  const payload = event.payload as AvailableAOVsResultPayload;
  const names = payload.aovs || payload.available || [];
  if (names.length > 0) {
    setAvailableAOVs(names);
  }
  break;
}

Data-Channel Size Limit

Some WebRTC data-channel paths fail above roughly 65535 bytes per message. getPropertiesResponse is the common offender on complex prims.

Cap payloads before sending:

MAX_MESSAGE_BYTES = 60000

def capped_event(event_type: str, payload: dict) -> tuple[dict, bool]:
    encoded = json.dumps({"event_type": event_type, "payload": payload}, default=str)
    if len(encoded.encode("utf-8")) <= MAX_MESSAGE_BYTES:
        return payload, False
    if "properties" in payload:
        trimmed = {}
        omitted = 0
        for key, value in payload["properties"].items():
            candidate = {**payload, "properties": {**trimmed, key: value}}
            size = len(json.dumps({"event_type": event_type, "payload": candidate}, default=str).encode("utf-8"))
            if size > MAX_MESSAGE_BYTES:
                omitted += 1
            else:
                trimmed[key] = value
        return {**payload, "properties": trimmed, "truncated": True, "omitted_count": omitted}, True
    return {**payload, "truncated": True, "error": "payload too large"}, True

Prefer a capped single response over chunking unless the frontend already supports chunk assembly. If adding pagination, make it opt-in with request fields such as max_bytes, offset, or cursor, and keep the original response shape for older clients.

For selected-prim property panels, avoid sending full mesh buffers in the first place. Serialize array attributes such as points, normals, faceVertexIndices, faceVertexCounts, and primvars:st* as {length, preview, truncated} summaries unless the user explicitly requests a full geometry dump or paginated array viewer.

Adding A Message

If a project has server/config.py, add a message constant there; otherwise a literal string is acceptable. If generating code files, include these parts:

1. Your app's message router (suggested server/message_router.py, or an existing handler module): handler function, dictionary entry, payload validation, and guarded send helper call. Build this adapter using server-handler-map.md; no Python router is shipped with this package.

2. Runtime or worker module: a render-thread command for live viewer state, a USD query worker for OpenUSD reads, or the configured physics owner when the handler needs those capabilities.

3. frontend/src/types/usd.ts: TypeScript payload interfaces and discriminated event type.

4. frontend/src/App.tsx or the relevant component: sendMessage({ event_type, payload }) and response routing in onCustomEvent.

def _handle_my_feature(self, payload):
    result = self._do_something(payload.get("some_param", ""))
    self._send_message("myFeatureResponse", {"result": result})

sendMessage({ event_type: 'myFeatureRequest', payload: { some_param: 'value' } });

Backward Compatibility Rules

• Never change the outer {event_type,payload} envelope.

• Add optional payload fields; do not rename required fields in place.

• Accept known aliases (makePrimsPickable/makePrimsSelectable, resetStage/resetStageRequest, aov/name) and normalize internally.

• Unknown request fields should be ignored, not treated as fatal.

• Unknown event types should log a warning and return an error response only if the frontend expects one.

• Optional feature messages should fail closed when the backend capability is absent or when the selected runtime cannot verify a safe apply path.

• Keep request/response names stable across server and frontend; verify the active onCustomEvent router before changing names.

• Send current state on connect for browser clients that attach after startup.

See also: aov-switching, streaming-client, streaming-server, streaming-lifecycle, viewer-input-routing, stage-hierarchy, stage-management.