Omniverse Realtime Viewer

Streaming Interaction Features

9. Add Camera Controls

Read references/viewer-input-routing first for transport button mapping, viewport ownership, wheel deltas, and click-vs-drag dispatch. Then implement camera math in server/input_router.py and server/camera_controller.py:

• Track mouse press, movement, release, buttons, wheel, keyboard modifiers, and viewport dimensions from ovstream InputEvent callbacks.

• Convert browser input coordinates into render-image coordinates, accounting for letterboxing or scaling.

• Use drag thresholding so a short left press/release selects and a left drag orbits.

• Support orbit, pan, dolly/zoom, wheel zoom, fit-to-stage, and optional WASD fly mode when requested.

• Set camera aspect from the fixed RenderProduct resolution; keep horizontal aperture stable and derive vertical aperture from height / width.

• Sanitize camera state before input handling and before writing matrices.

• Write the viewer camera transform to the camera prim through ovrtx live attributes.

Critical contracts:

• The camera is a USD prim. Camera movement writes omni:xform on the viewer camera path.

• Use the row-vector matrix layout expected by USD/ovrtx: basis vectors in rows and translation in the final row.

• Clamp elevation away from straight up/down singularities.

• Clamp camera distance above a small positive minimum.

• Skip camera writes when any matrix value is non-finite.

• Use the same world-up convention as the scene when fitting and orbiting.

• Left-click selection fires only on release if movement stayed below the drag threshold.

• Do not use JSON messages for routine pointer movement.

• The browser displays the fixed-resolution stream with object-fit: contain; use the letterbox transform for any app-owned coordinate math, while NVST handles stream input mapping.

Decision points:

• If the user wants DCC-style navigation, use Alt+left for orbit, Alt+middle for pan, and Alt+right for dolly while preserving left-click selection.

• If the user wants simple browser navigation, use left drag orbit, middle drag pan, right drag dolly, and wheel zoom.

• If the stage has an authored camera and the app policy is to use it, initialize viewer camera settings from that camera before allowing interaction.

• If the user requests camera UI buttons, send high-level camera commands over the data channel; keep continuous pointer input in ovstream input.

Common failure modes:

• Putting matrix basis vectors in columns instead of rows places the camera inside, behind, or under the scene.

• Ignoring letterboxing makes picks and orbit centers offset from the visible image.

• Treating every left release as selection causes accidental selections after orbit drags.

Read for depth: see references/viewer-input-routing, references/camera-controls, and references/stage-hierarchy for the full input, camera math, and bounds contracts.

10. Add Object Selection And Highlighting

Do this in server/selection_controller.py; the completed click gesture should come from references/viewer-input-routing:

• On left mouse release after a click gesture, map the visible image coordinate to the render pixel coordinate and enqueue a native ovrtx pick query.

• After the next render step, read the synthetic ovrtx_pick_hit render var, validate its params, resolve primPath ids to USD prim paths, and deduplicate the result.

• Maintain selected prim paths in server state.

• Send stageSelectionChanged whenever selection changes.

• Support selectPrimsRequest from the frontend for tree-driven selection.

• Clear selection on selectPrimsRequest with an empty path list, scene reset, or scene switch.

• Apply visual selection feedback by writing native selection outline group attributes on the runtime stage, not by permanently editing the user USD.

Critical contracts:

• The pick query rectangle uses render-product pixel space after letterbox/scaling correction.

• Picking coordinates must use render-pixel space after letterbox/scaling correction.

• Selection state is server-authoritative. The frontend mirrors it from stageSelectionChanged.

• Enable selection outlines at renderer creation and set per-group outline/fill colors with Renderer.set_selection_group_styles(...).

• Clear previous outlines by writing group 0; assign selected prims to a non-zero group such as 1.

• Do not let selection picking run while the scene is loading or the renderer is resetting.

• Check operation status for the stage load, render step, and pick query before changing selection. An empty or failed pick should not corrupt the previous selected state.

Decision points:

• If the user only asks for tree selection, implement selectPrimsRequest first and defer click picking.

• If the user asks for visual highlight, use native selection outlines after selection state works.

• If the user asks for hover, multi-select, or marquee selection, extend the protocol with explicit event names and keep final selected state server-authoritative.

• If selection needs property display, trigger or expect a getPropertiesRequest for the selected prim rather than stuffing full properties into every selection event.

Common failure modes:

• Selection appears offset when coordinate transforms ignore video scaling or letterboxing.

• No pick result arrives when the query was enqueued for a different RenderProduct than the next renderer.step().

• Highlight persists across scene loads when old selection outline groups are not cleared.

• Frontend and server selection diverge when the frontend mutates local selection without waiting for stageSelectionChanged.

Read for depth: see references/viewer-input-routing, references/object-selection, references/selection-feedback, and references/prim-info-display for the full input, picking, highlighting, and info contracts.

11. Add Scene Switching And Asset Browsing

Do this in server/scene_manager.py, server/assets.py, and frontend/src/components/ScenePicker.tsx:

• Build a server-side scene registry from configured local sample paths or an allowed asset root.

• Expose scene choices to the frontend through a message or static config that does not leak arbitrary server filesystem paths.

• When the user selects a scene, send openStageRequest with the scene URL or registry id.

• In the server render loop, enter loading state, stop stepping the old scene, reset renderer stage state, create the new inline root/session data, load the new stage, restore persistent settings, fit or restore camera, clear selection, and resume streaming.

• Send loading progress/activity events during long loads.

• Send openStageResult after load success or failure.

• Send root hierarchy after successful load or when the frontend requests it.

Critical contracts:

• Never call renderer.step() concurrently with scene reset/load.

• Preserve render settings across scene switches unless the user explicitly asks for per-scene settings.

• Preserve camera across scenes only when that policy is requested and the old camera state is valid for the new bounds; otherwise fit to the new stage.

• Recompute hierarchy, properties cache, variants cache, bounds, pickability filters, and selection outline state for the new stage.

Decision points:

• If assets are local files, validate paths against an allowed root and reject traversal outside it.

• If assets are cloud-backed, use references/cloud-assets and keep cloud logic behind the same asset registry interface.

• If scene load fails, keep the previous scene streaming if it is still valid, or enter idle/error state with a clear frontend error.

• If variant changes require stage reload, route them through the same load lock and loading-state path as scene switching.

Common failure modes:

• Leaving render loop active during reset produces intermittent crashes or corrupted frames.

• Not clearing caches after scene switch shows stale tree children or properties.

• Persisting a camera blindly can place the viewer far away from a very different scene.

• Returning raw absolute server paths to the browser exposes local filesystem details.

Read for depth: see references/stage-management, references/stage-loading, references/stage-hierarchy, and references/cloud-assets for the full scene switching and asset contracts.

12. Add Hierarchy, Properties, And Variants

Do this in server/stage_queries.py, frontend/src/components/StageTree.tsx, and frontend/src/components/PrimInfoPanel.tsx:

• Query root children after a scene loads.

• Load tree children lazily through getChildrenRequest.

• Represent each prim with name, path, type, and child-load state.

• Fetch properties for the selected prim through getPropertiesRequest.

• Fetch variants through getVariantsRequest and apply changes through setVariantRequest.

• After variant changes, refresh affected hierarchy, properties, bounds, and selection if the variant changes composition.

Critical contracts:

• Do not perform long USD traversal in ovstream callback threads.

• Include prim_path in every response so the frontend can discard stale data.

• Keep children response semantics consistent with the frontend tree implementation.

• Do not assume all USD property values are JSON-serializable without conversion. Normalize arrays, tokens, paths, numbers, booleans, and fallback display strings.

• Avoid loading the entire stage tree by default for large scenes.

Decision points:

• Use ovrtx query_prims for basic hierarchy roots and prim discovery whenever it provides the needed data.

• If direct pxr imports are stable in the server process, direct query helpers are acceptable for USD features not covered by ovrtx native queries after ovrtx initialization.

• If imports conflict or the platform is Windows, use a subprocess query mode for those remaining pxr queries.

• If the user asks for full property editing, treat it as a separate feature and add explicit edit/apply/reload contracts.

Common failure modes:

• Large scenes freeze the stream when the full hierarchy is traversed synchronously.

• Non-serializable USD values break data-channel sends.

• Variant changes leave stale property and child rows unless caches are invalidated.

Read for depth: see references/stage-hierarchy, references/prim-info-display, and references/streaming-messages for the full hierarchy and property contracts.

13. Add Render Settings And Lighting Controls

Do this in server/render_settings.py, server/settings_store.py, and frontend/src/components/RenderSettingsPanel.tsx:

• Define a small persistent settings model for validated render settings, camera policy, segmentation/debug state, stream/profile defaults, and non-live defaults after the backend capability list is known.

• Build a server-owned supported-settings capability list from verified backend apply paths. The frontend render settings panel must render from that list, not from hard-coded optimistic controls.

• Load settings at server start.

• Apply validated immediate settings and accepted profile/default settings after every scene load.

• Send the effective settings and capabilities to the frontend after connection and after every change.

• Persist user changes after validation.

• Keep scene-independent settings separate from scene-specific transient state.

Critical contracts:

• Do not add viewer lights by default. Only create viewer-owned lights when the user requests lighting controls.

• If changing a user-selected fixed resolution, treat it as a render product and stream config change with explicit reconfiguration or restart. Do not tie stream resolution to browser CSS viewport size.

• If changing render vars, update scene setup and frame extraction together.

• Persist validated settings across scene switches.

• Validate every setting value from the client before applying it.

• Reject unsupported setting keys. Do not report success for client-side form state alone.

• Send the full effective settings after applying a change so the UI reflects clamped values, applied, applies_at, requires_reload, and any message.

Decision points:

• If the user wants a basic Omniverse Realtime Viewer, provide only verified immediate controls by default. AOV/debug view is acceptable when implemented through aov-switching; exposure/tone mapping are acceptable only when backed by frame conversion or a verified ovrtx path.

• If the user wants material or renderer internals, read references/render-settings before exposing them.

• If the setting can be applied live through ovrtx attributes, enqueue a render-thread command.

• If the setting requires stage reload, expose it as an explicit render-profile or scene-load action rather than a live control.

Common failure modes:

• UI says a setting changed while the renderer is still using the old value because the server did not echo effective settings.

• UI exposes controls that cannot apply because React hard-coded settings that the backend did not advertise as capabilities.

• Viewer-created lights change the look of authored scenes unexpectedly.

• Resolution changes break video because the frontend, ovstream config, render product, and frame pitch no longer agree.

Read for depth: see references/render-settings and references/stage-management for the full settings contract.