Omniverse Realtime Viewer

ovui Local Interaction Features

7. Add Camera Controls

Read references/viewer-input-routing first for ovui button normalization, click-vs-drag dispatch, and input/render-loop ownership. Then implement camera math in local_app/input_controller.py and local_app/camera_controller.py:

• Track mouse press, movement, release, buttons, wheel, keyboard modifiers, viewport dimensions, and drag threshold state from ovui callbacks.

• Convert ovui screen coordinates into render-image coordinates, accounting for widget position, preserve-aspect scaling, and letterboxing.

• Use drag thresholding so a short left press/release selects and a left drag orbits.

• Support orbit, pan, dolly/zoom, wheel zoom, fit-to-stage, and optional WASD fly mode when requested.

• If selected-prim transform is enabled, test transform-gizmo presses before normal left-drag orbit and before click selection.

• Sanitize camera state before input handling and before writing matrices.

• Enqueue the viewer camera's local omni:xform through the OVStage runtime adapter, then publish it before rendering the committed ordinal. Use direct renderer writes only for an explicitly standalone compatibility path.

Critical contracts:

• The camera is a USD prim. Camera movement writes omni:xform on the viewer camera path.

• Use the row-vector matrix layout expected by USD/ovrtx: basis vectors in rows and translation in the final row.

• Clamp elevation away from straight up/down singularities.

• Clamp camera distance above a small positive minimum.

• Skip camera writes when any matrix value is non-finite.

• Use the same world-up convention as the scene when fitting and orbiting.

• Left-click selection fires only on release if movement stayed below the drag threshold.

• A transform-gizmo mouse-down owns the full gesture until release; it suppresses camera orbit and click-pick for that mouse-down.

• Map ovui button IDs to the camera helper's expected button IDs before interpreting gestures.

Decision points:

• If the user wants simple navigation, use left drag orbit, middle drag pan, right drag dolly, and wheel zoom.

• If the user wants DCC-style navigation, use Alt+left for orbit, Alt+middle for pan, and Alt+right for dolly while preserving left-click selection.

• If the stage has an authored camera and the app policy is to use it, initialize viewer camera settings from that camera before allowing interaction.

• If the user requests camera UI buttons, wire them to local runtime commands rather than duplicating pointer gesture logic in widgets.

• If a minimal ovui gizmo renders handles but does not expose handle-drag callbacks, project the selected pivot into viewport space and implement a direct-manipulation fallback that converts mouse deltas into camera-plane world deltas.

Common failure modes:

• Putting matrix basis vectors in columns instead of rows places the camera inside, behind, or under the scene.

• Ignoring letterboxing makes orbit centers and picks offset from the visible image.

• Treating every left release as selection causes accidental selections after orbit drags.

• Letting the camera controller see a transform-gizmo press first makes the gizmo appear inert because orbit owns the drag.

• Forgetting ovui's button order makes right-drag and middle-drag modes swap.

Read for depth: see references/viewer-input-routing, references/camera-controls, references/local-viewer, and references/stage-hierarchy for input routing, camera math, and bounds contracts.

8. Add Picking, Selection, And Highlighting

Read references/native-picking-selection, references/object-selection, and references/selection-feedback before implementing this section. Those references own the current native OVRTX picking and selection shapes; this recipe should route to those contracts instead of generating compatibility branches or fallback picker modules.

Do this in local_app/selection_controller.py; the completed click or marquee gesture should come from references/viewer-input-routing:

• On left mouse release after a click gesture, map the screen coordinate to a render pixel and enqueue a native ovrtx pick query.

• After the next render step, read the synthetic ovrtx_pick_hit render var, validate its params, resolve primPath ids to USD prim paths, and deduplicate the result.

• Maintain selected tree path and selected mesh paths separately when an Xform or Scope expands to descendant geometry.

• Keep selection state in runtime memory and update viewport highlight, tree row selection, and prim info from that single source.

• Clear selection on scene reset, scene switch, empty tree selection, or explicit clear command.

• Apply visual selection feedback with native renderer selection-group helpers, not by writing legacy attributes or permanently editing the user USD.

Use the current shapes from native-picking-selection; the essential contracts are a normalized synchronous enqueue, renderer-owned selection groups, and an automatic pick-hit output:

renderer.enqueue_pick_query(
    render_product_path=render_product_path,
    left_ndc=left / render_width,
    top_ndc=top / render_height,
    right_ndc=right / render_width,
    bottom_ndc=bottom / render_height,
)
path = renderer.resolve_prim_path_id(int(prim_path_id))
renderer.set_selection_group_styles({
    1: ovrtx.SelectionGroupStyle(
        outline_color=(1.0, 0.6, 0.0, 1.0),
        fill_color=(0.0, 0.0, 0.0, 0.0),
    )
})
renderer.set_selection_outline_group_strings(
    paths,
    groups_uint8.tolist(),
)

Decode the post-step synthetic pick render var with ovrtx.OVRTX_RENDER_VAR_PICK_HIT, validate magic, version, and hitCount, read the named primPath tensor, and resolve IDs before mutating selection. Do not scaffold GpuPicker, CPU ray/AABB fallback, segmentation ID maps, or Warp outline systems for new generated apps.

Critical contracts:

• The pick query rectangle uses render-product pixel space after letterbox/scaling correction.

• Convert those pixels to normalized top-left bounds for enqueue_pick_query(); do not add ovrtx_pick_hit to orderedVars because the query produces it.

• Picking coordinates must use render-pixel space after letterbox and scaling correction.

• Selection state is local-runtime authoritative. Widgets mirror it; they do not independently mutate it.

• Enable selection outlines at renderer creation and set per-group outline/fill colors with Renderer.set_selection_group_styles(...).

• Clear previous outlines with group 0; assign selected prims to a non-zero group such as 1 through set_selection_outline_group_strings().

• Do not let selection picking run while the scene is loading or the renderer is resetting.

• Check operation status for the stage load and render step before changing selection. enqueue_pick_query() registers synchronously; use the async form only when its returned operation is explicitly awaited.

Decision points:

• If the user only asks for tree selection, implement tree-driven selection first and defer viewport picking.

• If the user asks for visual highlight, use native selection outlines after selection state works.

• If the user asks for hover, multi-select, or marquee selection, extend the local state model explicitly and keep final selected state centralized.

• If selection needs property display, trigger the info panel to query the selected prim rather than stuffing full properties into every selection update.

Common failure modes:

• Selection appears offset when coordinate transforms ignore image scaling or letterboxing.

• No pick result arrives when the query was enqueued for a different RenderProduct than the next renderer.step().

• Highlight persists across scene loads when old selection outline groups are not cleared.

• Tree selection and viewport selection diverge when each widget tracks its own selected path.

Read for depth: see references/viewer-input-routing, references/native-picking-selection, references/object-selection, references/selection-feedback, references/prim-info-display, and references/stage-hierarchy for the full input, picking, highlighting, and info contracts.

9. Add Scene Switching And Asset Browsing

Put the scene registry and load state in local_app/scene_manager.py; the scene picker displays friendly labels and submits stable paths or asset IDs. Use references/stage-management for one preparation-and-handoff flow, and references/cloud-assets for remote preparation.

Keep the current scene interactive while resolving/downloading the next asset. Only the latest requested replacement may reach the runtime handoff. At that point, retire old scene queries/picks, populate a fresh Stage, and restore the chosen settings/camera policy before reporting its first valid frame. A failed handoff keeps the last image visible with an error state and a retry action.

Use the same replacement path for Reload. If the product needs to retain the old live scene until the new scene succeeds, budget for both generations rather than assuming reset provides rollback. Additive edits use the separate composition path in references/ovstage-population.

10. Add Stage Hierarchy, Properties, And Variants

Do this in local_app/stage_queries.py, local_app/widgets/stage_tree.py, and local_app/widgets/prim_info_panel.py:

• Query the current stage through native runtime queries first, then through the chosen pxr subprocess mode for OpenUSD features that still require it.

• Query root children after a scene loads.

• Load tree children lazily when a row expands.

• Represent each prim with name, path, type, and child-load state.

• Fetch properties for the selected prim when selection changes.

• Fetch variants for the selected prim and apply variant changes through the scene manager.

• After variant changes, refresh affected hierarchy, properties, bounds, native pickability state, and selection if composition changed.

Critical contracts:

• Avoid full recursive hierarchy traversal by default for large scenes.

• Keep children semantics stable: expandable-but-not-loaded is distinct from loaded children and leaf rows.

• Do not assume all USD property values are directly displayable. Normalize arrays, tokens, paths, numbers, booleans, and fallback display strings.

• Include the prim path with every local query result so widgets can ignore stale data after selection or scene changes.

• If query work can block the UI, move it to a worker and return results to the UI/render loop safely.

• Tree selection should call the same runtime selection path as viewport picking.

Decision points:

• Use direct pxr imports only after verifying the exact local process import path for the selected OVRTX/OVStage/USD wheels.

• Otherwise, use a subprocess query mode and keep logs on stderr while requests and responses use structured data.

• If the user asks for variant editing, treat it as a scene mutation and route it through scene management.

• If the user asks for full property editing, treat it as a separate feature and add explicit edit/apply/reload contracts.

Common failure modes:

• Large scenes freeze the UI when the full hierarchy is traversed synchronously.

• Non-serializable or non-displayable USD values break property rendering.

• Variant changes leave stale property and child rows unless caches are invalidated.

• Direct pxr imports can conflict with ovrtx if import order or library paths are inconsistent.

Read for depth: see references/stage-hierarchy, references/prim-info-display, references/stage-management, and references/windows-native-setup for the full hierarchy and property contracts.

11. Add Render Settings And Lighting Controls

Do this in local_app/render_settings.py, local_app/settings_store.py, and local_app/widgets/render_settings_panel.py:

• Define a small persistent settings model for validated render settings, camera policy, segmentation/debug state, viewport/profile defaults, and non-live defaults after the runtime capability list is known.

• Build a runtime-owned supported-settings capability list from verified backend apply paths. The render settings panel must render from that list, not from hard-coded optimistic controls.

• Load settings at app start.

• Apply validated immediate settings and accepted profile/default settings after every scene load.

• Persist user changes after validation.

• Keep scene-independent settings separate from scene-specific transient state.

• Echo the effective settings and capabilities back into UI state after every change so controls reflect clamped values, applied, applies_at, requires_reload, and any message.

Critical contracts:

• Do not add viewer lights by default. Only create viewer-owned lights when the user requests lighting controls.

• If changing resolution, treat it as a render product, image bridge, pick coordinate, and viewport math reconfiguration.

• If changing render vars, update scene setup and frame extraction together.

• Persist validated settings across scene switches.

• Validate every setting value from the UI before applying it.

• Reject unsupported setting keys. Do not report success for client-side form state alone.

• Do not invent write_attribute names for renderer internals; expose only controls supported by the active ovrtx build and documented by references/render-settings.

Decision points:

• If the user wants a basic Omniverse Realtime Viewer, provide only verified immediate controls by default. AOV/debug view is acceptable when implemented through aov-switching; lighting controls are acceptable only when backed by a verified live apply path or explicit reload/profile workflow.

• If the user wants material or renderer internals, read references/render-settings before exposing them.

• If the setting can be applied live through ovrtx attributes, apply it from the render loop.

• If the setting requires stage reload, expose it as an explicit render-profile or scene-load action rather than a live control.

• If the user wants pick-driven effects, runtime transforms, or physics impulses, expose them as separate optional capabilities and route accepted changes through the runtime owner. OVPhysX USD population should run in a worker unless PhysX.attach_ovstage(stage, read_ordinal=...) is verified for the exact installed OVStage/OVPhysX ABI.

Common failure modes:

• UI says a setting changed while the renderer is still using the old value because effective settings were not echoed back.

• UI exposes controls that cannot apply because the panel did not render from backend-advertised capabilities.

• Viewer-created lights change the look of authored scenes unexpectedly.

• Resolution changes break picking because render product, image bridge, pick coordinate mapping, and letterbox math no longer agree.

• Extra render vars increase memory and frame time even when no feature uses them.

Read for depth: see references/render-settings, references/stage-management, and references/stage-loading for the full settings contract.

12. Manage Runtime State And Shutdown

Do this in local_app/runtime.py:

• Maintain explicit runtime states: starting, idle/no scene, loading, rendering, error, and shutting down.

• Drain queued commands from the UI/render loop before stepping the next frame.

• Keep the latest loaded stage path, hierarchy root, selection, render settings, loading state, and viewer error in runtime memory.

• Handle app shutdown by stopping the ovui loop, preventing new scene commands, releasing renderer-owned resources if the API exposes them, and closing any query subprocesses.

• Guard long-running operations so the UI can display loading or error state instead of silently freezing.

Critical contracts:

• UI callbacks must not call renderer load/reset/step/write APIs directly if they can run during another renderer operation.

• Scene load and reset operations must be mutually exclusive with frame stepping.

• Errors should clear active drag state, loading state, and stale selection when needed.

• Query subprocesses must be terminated on shutdown and restarted only through a controlled path.

• The app should exit cleanly on interrupt or window close without leaving stale GPU processes.

Decision points:

• If all work stays on the UI thread, keep command handling simple and deterministic.

• If loading large scenes needs a worker thread, make the render thread the only component that mutates the ovrtx renderer.

• If hierarchy queries are slow, cache per-prim results and invalidate them on scene switch or variant change.

• If the app needs autosave settings, write only app settings JSON, never user USD files.

Common failure modes:

• Calling renderer methods from both callbacks and the render loop creates rare crashes that are hard to reproduce.

• Exceptions during input can leave a drag gesture active and cause later clicks to orbit or select unexpectedly.

• Query workers that write logs to stdout can corrupt structured responses.

• App exits that skip GPU cleanup may leave stale Python processes holding CUDA/RTX state.

Read for depth: see references/local-viewer, references/ovrtx-rendering, references/stage-management, and references/stage-hierarchy for runtime and shutdown contracts.