Omniverse Realtime Viewer

Viewer Input Routing

Triggers

Use this skill when implementing or debugging viewport controls, input callbacks, WebRTC on_input, SHM/native input, ovui mouse handlers, camera drag, wheel zoom, click picking, drag selection, DOM panel input gating, or wrong/no selection after a click.

This skill owns transport input normalization and dispatch. Camera math stays in camera-controls; native pick query decode stays in native-picking-selection and object-selection; React UI state and DOM layout stay in streaming-client.

First Rules

• Use native transport input for pointer/key/wheel traffic. WebRTC apps receive ovstream.InputEvent structs through server.on_input; SHM clients send the same native structs.

• A browser WebRTC app may use the narrowly scoped JSON mouseInput fallback only after an explicit diagnostic proves that its data channel works while a viewport interaction produces no server-side native on_input callback. Treat native and fallback input as mutually exclusive modes; never send both for the same gesture. SHM and in-process clients do not use this fallback.

• Normalize every transport's button ids before calling shared camera or selection helpers.

• Treat left click as selection only on button release and only when movement stayed under the drag threshold.

• Keep input callbacks fast. Enqueue camera writes, pick queries, and selection changes for the renderer owner thread; do not call renderer.step(), scene load/reset, or live attribute writes directly from input callbacks.

• During scene load/reset, cancel active drags and ignore or defer input. Clear pending picks when the stage changes.

Button Conventions

Use one app-internal camera helper convention:

Normalize transport ids into that convention:

def camera_button_from_ovui(button: int) -> int | None:
    # ovui: 0=left, 1=right, 2=middle
    return {0: 0, 2: 1, 1: 2}.get(button)

def camera_button_from_ovstream(raw_button, ovstream) -> int | None:
    # ovstream.MouseButton: NONE=0, LEFT=1, MIDDLE=2, RIGHT=3
    try:
        button = raw_button if isinstance(raw_button, ovstream.MouseButton) else ovstream.MouseButton(raw_button)
    except Exception:
        return None
    if button == ovstream.MouseButton.LEFT:
        return 0
    if button == ovstream.MouseButton.MIDDLE:
        return 1
    if button == ovstream.MouseButton.RIGHT:
        return 2
    return None

Do not compare mouse.data to DOM button ids. For wheel events, prefer mouse.scroll_y when the binding exposes it and fall back to mouse.data only for older builds.

def wheel_delta(mouse) -> float:
    delta = getattr(mouse, "scroll_y", 0) or getattr(mouse, "data", 0)
    return float(delta)

WebRTC Viewport Ownership

Browser DOM controls can sit beside or above the stream, but raw native input still reaches the server. Maintain an app-level viewport ownership flag:

• Server default should be viewport_input_active = True when the only native input source is the stream surface.

• DOM panels, trees, inspectors, menus, and toolbars send setViewportInputActive {active:false} on pointer enter/down/wheel.

• The viewport surface sends active:true on pointer enter/down and active:false on pointer leave.

• When the server receives active:false, cancel the current camera gesture and suppress picking until re-enabled.

Starting inactive creates a first-click race: the mouse-down can arrive through native input before the React activation message arrives through the data channel, so the release has no matching press and selection never queues.

Browser 2D gizmo ownership

A browser UI handle over the stream may emit semantic parameter/transform commands over the app channel while camera and picking retain native input. This is a separate interaction target, not the viewport JSON fallback. Give the handle exclusive ownership from press through release/cancel: prevent the same gesture from reaching the SDK's native mouse/wheel listeners as well as React handlers. A late asynchronous viewport-gating message alone cannot prevent a press race; verify suppression at the SDK's actual event attachment points.

Use the browser's verified capture or window-listener lifecycle to finish outside the handle. Cancel on lost capture, blur, disconnect, teardown, or scene change; restore the drag baseline according to the tool's policy. Tag queued intents with scene generation and drag identity, coalesce superseded moves, and reject late updates after cancellation. Release suppression so ordinary camera/picking input resumes. Follow the desktop-specific restrictions below for WebViews.

Desktop WebView Pointer Lifecycle

For Tauri or another desktop WebView, make the viewport component own one pointer gesture from press through cleanup. On pointerdown, reject non-viewport controls, record the pointer id, button, and render-pixel press position, and enqueue one button-down intent. Route window-level pointermove only for that active pointer. Finish from window-level pointerup or pointercancel: enqueue cancel or button-up once, and clear the active pointer, press position, and drag state. Use this window-listener path without setPointerCapture in Tauri on Linux/WebKit2GTK; see tauri-shm-transform-gizmo for overlay input ownership.

Window-level listeners are required because a drag may leave the viewport and some WebView button paths do not deliver the final event to the original element. Do not attach duplicate window listeners on every render; register them for the active gesture and remove them during completion and component teardown. A new stage load, window blur, or component teardown must follow the same cancel path.

Browser JSON Fallback For Verified Native-Input Failure

Keep input_mode = "native" as the default. Switch to input_mode = "json-fallback" only after all of these are true:

1. The browser has a live stream and the app data channel can complete a small request/response round trip.

2. The user performs a pointer, button, and wheel test over the active viewport.

3. Server logging confirms that on_input received none of those native events.

Record the evidence and selected mode in the validation report. Do not infer fallback mode merely because a camera did not move: check viewport ownership, the camera command queue, and the native callback log first. In fallback mode, map DOM coordinates through the visible video rectangle to RenderProduct pixels, stop propagation to the streaming surface, and send mouseInput over the app data channel. The server ignores native input while this mode is active and passes fallback events through the same normalized camera, drag-threshold, and pick dispatch as native events. Switching modes cancels any active drag.

Input Router Skeleton

DRAG_THRESHOLD_PX = 4.0


class InputRouter:
    def __init__(self, commands, render_width: int, render_height: int):
        self.commands = commands
        self.render_width = render_width
        self.render_height = render_height
        self.viewport_input_active = True
        self.input_mode = "native"
        self._active_button: int | None = None
        self._press_pos: tuple[float, float] | None = None
        self._dragged = False

    def set_viewport_input_active(self, active: bool) -> None:
        self.viewport_input_active = bool(active)
        if not self.viewport_input_active:
            self.commands.enqueue_cancel_interaction()
            self._active_button = None
            self._press_pos = None
            self._dragged = False

    def on_input(self, event, ovstream) -> None:
        if self.input_mode != "native":
            return
        if not self.viewport_input_active:
            self.commands.enqueue_cancel_interaction()
            return
        if event.type != ovstream.InputEventType.MOUSE:
            return

        mouse = event.mouse
        if mouse.type == ovstream.MouseEventType.MOVE:
            x, y = float(mouse.x), float(mouse.y)
            if self._press_pos is not None:
                dx = x - self._press_pos[0]
                dy = y - self._press_pos[1]
                if abs(dx) > DRAG_THRESHOLD_PX or abs(dy) > DRAG_THRESHOLD_PX:
                    self._dragged = True
            self.commands.enqueue_camera_move(x, y, mouse.modifiers)
            return

        if mouse.type == ovstream.MouseEventType.WHEEL:
            self.commands.enqueue_camera_scroll(wheel_delta(mouse), float(mouse.x), float(mouse.y))
            return

        if mouse.type != ovstream.MouseEventType.BUTTON:
            return

        button = camera_button_from_ovstream(mouse.data, ovstream)
        if button is None:
            return

        is_down = mouse.button_state == ovstream.KeyState.DOWN
        if is_down:
            self._active_button = button
            self._press_pos = (float(mouse.x), float(mouse.y))
            self._dragged = False
            self.commands.enqueue_camera_button_down(float(mouse.x), float(mouse.y), button)
            return

        was_click = button == self._active_button and not self._dragged
        self.commands.enqueue_camera_button_up(float(mouse.x), float(mouse.y), button)
        if button == 0 and was_click:
            self.commands.enqueue_pick(float(mouse.x), float(mouse.y))
        self._active_button = None
        self._press_pos = None
        self._dragged = False

commands must execute on the renderer owner thread or be drained by that thread before renderer.step().

Coordinate Ownership

For WebRTC native input, NVST maps stream-surface coordinates for the fixed stream resolution. For the JSON fallback and other app-owned DOM math, measure the visible video rectangle, reject letterboxed areas, and convert to RenderProduct pixels before camera or pick dispatch.

Keep the RenderProduct size fixed for the session. Do not resize the server renderer because the browser CSS viewport changed.

Scene And Render Loop Coordination

• Render loop owns native pick enqueue/result decode, selection outline writes, and camera omni:xform writes.

• Input callbacks enqueue intent: camera_move, camera_button, scroll, pick_at, cancel_interaction, or set_viewport_input_active.

• Scene load/reset holds the same renderer mutation lock used by the render loop. While loading, discard pending picks and cancel drags.

• Log pick state transitions during validation: Queueing viewport pick when a click queues a pick, and Selection changed after decoded pick paths update selection.

For ovui, adapt (delta_x, delta_y, modifiers) separately from ovstream MouseEvent. Forward the vertical delta to zoom and preserve modifier flags; see local-viewer.

Validation Checklist

• [ ] First left click after page load queues a pick and selects a prim.

• [ ] Left drag orbits without selecting on release.

• [ ] Middle drag pans, right drag dollies or fly-looks, and wheel zooms.

• [ ] Sidebar/tree/inspector clicks and wheels do not move the camera or pick.

• [ ] Selection stays synchronized between viewport, tree, and property panel.

• [ ] Scene switch cancels active drag and clears stale pending picks.

• [ ] Server logs show queued pick and selection-changed events for click tests.

• [ ] A drag leaving the desktop viewport ends exactly once on pointer-up, pointer-cancel, window blur, or scene switch.

• [ ] Native input is the only active camera/picking path in the normal browser deployment; browser UI handles consume their own gestures.

• [ ] If the JSON fallback is enabled, its data-channel/native-callback failure evidence is recorded, fallback coordinates match the render product, and native events are ignored so no gesture is applied twice.

See also: streaming-server, streaming-client, streaming-messages, camera-controls, native-picking-selection, object-selection, selection-feedback, local-viewer, and webgl-shm-transport.