Omniverse Realtime Viewer

Camera Controls

Triggers

Use this skill for requests mentioning orbit camera, pan, zoom, camera controls, viewport navigation, fit to scene, camera aspect, letterbox coordinates, camera gizmos, row-major camera matrices, or cameras inside geometry.

ovrtx does not provide native camera input handling. The camera is a USD prim, and the app updates its omni:xform every frame or after input changes.

Attached OVStage Runtime

When the renderer is attached to OVStage, keep camera state behind the application runtime adapter. The controller may calculate an orbit pose locally, but the adapter performs the live omni:xform and lens writes to OVStage, waits, advances the write floor, and supplies the committed ordinal to the render loop. Do not call renderer.write_attribute(), bind_attribute(), or open a second USD stage for attached-viewer camera updates.

Direct USD and renderer examples in this reference are standalone compatibility or offline tooling only. Before implementing an attached path, read the current OVStage runtime/data-plane/integration references and current upstream OVStage and OVRTX agent guidance, skills, and examples.

Read viewer-input-routing first when the task involves WebRTC/SHM input callbacks, ovui button ids, viewport input gating, wheel events, or click-vs-drag dispatch. This skill owns camera state and camera math.

Input Mapping

This section is a camera-facing summary. viewer-input-routing is the primary source for transport normalization and input ownership.

For local ovui callbacks, button ids differ from the OrbitCamera helper:

• ovui: 0=left, 1=right, 2=middle

• OrbitCamera: 0=left, 1=middle, 2=right

• Local ovui maps exactly as 0 -> left/orbit, 2 -> middle/pan, and 1 -> right/dolly.

def camera_button_from_ovui(button: int) -> int | None:
    return {0: 0, 2: 1, 1: 2}.get(button)

For WebRTC ovstream.InputEvent callbacks, do not treat raw button integers as browser DOM button ids. ovstream.MouseButton uses NONE=0, LEFT=1, MIDDLE=2, RIGHT=3. Normalize to the shared camera helper convention before calling camera or pick code:

def camera_button_from_ovstream(raw_button) -> int | None:
    try:
        button = raw_button if isinstance(raw_button, ovstream.MouseButton) else ovstream.MouseButton(raw_button)
    except Exception:
        return None
    if button == ovstream.MouseButton.LEFT:
        return 0
    if button == ovstream.MouseButton.MIDDLE:
        return 1
    if button == ovstream.MouseButton.RIGHT:
        return 2
    return None

Use left drag for orbit, middle drag for pan, right drag for dolly/zoom, and wheel for zoom. For desktop apps with modifier keys, use Alt+LMB for orbit, Alt+MMB for pan, Alt+RMB for dolly. Optionally support RMB+WASD fly mode for free camera movement. Left-click selection should fire only on release when movement stayed below the drag threshold. If a prompt asks for switchable or tool-specific schemes, keep this as the default and layer navigation-presets/README.md over the same camera state and input normalization.

Render Aspect

For an attached viewer, send the desired render dimensions to the runtime camera/render-product command adapter. It owns the relevant OVStage mutation and publication. The direct USD helper below is for a standalone or offline path only.

When creating or explicitly reconfiguring the render product resolution, update camera viewport dimensions and projection aspect in the same operation. For USD cameras, keep horizontal aperture stable and derive vertical aperture from the render size:

def update_camera_aspect(stage, camera_path: str, width: int, height: int) -> None:
    cam = stage.GetPrimAtPath(camera_path)
    if not cam or not cam.IsValid() or width <= 0 or height <= 0:
        return
    h_attr = cam.GetAttribute("horizontalAperture")
    v_attr = cam.GetAttribute("verticalAperture")
    h_aperture = float(h_attr.Get() or 20.955)
    v_attr.Set(h_aperture * float(height) / float(width))

Browser streaming should keep a fixed server render resolution, display the video with object-fit: contain, and avoid sending resize messages for CSS layout changes. NVST handles letterbox coordinate mapping for WebRTC input carried as binary InputEvent structs; app-owned DOM math should still use the visible image rectangle before orbit, pan, zoom, or pick calculations.

Input transport rules:

• WebRTC: use the NVST native input channel and handle InputEvent structs from ovstream callbacks. Only after viewer-input-routing verifies that native callbacks are absent while the data channel works may a browser use its mutually exclusive JSON fallback.

• SHM: use ovstream.Client(ovstream.ClientType.SHM, stream_name="...").send_input_event(event) from Python, or ovstream_client_send_input_event(client, event) from C, with InputEvent structs; do not send JSON mouseInput.

• In-process: call camera controller methods directly from the Python/C++ UI event loop.

For browser-streamed React apps, gate native input with an app-level viewport ownership flag. UI panels should send setViewportInputActive {active:false}; the viewport sends active:true on pointer entry/down and active:false on pointer leave. The server should ignore native input while inactive and cancel any drag state:

def set_viewport_input_active(self, active: bool) -> None:
    self._viewport_input_active = bool(active)
    if not self._viewport_input_active:
        self.camera.cancel_interaction()

def handle_input(self, event):
    if not self._viewport_input_active:
        self.camera.cancel_interaction()
        return
    # Normal orbit, pan, zoom, and click-to-pick handling.

This prevents sidebar, tree, top-bar, and inspector interactions from reaching the orbit controller as stale WebRTC mouse input.

For WebRTC servers, initialize the viewport input gate to active when the only native input source is the stream surface. Otherwise the first mouse-down of a click can arrive before the React setViewportInputActive {active:true} data channel message, and the release will not be recognized as a click. DOM panels should still send active:false on pointer enter/down to disable camera and picking while the user interacts with UI chrome.

Whether input arrives through the native callback or the verified browser JSON fallback, route it to the same on_mouse_move, on_mouse_button_down, on_mouse_button_up, and on_scroll camera operations. The fallback server handler must not implement separate camera math or bypass click-vs-drag state.

Drag Threshold (Click vs Drag Discrimination)

A short press-and-release should be treated as a click (selection, context menu), not a drag (orbit, pan, dolly). Track movement from press to release and compare against a threshold. The default desktop threshold is a 1 px delta: any move event with abs(dx) > 1.0 or abs(dy) > 1.0 turns the gesture into a drag.

DRAG_THRESHOLD_PX = 1.0  # pixels of movement before press becomes drag

class InputState:
    def __init__(self):
        self.last_x: float = 0.0
        self.last_y: float = 0.0
        self.exceeded_threshold: bool = False

    def on_press(self, x: float, y: float):
        self.last_x = x
        self.last_y = y
        self.exceeded_threshold = False

    def on_move(self, x: float, y: float) -> bool:
        """Returns True if this motion exceeds the drag threshold."""
        dx = x - self.last_x
        dy = y - self.last_y
        self.last_x = x
        self.last_y = y
        if not self.exceeded_threshold:
            if abs(dx) > DRAG_THRESHOLD_PX or abs(dy) > DRAG_THRESHOLD_PX:
                self.exceeded_threshold = True
        return self.exceeded_threshold

    def was_click(self) -> bool:
        """Call on release — True means the gesture was a click, not a drag."""
        return not self.exceeded_threshold

Usage rules:

• LMB: if was_click() → fire selection pick at release position. If threshold exceeded → it was an orbit drag, do not select.

• RMB: if was_click() → show context menu (see local-viewer). If threshold exceeded → it was a look/dolly, suppress menu.

• MMB: always pan (no click action on middle button).

• Transform gizmo: if the press begins on or near a selected transform handle/pivot, enter transform-drag mode for the whole mouse-down and suppress orbit and click-pick on release.

• Use the same coordinate space passed to the camera helper. Local and Tauri pointer events should be mapped through the letterboxed image rect first, so the camera sees render-pixel coordinates.

• Use a 1 px threshold for precise desktop input. Increase to 8–10 only for touch-first input.

• For browser-streamed React apps, a 4–6 px threshold is often more tolerant of WebRTC/browser pointer jitter around click selection.

Gizmo Hit Testing And Input Ownership

For lightweight local viewers that combine a SceneView overlay with app-owned mouse callbacks, keep a single input owner for each mouse-down. Project the selected prim pivot into the visible rendered image rectangle and treat a press near that point as transform intent; otherwise route the press through normal camera/pick behavior.

def project_world_to_viewport(point, view, proj, image_rect, widget_origin):
    p = np.array([point[0], point[1], point[2], 1.0], dtype=np.float64)
    clip = proj @ (view @ p)
    if abs(float(clip[3])) < 1e-8:
        return None
    ndc = clip[:3] / clip[3]
    if not np.isfinite(ndc).all() or ndc[2] < -1.0 or ndc[2] > 1.0:
        return None
    off_x, off_y, draw_w, draw_h = image_rect
    x = widget_origin[0] + off_x + (ndc[0] * 0.5 + 0.5) * draw_w
    y = widget_origin[1] + off_y + (1.0 - (ndc[1] * 0.5 + 0.5)) * draw_h
    return float(x), float(y)

def pointer_is_near_selected_gizmo(screen_x, screen_y, selected_pivot):
    projected = project_world_to_viewport(selected_pivot, view, proj, image_rect, widget_origin)
    if projected is None:
        return False
    dx = screen_x - projected[0]
    dy = screen_y - projected[1]
    return dx * dx + dy * dy <= 160.0 * 160.0

This fallback does not replace a real axis-handle manipulator when the shell has one. It ensures the viewer still satisfies direct manipulation when a standalone ovui build displays the gizmo but does not deliver lower-level handle drag events into the app's transform model.

Sanitize State

NaN camera state poisons projection, picking, overlays, and ovrtx writes.

MIN_DISTANCE = 0.01
MAX_ELEVATION = math.pi / 2 - 0.01

def sanitize_camera(camera) -> None:
    if not math.isfinite(float(camera.azimuth)):
        camera.azimuth = -1.5708
    if not math.isfinite(float(camera.elevation)):
        camera.elevation = 0.0
    camera.elevation = max(-MAX_ELEVATION, min(MAX_ELEVATION, camera.elevation))
    try:
        camera.distance = max(MIN_DISTANCE, float(camera.distance))
    except Exception:
        camera.distance = MIN_DISTANCE
    if not math.isfinite(camera.distance):
        camera.distance = MIN_DISTANCE
    target = np.asarray(camera.target, dtype=np.float64)
    if target.shape != (3,) or not np.isfinite(target).all():
        target = np.array([-74.5, 103.0, -22.5], dtype=np.float64)
    camera.target = target

Call this before handling input and before generating matrices.

Stage Up-Axis Is Camera State

Read the source USD up-axis before wrapper composition through the metadata worker in stage-loading. Retain that result with its load request and apply it after the matching scene loads; the wrapper must author the same metrics on its root. Preserve the up-axis in server-owned camera state; do not rotate the user USD or compensate with mesh transforms.

def configure_camera_up_axis(camera, up_axis: str) -> None:
    axis = str(up_axis).upper()
    camera.up_axis = axis if axis in {"Y", "Z"} else "Y"
    camera.world_up = np.array(
        [0.0, 0.0, 1.0] if camera.up_axis == "Z" else [0.0, 1.0, 0.0],
        dtype=np.float64,
    )

Call this before fit-to-stage and keep the same camera.world_up for orbit basis construction, middle-drag pan axes, and optional fly movement. A stage that renders but appears sideways almost always has a hard-coded Y-up camera path rather than a rendering or source-USD problem.

Row-Major ovrtx Camera Matrix

ovrtx consumes USD GfMatrix4d row-vector layout:

M = np.eye(4, dtype=np.float64)
M[0, :3] = right       # X basis
M[1, :3] = up          # Y basis
M[2, :3] = -forward    # camera local -Z looks forward
M[3, :3] = eye         # translation

Build the basis from the active stage axis, not a hard-coded default:

forward = target - eye
forward /= np.linalg.norm(forward)
right = np.cross(forward, camera.world_up); right /= np.linalg.norm(right)
up = np.cross(right, forward)

camera.world_up is [0, 1, 0] for Y-up and [0, 0, 1] for Z-up. Use the same basis for pan: horizontal movement follows right, and vertical movement follows up. The common mistake is putting axes in columns, which puts the camera inside or under geometry.

If your camera helper returns a GL view matrix, convert it:

world_matrix = np.ascontiguousarray(np.linalg.inv(view_matrix).T, dtype=np.float64)

Commit Camera State

Attached OVStage Viewer

pose = camera.get_camera_xform()
if pose.shape == (4, 4) and np.isfinite(pose).all():
    committed = runtime.camera_commands.set_orbit_pose(
        camera_path="/Session/Cameras/Main",
        xform=pose,
        focal_length=camera.focal_length,
    )
    renderer.step(render_products, delta_time, ordinal=committed.ordinal)

set_orbit_pose is an application adapter operation: it resolves the active path through OVStage, writes at a new ordinal, waits, and advances the write floor. It is not a direct OVRTX write call.

Standalone Compatibility Path

xform = np.ascontiguousarray(camera.get_camera_xform(), dtype=np.float64)
if xform.shape == (4, 4) and np.isfinite(xform).all():
    renderer.write_attribute(
        prim_paths=["/Session/Cameras/Main"],
        attribute_name="omni:xform",
        tensor=xform.reshape(1, 4, 4),
        semantic=ovrtx.Semantic.XFORM_MAT4x4,
        prim_mode=ovrtx.PrimMode.CREATE_NEW,
    )

Use the actual inline session camera path from stage-loading.

Fit Camera To Stage

Search for authored UsdGeom.Camera prims first. If the app policy allows stage cameras, copy the selected authored camera's focal length, apertures, clipping range, projection, and transform into the viewer camera before falling back to bounds fitting.

If no authored camera exists, configure the camera from the loaded stage up_axis, then compute a world bbox via stage-hierarchy. Set target to the bbox center and distance from max dimension and focal length/field of view using the resulting camera.world_up basis. Use that same state for later orbit and pan; fitting with one axis and navigating with another makes a correctly fitted scene appear to roll or lie sideways. Choose the initial view for the kind of stage:

• For general object/prop scenes, a three-quarter orbit view is usually safe.

• For Z-up exterior or architectural scenes, avoid a steep roof-down first view. Prefer a lower elevation overview so walls, entrances, windows, racks, and scene context are visible.

• For very wide or flat scenes, increase distance and lower elevation rather than aiming straight down.

When the first view matters, render 4-6 candidate camera poses, build a small contact sheet, and choose the least occluded view. Candidate sets should vary azimuth, elevation, and distance while keeping the same bbox target.

Inline Local Camera Gizmo

For local ovui, build the gizmo directly in the viewport ZStack with omni.ui_scene.SceneView; do not use the streaming server's headless overlay compositor.

class OverlayCamera(sc.AbstractManipulatorModel):
    def get_as_floats(self, item):
        if item == self.get_item("projection"):
            f = 1.0 / math.tan(math.radians(30.0))
            return [f,0,0,0, 0,f,0,0, 0,0,-1.002,-1, 0,0,-0.2002,0]
        if item == self.get_item("view"):
            return [1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,-4,1]
        return []

class OrbitRingManipulator(sc.Manipulator):
    def __init__(self, on_orbit_delta, pixel_scale: float, **kwargs):
        super().__init__(**kwargs)
        self._on_orbit_delta = on_orbit_delta
        self._pixel_scale = float(pixel_scale)
        self._drag = sc.DragGesture(on_changed_fn=self._on_changed, on_began_fn=lambda _s: self.invalidate(), on_ended_fn=lambda _s: self.invalidate())
        self._drag.mouse_button = 0
    def on_build(self):
        for axis, color in enumerate((0xD134BCFF, 0x9EFF6BFF, 0x94C7FFFF)):
            pts = [([math.cos(i*math.tau/72), math.sin(i*math.tau/72), 0], [math.cos(i*math.tau/72), 0, math.sin(i*math.tau/72)], [0, math.cos(i*math.tau/72), math.sin(i*math.tau/72)])[axis] for i in range(73)]
            for a, b in zip(pts, pts[1:]):
                sc.Line(a, b, color=color, thickness=3.0, intersection_thickness=18.0, gesture=self._drag)
        sc.Screen(gesture=self._drag)
    def _on_changed(self, sender):
        payload = getattr(sender, "gesture_payload", None)
        if payload is not None:
            dx_ndc, dy_ndc = payload.mouse_moved
            self._on_orbit_delta(float(dx_ndc), float(-dy_ndc), self._pixel_scale)

Toggle the gizmo from a header button. DragGesture instances must be created once and reused.

Gotchas

• In an attached viewer, use the OVStage data plane rather than authored USD or renderer attribute APIs for live camera state.

• The Semantic.XFORM_MAT4x4 / PrimMode.CREATE_NEW example is standalone compatibility guidance only; verify current OVRTX API shape before using it.

• Skip writes if the 4x4 matrix is non-finite.

• Clamp local mouse coordinates through the visible rendered image rect so letterboxing does not skew orbit/pick math.

Alt+Modifier Input Mapping (Desktop Apps)

For Qt or native windowing with modifier keys:

def on_mouse_press(event):
    if event.modifiers() & Alt:
        if event.button() == LeftButton:
            mode = "orbit"
        elif event.button() == MiddleButton:
            mode = "pan"
        elif event.button() == RightButton:
            mode = "dolly"
    elif event.button() == RightButton:
        mode = "fly_look"  # enter RMB+WASD fly mode
    elif event.button() == LeftButton:
        mode = "select"  # click-to-select (fire on release if no drag)

WASD Fly Mode

When the right mouse button is held, enable keyboard-driven fly movement:

class FlyState:
    def __init__(self):
        self.keys_held: set[str] = set()
        self.speed = 2.0  # units/second, adjustable via scroll wheel while RMB held

    def update(self, camera, dt: float):
        if not self.keys_held:
            return
        forward = camera.forward_vector()
        right = camera.right_vector()
        up = camera.world_up  # [0,0,1] for Z-up, [0,1,0] for Y-up
        move = np.zeros(3, dtype=np.float64)
        if "w" in self.keys_held: move += forward
        if "s" in self.keys_held: move -= forward
        if "d" in self.keys_held: move += right
        if "a" in self.keys_held: move -= right
        if "e" in self.keys_held: move += up
        if "q" in self.keys_held: move -= up
        norm = np.linalg.norm(move)
        if norm > 1e-6:
            move = move / norm * self.speed * dt
        camera.target += move
        # eye moves with target (no orbit change)

While in fly mode, mouse movement rotates the camera view (adjust azimuth/elevation without changing distance). Scroll wheel adjusts fly speed.

Generated Module Checklist - camera.py

• [ ] OrbitCamera.__init__(width: int, height: int)

• [ ] OrbitCamera.on_mouse_button_down(x: float, y: float, button: int) -> None

• [ ] OrbitCamera.on_mouse_button_up(x: float, y: float, button: int) -> bool

• [ ] OrbitCamera.on_mouse_move(x: float, y: float) -> None

• [ ] OrbitCamera.orbit_delta(dx: float, dy: float, scale: float = 1.0) -> None

• [ ] OrbitCamera.on_scroll(delta: float) -> None

• [ ] OrbitCamera.get_camera_xform() -> np.ndarray

• [ ] OrbitCamera.get_view_matrix() -> np.ndarray

• [ ] OrbitCamera.get_projection_matrix(aspect_ratio=None) -> np.ndarray

• [ ] OrbitCamera._sanitize_state() -> None

• [ ] Press/release state distinguishes click from drag using the 1 px threshold.

• [ ] Matrix rows are right, up, negative-forward, translation.

Generated Module Checklist - server input routing

• [ ] MessageHandler.on_input(event) -> None

• [ ] Mouse move calls camera.on_mouse_move(x, y).

• [ ] Left-button release calls camera.on_mouse_button_up(..., 0) and picks only when it returns True.

• [ ] Middle-button input maps to camera button 1.

• [ ] Right-button input maps to camera button 2.

• [ ] Wheel input calls camera.on_scroll(delta).

• [ ] Browser pointer events are not duplicated as JSON messages.

See also: viewer-input-routing, local-viewer, stage-loading, stage-hierarchy, prim-info-display, viewport-overlays.