Omniverse Realtime Viewer

Transform Manipulator

Use this when an Omniverse Realtime Viewer needs interactive translate, rotate, or scale controls for a selected USD prim. Also read prim-transform-safety when live omni:xform attributes or selection animation are involved, and viewport-overlays for the WebRTC headless ovui overlay compositor.

For ovui-owned gizmo, widget, or headless overlay behavior beyond this skill, read references/dependencies for acquisition guidance and supplemental dependency documentation.

Implementation Contract

Implement these pieces in the generated app as needed:

• Local ovui frame gizmo: draw translate, rotate, and scale handles directly into the numpy RGBA frame before ByteImageProvider receives it.

• Viewport input router: convert letterboxed viewport coordinates into render coordinates and give the gizmo pointer priority before camera controls.

• App tool state: own active tool, keyboard shortcuts, toolbar state, selected path synchronization, animation pause/resume, and pending transform updates.

• Runtime bridge: write live omni:xform values for immediate ovrtx feedback, then commit final persistent edits through the selected session-layer path.

• WebRTC headless overlay: draw the same manipulator grammar through ovui overlay primitives, then alpha-composite the RGBA overlay over the streamed BGRA frame.

• Message/input handler: route pointer and keyboard input to the gizmo first, then to camera or selection handlers only when the gizmo does not consume it.

• Shared projection helper: expose one world_to_screen() convention for headless overlays, hit testing, labels, and tooltips.

• CUDA/Warp compositor: blend overlay RGBA over the ovrtx BGRA stream buffer without copying the full video frame through browser-side 3D rendering.

Architecture

There are two supported manipulator paths. Both share the same state and math: selected path, active tool, projected axis endpoints, hit testing, drag state, and USD transform authoring.

WebRTC path:
  ovrtx renderer.step() -> LdrColor CUDA frame
  -> headless ovui TransformGizmoOverlay draws SceneView primitives
  -> copy ovui RGBA output to CUDA
  -> CUDA alpha blend over BGRA stream buffer
  -> ovstream WebRTC video frame

Local ovui path:
  ovrtx renderer.step() -> CPU numpy RGBA frame
  -> TransformGizmo.draw(frame) draws anti-aliased lines/polygons
  -> ImageBridge.update(frame) -> ByteImageProvider

For the WebRTC browser path, do not implement browser-side WebGL manipulators. Browser delivery displays the ovstream video frame; USD rendering and gizmo composition remain server-side. For the local Tauri SHM WebView path, use tauri-shm-transform-gizmo: it renders a 2D canvas overlay over already-rendered SHM pixels, not a browser 3D scene.

Input Priority

The gizmo must receive mouse input before the camera controller. If the camera sees the press first, orbit steals the drag and the manipulator feels broken.

def on_mouse_pressed(x, y, button, modifier):
    local = viewport_to_render_coords(x, y, clamp=False)
    if local and transform_gizmo.mouse_pressed(local[0], local[1], button):
        camera_dragging = False
        return

    mapped = camera_button_from_ovui(button)
    if mapped is not None:
        lx, ly = viewport_to_render_coords(x, y, clamp=True)
        camera.on_mouse_button_down(lx, ly, mapped)

During drag, keep routing all move and release events to the gizmo, even when the pointer leaves the rendered image. Clamp coordinates while dragging so release can finish cleanly.

def on_mouse_moved(x, y):
    dragging = bool(getattr(transform_gizmo, "dragging", False))
    local = viewport_to_render_coords(x, y, clamp=dragging)
    if local and transform_gizmo.mouse_moved(local[0], local[1]):
        return
    if camera_dragging:
        camera.on_mouse_move(*viewport_to_render_coords(x, y, clamp=True))

For WebRTC/headless overlays, the server input bridge follows the same rule: transform_gizmo.handle_input(event) runs before camera input dispatch, and returns True when the gizmo consumed the event.

In lightweight local ovui shells, do not assume that drawing a SceneView gizmo also wires low-level handle drag events into the application. If the selected handle/pivot can be hit but no transform callback fires, add an app-owned fallback: project the selected prim pivot into the visible image rect, treat an LMB press within a generous radius as transform intent, and route the whole mouse-down to the transform model. The release must not also enqueue a pick.

Hit Testing

Project the gizmo origin and each world-axis tip to screen space every frame:

AXES = {
    "x": np.array([1.0, 0.0, 0.0]),
    "y": np.array([0.0, 1.0, 0.0]),
    "z": np.array([0.0, 0.0, 1.0]),
}
HIT_RADIUS_PX = 35.0
AXIS_LENGTH_PX = 88.0
RING_RADII_PX = {"x": 76.0, "y": 64.0, "z": 52.0}

For translate and scale, choose the closest axis line segment from center to tip.

def distance_to_segment(point, start, end):
    segment = end - start
    denom = float(np.dot(segment, segment))
    if denom <= 1e-9:
        return float(np.linalg.norm(point - start))
    t = np.clip(np.dot(point - start, segment) / denom, 0.0, 1.0)
    return float(np.linalg.norm(point - (start + segment * t)))

hit = min(axis_cache, key=lambda item: distance_to_segment(point, origin, item.end))
axis = hit.name if hit.distance <= HIT_RADIUS_PX else None

For rotate, compare the click radius to the ring radii and choose the nearest ring.

distance = float(np.linalg.norm(point - screen_origin))
delta, axis = min(
    (abs(distance - radius), axis_name)
    for axis_name, radius in RING_RADII_PX.items()
)
return axis if delta <= HIT_RADIUS_PX else None

Use 35px, not a tighter 18px; WebRTC latency and pointer jitter otherwise make the axes hard to grab. Treat degenerate projected segments as center hits or fall back to a stable screen direction when an axis points into the camera.

Projection And Drag Math

Project the axis direction, not only the axis endpoint. Try larger world-space scales until the projected direction has a usable length.

FALLBACK_AXIS_SCREEN = {
    "x": np.array([1.0, 0.0]),
    "y": np.array([0.0, -1.0]),
    "z": np.array([-0.72, 0.52]),
}

def project_axis_direction(origin, world_axis, axis_name):
    for scale in (1.0, 10.0, 100.0, 1000.0):
        projected = world_to_top_left_screen(origin + world_axis * scale)
        if projected is None:
            continue
        delta = projected - screen_origin
        length = float(np.linalg.norm(delta))
        if length > 0.5 and math.isfinite(length):
            return delta / length, length / scale

    fallback = FALLBACK_AXIS_SCREEN[axis_name]
    return fallback / np.linalg.norm(fallback), 1.0

Drag math is screen-space first. Store the starting transform and compute each update from that base; do not accumulate small increments frame-to-frame.

delta_px = float(np.dot(mouse - drag.start_mouse, drag.screen_direction))

if drag.tool == "translate":
    amount = delta_px / max(drag.pixels_per_world_unit, 1e-6)
    next_position = drag.start_position + AXES[drag.axis] * amount
    next_transform = drag.start_transform.copy()
    next_transform[3, :3] = next_position
elif drag.tool == "rotate":
    next_transform = rotate_transform(
        drag.start_transform,
        AXES[drag.axis],
        delta_px * 0.01,  # radians
    )
elif drag.tool == "scale":
    factor = max(0.05, 1.0 + delta_px / 140.0)
    next_transform = scale_transform(drag.start_transform, drag.axis, factor)

The viewer matrix convention is row-major with translation in row 3. Keep world_to_screen() coordinate conventions consistent: if the projection helper returns bottom-left origin, convert once to top-left origin before hit testing.

USD Transform Authoring

For immediate ovrtx feedback, every manipulator drag path must update the live runtime transform. Use renderer.write_attribute(..., "omni:xform", ...) with Semantic.XFORM_MAT4x4, PrimMode.CREATE_NEW, and DataAccess.SYNC, and snapshot the selected prim's current world transform at drag start. A visible manipulator without a live transform write is not complete.

Session-layer xformOp authoring is useful for persistent/editor-style edits, but it should not be the only path for a realtime viewer drag. If both are required, write live omni:xform during the drag and commit the final edit through the chosen session-layer/undo path on release.

Author manipulator edits into the session layer so user USD files remain non-destructive. Convert the desired world matrix into parent-local space before writing xform ops.

from pxr import Gf, Usd, UsdGeom

def apply_world_transform_to_prim(stage, path, world_transform):
    prim = stage.GetPrimAtPath(path)
    if not prim or not prim.IsValid() or not prim.IsA(UsdGeom.Xformable):
        return False

    desired_world = gf_matrix_from_numpy(world_transform)
    parent_world = Gf.Matrix4d(1.0)
    parent = prim.GetParent()
    if parent and parent.IsValid() and str(parent.GetPath()) != "/":
        parent_world = UsdGeom.XformCache(
            Usd.TimeCode.Default()
        ).GetLocalToWorldTransform(parent)
    local = desired_world * parent_world.GetInverse()

    translate, rotate_xyz, scale = decompose_common_matrix(local)
    with Usd.EditContext(stage, stage.GetSessionLayer()):
        return write_xform_ops(UsdGeom.Xformable(prim), translate, rotate_xyz, scale)

Write xform ops directly. Do not use UsdGeom.XformCommonAPI.SetRotate() for mixed assets because existing prims may have Gf.Vec3d rotate/scale attributes while SetRotate() expects Gf.Vec3f.

def set_vec3_op(op, values):
    attr = op.GetAttr()
    type_name = attr.GetTypeName().cppTypeName if attr else ""
    if "Vec3d" in type_name or "double" in type_name:
        op.Set(Gf.Vec3d(*values), Usd.TimeCode.Default())
    else:
        op.Set(Gf.Vec3f(*[float(v) for v in values]), Usd.TimeCode.Default())

def write_xform_ops(xformable, translate, rotate_xyz, scale):
    ops = {op.GetOpName(): op for op in xformable.GetOrderedXformOps()}
    t_op = ops.get("xformOp:translate") or xformable.AddTranslateOp()
    r_op = ops.get("xformOp:rotateXYZ") or xformable.AddRotateXYZOp()
    s_op = ops.get("xformOp:scale") or xformable.AddScaleOp()

    t_op.Set(Gf.Vec3d(*translate), Usd.TimeCode.Default())
    set_vec3_op(r_op, rotate_xyz)
    set_vec3_op(s_op, scale)
    return True

For live ovrtx state, coordinate this with prim-transform-safety: query real world transforms before binding omni:xform, initialize the binding before the next render step, and recreate bindings after scene reload.

Drawing

Use the same visual grammar in both render paths:

• X axis: red (1.0, 0.18, 0.14).

• Y axis: green (0.22, 0.92, 0.32).

• Z axis: blue (0.25, 0.54, 1.0).

• Active axis: lighter/brighter variant while dragging.

• Translate: axis lines with arrowheads.

• Rotate: concentric rings with different radii per axis.

• Scale: axis lines with square handles at endpoints.

• Center: small white square at the projected origin.

Local frame drawing should alpha-blend anti-aliased line and polygon masks into the numpy frame. Headless ovui drawing should use omni.ui_scene.scene.Line and PolygonMesh under the shared overlay SceneView, then let the existing CUDA compositor blend RGBA over the BGRA stream.

Animation Interaction

Selection animation and manipulator drags must not write transforms at the same time.

gizmo = TransformGizmo(
    width,
    height,
    on_transform_changed=apply_world_transform_to_prim,
    on_drag_start=lambda path: animator.freeze(path),
    on_drag_end=lambda path: animator.resume(path),
)

When selection animation changes the selected prim's visible transform, include that animation offset in the gizmo position so the handles stay attached to the rendered object.

world = np.array(base_world_transform, copy=True)
world[3, 0:3] += animator.current_offset(path)
gizmo.set_selection(path, world_transform=world)

On drag start, freeze the animation first, then use the current visible transform as start_transform. On drag end, update the animator base transform before resuming so it does not snap back to the pre-drag position.

Tool Selection

Use conventional editor shortcuts and toolbar buttons:

Q = none/select
W = translate
E = rotate
R = scale

The active tool gates visibility and behavior. No active tool means no manipulator is rendered and no manipulator hit testing should consume camera input.

def set_active_tool(tool):
    normalized = None if tool in (None, "", "none", "select") else tool
    gizmo.set_active_tool(normalized)
    toolbar.set_checked(normalized)

For local ovui, wire keyboard shortcuts on the viewport hit rectangle and toolbar buttons in the header. For WebRTC, send a JSON app message such as {"event_type": "setActiveTool", "payload": {"tool": "translate"}} and update the server-side overlay state.

Common Pitfalls

• XformCommonAPI.SetRotate() only accepts GfVec3f; directly write existing xform ops and preserve their attribute type.

• Read viewer-input-routing for ovstream mouse button normalization before combining gizmo, camera, and pick handlers.

• HIT_RADIUS_PX = 18 is too small for WebRTC. Use 35.

• Hover consumption is intentional: contains() == True can block camera input so the cursor can communicate that the gizmo is interactive. Only start transforms on press hits.

• Convert viewport coordinates through the same letterbox mapping used for picking and camera input.

• Do not call renderer.step() concurrently with scene reset, transform binding, or stage mutation.

• Keep one shared ovui headless overlay window; multiple windows can break frame export.

• If an axis projects to a near-zero screen length, use a deterministic fallback direction instead of producing NaNs.

• If the gizmo is visible but the prim stays stationary, verify input ownership first and then verify that the drag callback reaches the live omni:xform write path for the selected prim.

Checklist

• [ ] Read viewport-overlays for WebRTC/headless ovui composition or local-viewer for inline ByteImageProvider drawing.

• [ ] Read prim-transform-safety before writing live omni:xform or combining gizmo drags with selection animation.

• [ ] Route gizmo input before camera input for press, move, release, and wheel.

• [ ] Project origin and axes every frame from the current camera view/projection.

• [ ] Use start_transform and start_mouse as the base for each drag update.

• [ ] During drag, write the selected prim's live omni:xform; on release, optionally commit to a session-layer USD edit path.

• [ ] Author USD xform ops in the session layer with parent compensation and Vec3f/Vec3d type preservation.

• [ ] Validate with a measured transform delta, not only a screenshot of a visible manipulator.

• [ ] Freeze animation during drag and resume from the new base transform after drag end.

See also: viewport-overlays, prim-transform-safety, viewer-input-routing, camera-controls, object-selection, selection-animation, local-viewer, streaming-messages.