Isaac Lab 2.3.0 Glove Device Example (Python)

RokokoGloveDevice is an NVIDIA Isaac Lab 2.3.0 teleop device that presents solved Smartgloves hands to Isaac Lab. It works by impersonating Isaac Lab’s own OpenXRDevice: same raw-data contract — 26 XrHandJointEXT joints per hand, w-first quaternions, held poses on a stale hand — fed from the hand solver’s RGMP stream instead of an XR runtime.

Because the contract is identical, Isaac Lab’s own handtracking retargeters and teleop tasks run unchanged. Only the device is swapped.

flowchart LR
    Driver["SDK Driver<br/>rokoko-sdk"]
    Solver["rkk-hand-solver<br/>--emit-openxr"]
    Device["RokokoGloveDevice<br/>(this example)"]
    Isaac["Isaac Lab<br/>retargeters, teleop,<br/>HDF5 recording"]

    Driver -->|"TCP :12276 (RGMP)"| Solver
    Solver -->|"TCP :12277 (RGMP)"| Device
    Device --> Isaac

The example lives in $ROKOKO_SDK_HOME/examples/python/isaac. Everything except the DeviceBase subclass itself is pure Python with no Isaac or NumPy dependency, and rokoko_glove_device.py imports isaaclab lazily — so the decode, rebase, and calibration code can be read and exercised on any machine.

Module	Role
glove_stream.py	The solver’s RGMP stream → one latest HandFrame per hand
frames.py	xr_base (Y-up) → Isaac world (Z-up): one rigid transform per hand
calibration.py	The session inputs to that transform: yaw capture, wrist anchors
rokoko_glove_device.py	The Isaac Lab DeviceBase shim impersonating OpenXRDevice
tools/	Acceptance and recording scripts to run on the Isaac host

What you need

• rkk-hand-solver, the hand-solving binary in $ROKOKO_SDK_HOME/bin, started with --emit-openxr so it publishes the joints_openxr group.

  rkk-hand-solver --emit-openxr

  The Python hand solver next door does not emit that group, so it cannot feed this device. When Isaac runs on another machine, add --listen 0.0.0.0:12277; the solver has to be reachable from the Isaac host, and a tunnel such as Tailscale beats exposing the port.

• Isaac Lab 2.3.0 (nvcr.io/nvidia/isaac-lab:2.3.0, core isaaclab 0.47.1, bundling Isaac Sim 5.1.0). That is the release the device and its tools were validated against.

• The whole python/ directory on sys.path on the Isaac host. The package imports its sibling rgmp_core package, so copy the entire tree, not just isaac/.

Using the device

On an Isaac host, the device drops in where the stock OpenXR one would go:

from isaac.rokoko_glove_device import RokokoGloveDevice, RokokoGloveDeviceCfg

device = RokokoGloveDevice(RokokoGloveDeviceCfg(host="<glove host>", port=12277))
device.calibrate()          # operator holds a hand toward workspace +X
data = device.advance()     # OpenXRDevice-shaped dict (or retargeted commands)
device.close()

RokokoGloveDeviceCfg also takes the two wrist anchor poses and the head pose (Isaac Lab’s 7-element x y z qw qx qy qz layout) plus a facing_yaw_rad for the calibration gesture. calibrate() can be bound to an input via add_callback("CALIBRATE", ...) and trigger("CALIBRATE"). reset() follows OpenXRDevice.reset — it clears held poses but keeps the calibration, since an environment reset must not lose the session’s yaw offset.

The stream by itself

GloveStream runs the RGMP client on a background thread and keeps exactly the newest decoded joints_openxr frame per hand:

from isaac.glove_stream import GloveStream

with GloveStream("127.0.0.1", 12277) as stream:
    generation = stream.wait(0, timeout=5.0)      # first announce/frames
    for device_id, frame in stream.latest().items():
        print(frame.hand, frame.timestamp_us, frame.joints[0])

The depth-1 mailbox is deliberate. For realtime teleop a queued old pose is worse than a dropped one, and a consumer stepping on its own clock — which Isaac is — only ever wants the freshest frame each step.

Values pass through in the solver’s own terms, which is the wire contract --emit-openxr publishes:

• the 26 XrHandJointEXT joints in enum order, named in snake_case without the xr_ prefix (palm, wrist, thumb_metacarpal, …). These match isaaclab.devices.openxr.common.HAND_JOINT_NAMES exactly, so no mapping table is needed.
• one pose per joint against xr_base — the glove frame rotated +90° about X, so Y is up. Position in meters, then a quaternion as x y z w (Isaac Lab wants w x y z; the shim reorders).
• wrist-relative: the wrist stays at the origin, so an anchor pose has to place the hand in the workspace.
• yaw is magnetic-north-referenced with an arbitrary per-session zero, so pitch and roll are absolute but heading needs calibrating.

Rebasing into the Isaac world

Getting from xr_base into Isaac’s Z-up world takes one HandRebase per hand, composed from the fixed +90° rotation about X and the two session inputs calibration.py supplies:

from isaac.calibration import FixedWristAnchor, capture_yaw_offset
from isaac.frames import HandRebase, Pose

# Once per session: operator holds a hand toward the workspace +X.
yaw = capture_yaw_offset(stream.latest()[device_id])

anchors = FixedWristAnchor(
    left=Pose(position=(0.1, 0.4, 0.8)),
    right=Pose(position=(0.1, -0.4, 0.8)),
)
rebase = {
    d: HandRebase.for_isaac(yaw, anchors.pose(info.hand))
    for d, info in stream.hands().items()
}
world_frame = rebase[device_id].apply(stream.latest()[device_id])

One yaw capture covers both hands — it corrects the shared magnetic reference, not the hand. Heading spread on a real glove held still measured about 8° over 10 s (tremor plus sensor noise), so averaging the capture over a short window instead of a single frame is a worthwhile refinement.

FixedWristAnchor pins each wrist to a configured workspace pose, which is enough for tabletop teleop where finger and wrist rotation drive the robot. Arm travel needs a tracked anchor; WristAnchor is the protocol to implement for one.

Tools

isaac/tools/ holds the scripts used to validate the device on an Isaac host. They expect a solved-hand stream already running and python/ on sys.path. Each prints an …-OK line and exits 0 on success.

Script	What it does
isaac_device_check.py --host <solver host>	Constructs the device inside a real Isaac Lab app and verifies the OpenXR-shaped contract end to end: joint set, pose layout, finger motion. Takes --hands, since a real session may have only one glove.
isaac_gr1t2_retargeter_check.py	Drives Isaac Lab’s GR1T2Retargeter (dex-retargeting onto the Fourier GR1T2 hands, using the task’s own config) directly from the device, and checks that all 36 output dims — 2×7 wrist poses plus 22 hand joints — are finite and moving.
isaac_teleop_record.py --task <id> --steps N --out <dir>	Reuses a stock task’s handtracking teleop configuration verbatim, swaps in RokokoGloveDevice, and records episodes to HDF5. Validated on Isaac-Stack-Cube-Franka-IK-Rel-v0 and Isaac-PickPlace-GR1T2-Abs-v0.
measure_hop_latency.py	Taps the driver’s stream and the solver’s output side by side and reports the solver hop. Measured on one host: mean 0.09 ms, p95 0.12 ms, max 0.42 ms over 1656 samples — negligible. A remote glove host adds its network RTT on top.

A typical run inside the pinned container, with the python/ tree mounted at /workspace/rokoko and an output directory at /out:

./isaaclab.sh -p /workspace/rokoko/isaac/tools/isaac_teleop_record.py \
    --host <solver host> --port 12277 --task Isaac-Stack-Cube-Franka-IK-Rel-v0 \
    --steps 300 --out /out

Note

isaac_teleop_record.py does not call calibrate(), so the yaw-to-arm direction runs on raw magnetic north unless you add the gesture yourself.

Isaac Lab 2.3.0 notes

Things the scripts already handle, worth knowing before you touch the GR1T2 tasks or write your own environment script:

• the pick_place package — all GR1T2 tasks — is blacklisted from isaaclab_tasks’ bulk import. Import it explicitly to register the tasks.
• pinocchio must be imported before AppLauncher (Isaac Lab’s build has to beat Isaac Sim’s — the same reason record_demos.py has --enable_pinocchio), or Pink IK and dex_retargeting die with TypeError: No Python class registered for C++ class std::vector<…>.
• retargeters/dex/DexRetargeter is an unimplemented stub in this release. The humanoid retargeters (GR1T2Retargeter) are the real full-hand path.
• Isaac Lab’s GripperRetargeter reads only the thumb-tip to index-tip distance: closed below 3 cm, open above 5 cm, with hysteresis. A fist keeps the tips several centimeters apart and so does nothing — a pinch is the gesture that works. A real glove spans roughly 0.5–11 cm on that distance.
• the Franka stack task’s Se3RelRetargeter is configured with use_wrist_position=True, use_wrist_rotation=False, zero_out_xy_rotation=True, so with a fixed wrist anchor the only arm input is hand yaw. Position deltas log as zero until a tracked anchor supplies arm travel.
• EXPORT_ALL also writes an empty trailing episode group once the environment’s time-out auto-reset fires. isaac_teleop_record.py counts only episodes that have an actions dataset.
• the release rkk-hand-solver binary needs glibc ≥ 2.39. On an Ubuntu 22.04 host, run it in an ubuntu:24.04 container on the host network.

Running on a remote GPU host

The device was validated with the glove host (driver plus solver) on a laptop and Isaac Lab on a cloud GPU instance, joined by Tailscale. What carried over:

• Latency. A ~99 ms RTT between the two ends was fine for recording — frames are source-stamped — but noticeable for closed-loop teleop feel. Pick a region near the operator.
• Headless Isaac Sim (nvcr.io/nvidia/isaac-sim:5.1.0) is viewed with NVIDIA’s Isaac Sim WebRTC Streaming Client. Isaac Sim serves exactly one WebRTC client per instance and owns port 49100, so stop any standing streaming container before launching a session with --livestream 1. The firewall needs SSH (TCP 22), WebRTC signaling (TCP 49100) and media (UDP 47995–48012, 49100), restricted to the operator’s address.
• NVIDIA driver. Driver 595 crashed Isaac Sim 5.1’s RTX renderer plugins at startup (a breakpad fatal in librtx.scenedb, container restart loop). The 580 server branch (nvidia-driver-580-server-open, 580.173.02) works.

A live-view launch looks like this, with PUBLIC_IP set to the address the streaming client will connect to — Isaac Lab passes it through as --/app/livestream/publicEndpointAddress:

docker run -d --name isaac-lab-live --gpus all --network host \
  -e ACCEPT_EULA=Y -e PRIVACY_CONSENT=N -e PUBLIC_IP=<public ip> \
  -v ~/rokoko:/workspace/rokoko -v ~/rokoko-out:/out \
  -v ~/docker/isaac-lab/cache/kit:/isaac-sim/kit/cache \
  -v ~/docker/isaac-lab/cache/ov:/root/.cache/ov \
  -v ~/docker/isaac-lab/cache/glcache:/root/.cache/nvidia/GLCache \
  -v ~/docker/isaac-lab/cache/computecache:/root/.nv/ComputeCache \
  --entrypoint bash nvcr.io/nvidia/isaac-lab:2.3.0 -c \
  "cd /workspace/isaaclab && ./isaaclab.sh -p \
     /workspace/rokoko/isaac/tools/isaac_teleop_record.py \
     --host <solver host> --task Isaac-PickPlace-GR1T2-Abs-v0 \
     --livestream 1 --steps 100000 --out /out 2>&1 | tee /out/live.log"

Port 49100 opens about 30 s in, and the scene is stepping once the log says app ready — roughly 2.5 min for GR1T2, 1 min for Franka. With --livestream the recorder also prints the action vector once a second and keeps the retargeters’ debug markers on. Restarting the session restarts the stream server, so reconnect the client after every relaunch.

Where to go next

• Hand solving guide — the rkk-hand-solver sidecar that feeds this device, including the --emit-openxr group it consumes.
• Hand Solver (Python) — the pure-Python solver, for reading and tuning the solve itself.
• Smartgloves reference — sensors, frames, and the data behind the solve.
• RGMP specification — the wire format the device decodes.