[ WEBXR · HAND TRACKING · INPUT ]

Hand tracking in WebXR

Modern headsets can see your bare hands — 25 tracked joints per hand, in the browser. Learn how WebXR exposes them and how to design gestures that feel natural.

Watch the lecture, then read Max's deep-dive below.

▸ In this video you'll see

How WebXR exposes hand joints and poses
Detecting gestures (pinch, grab) from raw joint data
Building direct-touch interactions with no controllers
Doing pose detection efficiently (Web Workers / “Handy Work”)
Design considerations for hands-only interfaces

Hand tracking lets a user interact in VR and AR with their bare hands — no controllers. Headset cameras estimate the pose of each hand, and WebXR surfaces that to your web app as a live skeleton you can read every frame. It is the most natural input the immersive web offers: you reach out and touch things with the hands you already have.

25 joints per hand

WebXR's hand-input module models each hand as a skeleton of 25 joints — a wrist, plus joints down each finger from knuckle to tip. When hand tracking is active, each XRInputSource carries an XRHand, and for every frame you can ask for the pose (position + orientation) and even the radius of any joint:

for (const [jointName, joint] of inputSource.hand) {
  const pose = frame.getJointPose(joint, referenceSpace);
  // pose.transform.position  → where this joint is in space
  // pose.radius              → its approximate thickness
}

The index-finger-tip and thumb-tip joints are the two you will use most — the distance between them is the basis of the universal pinch gesture.

Detecting a pinch

Most hand interaction reduces to one primitive: the pinch (thumb tip meeting index tip), which acts as "click" or "grab." You detect it by measuring the distance between those two joint poses each frame and thresholding it:

const d = thumbTip.position.distanceTo(indexTip.position);
const isPinching = d < 0.02; // ~2 cm — tune per user

Add hysteresis (a slightly larger release threshold than press threshold) so a hand hovering near the boundary does not flicker between pressed and released.

Some platforms also expose a high-level select event: a system-recognized pinch fires the same select events as a controller trigger. Prefer that when you can — it respects each platform's calibration — and fall back to manual joint-distance detection for custom gestures the system does not provide.

Designing gestures that do not tire people

Keep hands low. Holding arms out in front — "gorilla arm" — fatigues within minutes. Design so interaction happens with elbows near the body.
Prefer a few reliable gestures. Pinch, grab, and point are understood instantly. Exotic multi-finger poses are hard to perform consistently and hard for the tracker to read.
Give feedback. Because there is no physical button and no haptic click, the user needs to see and ideally hear that a pinch registered — highlight the target, play a soft sound, snap the object.
Expect tracking loss. Hands leave the camera's view, or fingers occlude each other. Fail gracefully: pause, do not fling the held object away, and recover smoothly when tracking returns.

Done well, hand tracking is the moment the interface disappears — no learning a controller, just reaching out and doing. It is more forgiving to design around than it first appears, because almost everything useful can be built from pinch, grab, and point.

Keep going

MDN: XRHand — the joint model and API your code reads each frame
WebXR Hand Input spec — the official standard, with the full joint list

Hand Tracking WebXR Gestures Input