Note 01 · Foundations

How HandScenes is built

The shared machinery every scene sits on: hand tracking, the render loop, and the rules that keep it fast.

Runs on your device

Everything happens in the browser. The webcam feed, the hand tracking, and the graphics all run client side, and no frame ever leaves your machine. That single choice shaped the rest of the build. With no server to call, the model and the renderer both have to be light enough to live in a tab, and privacy stops being a policy and becomes a property of how the thing works.

The hand signal

Tracking comes from MediaPipe Tasks HandLandmarker, compiled to WebAssembly. It reads up to two hands in video mode and returns 21 landmarks for each. The GPU path is tried first, with a CPU fallback for the devices (older iOS Safari, mostly) that reject it.

Raw landmarks jitter frame to frame, so every point is smoothed with a small step toward its new reading, and fingertip velocity is tracked for the scenes that react to speed. Two gestures get computed once and reused everywhere: openness, where a fist reads 0 and an open hand reads 1, and pinch, the gap between thumb and index. Both are divided by the hand's own size, so they behave the same whether your hand is near the lens or across the room.

One renderer, scenes on demand

A single three.js WebGL renderer drives all of it. Each scene is a small module exposing the same handful of methods (mount, update, render, resize, dispose), which lets the shell stay ignorant of whether it is drawing a galaxy or a drum grid.

Scenes are not built until you open them. An earlier version compiled every scene at startup and paid the shader-compile cost for ones nobody had picked. Now a scene is constructed on first selection and held in a small cache of the three most recent. Anything older gets its geometry, materials, and render targets released, so GPU memory stays bounded no matter how long you browse.

Shipping it

Dependencies are vendored (three.js, the MediaPipe runtime and model, the fonts) and served from the same origin behind a strict content security policy. A few cookieless events measure the funnel, landed then started then picked a scene then changed a control, without identifying anyone. When a resource fails to load, a small fallback panel offers a reload rather than leaving a dead screen.

Next: Cat's Cradle →