← All modules

@maru shared this module so you can build on it and use it in your DAW.

AvailableMIDI & Modulation

HandMod

Map hand gestures to MIDI control.

Frames are analysed in this tab and discarded. No frame, still, landmark or metric ever leaves your browser, only the CC numbers you mapped.

About

HandMod turns your hands, face and body into a MIDI controller. Camera tracking follows what you do in front of the screen, and up to eight lanes turn those movements into MIDI CC messages for a synth or effect. It makes no sound of its own.

The idea

Most modulation is either a shape drawn in advance or a knob you need a spare hand for. HandMod knows what a hand is, so a lane can follow the roll of your wrist or the pinch of two fingers rather than a blur of movement, and a gesture you can repeat on purpose is one you can perform with.

There are 47 sources to choose from: position, roll, pitch, yaw, rotation, spread, pinch, area, extended fingers and gesture for each hand; the distance between your hands and between your index fingers, and their height difference; mouth, eyes, brows, smile and head angle from the face; shoulders, elbows, hips, torso lean and arm extension from the body.

The two-hand shape is the part worth trying first. Bring both thumbs and index fingers into view and HandMod measures the four-sided shape they frame: its area, centre and angle. Twist one hand past the other and the shape crosses over itself; HandMod reports that as a separate Shape Twist source, an on/off switch you make in the air.

Controls

Each of the eight lanes has:

  • Source: which movement it follows.
  • CC Number and MIDI Channel: where it sends.
  • Min and Max: the CC range the movement is scaled into.
  • Curve: linear, exponential, logarithmic or S-curve.
  • Smoothing and Rate: how much jitter is filtered out and how often values are sent.
  • Enabled: turns the lane on or off.

Capture settings choose whether the camera picture is mirrored, whether the tracking overlay and a visual style are drawn over it, and whether face, body and two-hand shape tracking are switched on.

Getting started

Allow camera access, hold a hand up and tilt it. The default setup, also the Classic Mod preset, sends the roll and pitch of each hand on CC 1 to 4. Other presets show the range: Theremin maps your right hand's height, position and pinch to modulation, cutoff and expression; Filter Sweep puts cutoff (CC 74) under your right hand; Face Control uses mouth, brows, head roll and smile; Conductor follows your arms and shoulders; and Quad Shape maps the two-hand shape, with its twist on the sustain pedal (CC 64). Body tracking is off by default, so switch on Track Pose for Conductor.

Place HandMod before an instrument or effect that responds to those CC numbers.

How it works.

A getUserMedia stream feeds MediaPipe's tasks-vision landmarkers, running on the GPU through WebGL where the browser allows it. Hands come back as 21 points each, the face as 478, the pose as 33, all in normalised frame coordinates. Each set is smoothed with a one-pole filter across frames before anything reads it, because raw landmark output jitters by a pixel or two even from a perfectly still hand and that jitter would otherwise arrive as audible noise on every lane.

The metric layer is pure maths over those points and is the part that is unit-tested: wrist-to-knuckle vectors become roll, pitch and yaw through a small orientation solver ported from the original tool; fingertip distances become spread and pinch; the convex extent becomes area. Shape mode takes four fingertips, orders them into a quad, and runs a shoelace area, a centroid, a long-axis angle and a segment-intersection test - that last one is what reports the twist, splitting the quad into two triangles and summing their areas instead.

Each enabled lane then reads one normalised value, applies its curve, scales it into its Min-Max window, smooths it again at lane level and emits a CC only when the rounded value actually changed, so a still hand does not spam identical bytes. Every event goes through the shared Audio & MIDI session to the selected destination, while a local parameter mirror keeps the page's controls visually in step without opening a second MIDI connection.

None of this is audio DSP, so HandMod has no kernel and no parity claim. It runs on the camera's frame clock rather than the audio block clock, and expect roughly 40-80 ms between a physical move and an audible change - most of it camera capture and model inference, not the mapping. That is fine for sweeps, swells and shapes, and it is not fine for triggering notes, which is why HandMod sends no notes.

What would you make with HandMod?

Describe what you want to make. Build it with the Studio agent.

Goes well with.

Prototype

CamMod

Map camera motion to MIDI CC.

Prototype

OSC Mini

A compact subtractive synth for learning the signal path.

Prototype

Chordex

Generate a voiced chord from one note.