Play with your camera

Your webcam can be an instrument. Camera tracking follows 21 points on each hand, the shape of your face and the pose of your body, many times a second. Maru turns those movements into control signals: the roll of your wrist, the pinch of two fingers, how wide you smile, how far you lean. Map them to a filter, a reverb or a whole synth, and perform with gestures instead of knobs.

Try it

• HandMod turns hands, face and body into up to eight MIDI CC lanes. Start with the Classic Mod preset, then try Theremin, Filter Sweep, Face Control or Conductor. Place it before an instrument or effect that responds to those CC numbers.
• Hand Chord picks a chord with one hand's gesture and strums twelve virtual strings with the other hand's index finger.

Allow camera access when asked. Tracking runs in your browser.

How it works

A small vision model finds your hands, face or body in each camera frame and places landmarks on them. Maru derives musical signals from those points: positions, angles, distances, finger counts, recognised gestures such as an open palm or a thumbs up, and the four-sided shape your two thumbs and index fingers frame. Each signal is scaled from 0 to 1 and smoothed before it reaches a parameter. Camera and gesture control in Learn explains ranges, jitter and what happens when a hand disappears.

Ideas

• A theremin you can see. Height for pitch bend, distance between hands for volume.
• Expressive effects. Open your hand to open a filter; tilt it to add vibrato. Build a camera-controlled effect walks through one step by step.
• Face as a pedal. Raise your brows for a reverb swell while both hands stay on a guitar.
• Conduct a texture. Use torso lean and arm extension to move between sounds in a pad or a morphing instrument.
• Air chords. Play harmony with gestures, as Hand Chord does.
• Performances and installations. Let visitors play a piece by moving in front of a screen.

Make it playable

• Give every movement a range. Map the part of a gesture you can repeat comfortably, not the whole camera frame.
• Smooth it. A still hand still jitters a little; add smoothing until the sound is steady, then stop.
• Decide what happens when you leave. When a hand drops out of view, hold the last value, glide back to a rest position or release the note.
• Mind your arms. Gestures held above the shoulder get tiring within a song.

Limits

• Light and background matter. Dim rooms, backlighting and busy backgrounds reduce tracking quality.
• Latency. Camera control is slower and less precise than a knob or a key. Use it for sweeps, swells and gestures rather than tight rhythm.
• Desktop browser only. Camera control runs on module pages in browsers with camera access. Desktop plug-in builds do not offer it yet.

Build your own

The code behind HandMod is the best place to start. Ask your coding agent to read it before it builds anything:

Read the Maru docs page "Camera control in a module", then read HandMod's
source with `maru module files hand-mod` and `maru module read hand-mod
src/index.ts`, including its ui folder. Build me an effect where my right
hand's height sweeps a low-pass filter with smoothing, and the filter glides
back to its knob position when my hand leaves the frame.

Developers can start from Camera control in a module.