# Play with your camera

> Use your hands, face and body as a controller through your webcam: how tracking works, how to make it playable, its limits and what you can build.

Canonical: https://maru.audio/help/play-with-your-camera
Documentation index: https://maru.audio/help/llms.txt

Your webcam can be an instrument. Camera tracking follows 21 points on each
hand, the shape of your face and the pose of your body, many times a second.
Maru turns those movements into control signals: the roll of your wrist, the
pinch of two fingers, how wide you smile, how far you lean. Map them to a filter,
a reverb or a whole synth, and perform with gestures instead of knobs.

## Try it

- [HandMod](https://maru.audio/modules/hand-mod) turns hands, face and body into up to eight MIDI
  CC lanes. Start with the **Classic Mod** preset, then try **Theremin**,
  **Filter Sweep**, **Face Control** or **Conductor**. Place it before an
  instrument or effect that responds to those CC numbers.
- [Hand Chord](https://maru.audio/modules/hand-chord) picks a chord with one hand's gesture and
  strums twelve virtual strings with the other hand's index finger.

Allow camera access when asked. Tracking runs in your browser.

## How it works

A small vision model finds your hands, face or body in each camera frame and
places landmarks on them. Maru derives musical signals from those points:
positions, angles, distances, finger counts, recognised gestures such as an
open palm or a thumbs up, and the four-sided shape your two thumbs and index
fingers frame. Each signal is scaled from 0 to 1 and smoothed before it reaches
a parameter. [Camera and gesture control](https://maru.audio/learn/camera-and-gesture-control.md)
in Learn explains ranges, jitter and what happens when a hand disappears.

## Ideas

- **A theremin you can see.** Height for pitch bend, distance between hands for
  volume.
- **Expressive effects.** Open your hand to open a filter; tilt it to add
  vibrato. [Build a camera-controlled effect](https://maru.audio/learn/camera-controlled-effect.md)
  walks through one step by step.
- **Face as a pedal.** Raise your brows for a reverb swell while both hands
  stay on a guitar.
- **Conduct a texture.** Use torso lean and arm extension to move between
  sounds in a pad or a morphing instrument.
- **Air chords.** Play harmony with gestures, as Hand Chord does.
- **Performances and installations.** Let visitors play a piece by moving in
  front of a screen.

## Make it playable

- **Give every movement a range.** Map the part of a gesture you can repeat
  comfortably, not the whole camera frame.
- **Smooth it.** A still hand still jitters a little; add smoothing until the
  sound is steady, then stop.
- **Decide what happens when you leave.** When a hand drops out of view, hold
  the last value, glide back to a rest position or release the note.
- **Mind your arms.** Gestures held above the shoulder get tiring within a song.

## Limits

- **Light and background matter.** Dim rooms, backlighting and busy backgrounds
  reduce tracking quality.
- **Latency.** Camera control is slower and less precise than a knob or a key.
  Use it for sweeps, swells and gestures rather than tight rhythm.
- **Desktop browser only.** Camera control runs on module pages in browsers
  with camera access. Desktop plug-in builds do not offer it yet.

## Build your own

The code behind HandMod is the best place to start. Ask your coding agent to
read it before it builds anything:

```text
Read the Maru docs page "Camera control in a module", then read HandMod's
source with `maru module files hand-mod` and `maru module read hand-mod
src/index.ts`, including its ui folder. Build me an effect where my right
hand's height sweeps a low-pass filter with smoothing, and the filter glides
back to its knob position when my hand leaves the frame.
```

Developers can start from [Camera control in a module](https://maru.audio/docs/camera-control-in-a-module.md).