# Camera control in a module

> Use hand, face and body tracking from the host's camera task as control signals: declare the tracking you need, read observations in the face, and map them to parameters or MIDI.

Canonical: https://maru.audio/docs/camera-control-in-a-module
Documentation index: https://maru.audio/llms.txt

Camera control turns movement into music: a hand that sweeps a filter, a smile
that opens a reverb, two hands that frame a shape you can play. It is part of
[AI tasks in your module](https://maru.audio/docs/ai-tasks.md). For ideas, limits and how
it feels to perform with, read
[Play with your camera](https://maru.audio/help/play-with-your-camera.md) in the Help
Center, and try [HandMod](https://maru.audio/modules/hand-mod).

## 1. Declare the tracking you need

Request only the families you use. Each one costs camera time and a model
download.

```ts
ui: {
  entry: "./ui/index.tsx",
  hostCapabilities: {
    optional: [
      "vision.hand-landmarks/v1",
      "vision.face-landmarks/v1",
      "vision.pose-landmarks/v1",
    ],
  },
},
```

`vision.hand-gestures/v1` adds recognised gestures (open palm, thumbs up,
victory and others) on top of hand landmarks.

## 2. Start the camera from the face

`face.vision` owns the camera, permissions and the tracker. The face asks for
it and reads the results:

```tsx
const vision = face.vision
if (!vision || vision.status === "unavailable") return <p>No camera here.</p>

<Button onClick={() => vision.enable()} disabled={vision.busy}>
  {vision.running ? "Camera on" : "Start camera"}
</Button>
```

| Member | Purpose |
| --- | --- |
| `status` | `idle`, `prompt`, `loading`, `running`, `denied`, `error` or `unavailable` |
| `enable()`, `stop()` | Ask for camera access and start tracking, or stop |
| `devices`, `selectDevice(id)` | Choose a camera |
| `families`, `setFamilies(...)` | Switch hand, face and pose tracking on or off |
| `readObservation()` | The latest observation, or `null` |
| `subscribeObservations?(listener)` | Every observation, before preview throttling; unsubscribe on detach |
| `attachPreview(...)` | Draw the mirrored video with overlays into a canvas the face owns |

Handle `denied` with a clear message and a way to try again; never block the
rest of the module on the camera.

## 3. Read an observation

Each observation is bounded and versioned:

| Field | Contents |
| --- | --- |
| `tracking` | `empty`, `partial`, `tracked` or `error` |
| `hands` | Up to two hands, each with `handedness`, `confidence` and 21 points |
| `face`, `pose` | Landmark points when those families are on |
| `signals` | Up to 128 normalized control values, each with an `id`, a `value` and `present` |
| `source` | Camera size and whether the picture is mirrored |

Prefer `signals` for control: they are already normalized (hand position, roll,
pinch, extended fingers, gestures, two-hand shape, mouth, brows, torso lean and
more). Use raw points only for drawing or a measurement the signals do not
cover. Hands are the player's physical hands; the mirrored preview does not swap
them.

## 4. Turn signals into sound

Vision runs at control rate, outside the audio callback. Bring a signal into the
audio path the same way any control does:

- set a parameter of your module from the face, with smoothing; or
- send MIDI CC to the next module, as HandMod does through a data input port and
  a mapper graph that emits CC.

Give every mapping a range, a curve and smoothing, and decide what happens when
`present` is false: hold the last value, glide to a rest value, or release.

## Ask your agent

```text
Read the Maru docs page "Camera control in a module", then read HandMod's
source with `maru module files hand-mod` and `maru module read hand-mod
src/index.ts`, including its ui folder. Add camera control to my module: the
right hand's height should sweep the filter cutoff with smoothing, and the
filter should glide back to its knob position when the hand leaves the frame.
```

[Hand Chord](https://maru.audio/modules/hand-chord) (`hand-chord`) is a second reference: it
picks chords with recognised gestures and strums twelve virtual strings with
the other hand's index finger, sending MIDI notes rather than CC.

## Availability

Camera tasks run on website module pages and the `module:dev` workbench, in
browsers with camera access. Native plug-ins, mobile and hardware targets do not
offer them; a module there keeps its ordinary controls.