Camera control in a module

Camera control turns movement into music: a hand that sweeps a filter, a smile that opens a reverb, two hands that frame a shape you can play. It is part of AI tasks in your module. For ideas, limits and how it feels to perform with, read Play with your camera in the Help Center, and try HandMod.

1. Declare the tracking you need

Request only the families you use. Each one costs camera time and a model download.

ui: {
  entry: "./ui/index.tsx",
  hostCapabilities: {
    optional: [
      "vision.hand-landmarks/v1",
      "vision.face-landmarks/v1",
      "vision.pose-landmarks/v1",
    ],
  },
},

vision.hand-gestures/v1 adds recognised gestures (open palm, thumbs up, victory and others) on top of hand landmarks.

2. Start the camera from the face

face.vision owns the camera, permissions and the tracker. The face asks for it and reads the results:

const vision = face.vision
if (!vision || vision.status === "unavailable") return <p>No camera here.</p>

<Button onClick={() => vision.enable()} disabled={vision.busy}>
  {vision.running ? "Camera on" : "Start camera"}
</Button>

Member	Purpose
status	idle, prompt, loading, running, denied, error or unavailable
enable(), stop()	Ask for camera access and start tracking, or stop
devices, selectDevice(id)	Choose a camera
families, setFamilies(...)	Switch hand, face and pose tracking on or off
readObservation()	The latest observation, or null
subscribeObservations?(listener)	Every observation, before preview throttling; unsubscribe on detach
attachPreview(...)	Draw the mirrored video with overlays into a canvas the face owns

Handle denied with a clear message and a way to try again; never block the rest of the module on the camera.

3. Read an observation

Each observation is bounded and versioned:

Field	Contents
tracking	empty, partial, tracked or error
hands	Up to two hands, each with handedness, confidence and 21 points
face, pose	Landmark points when those families are on
signals	Up to 128 normalized control values, each with an id, a value and present
source	Camera size and whether the picture is mirrored

Prefer signals for control: they are already normalized (hand position, roll, pinch, extended fingers, gestures, two-hand shape, mouth, brows, torso lean and more). Use raw points only for drawing or a measurement the signals do not cover. Hands are the player's physical hands; the mirrored preview does not swap them.

4. Turn signals into sound

Vision runs at control rate, outside the audio callback. Bring a signal into the audio path the same way any control does:

• set a parameter of your module from the face, with smoothing; or
• send MIDI CC to the next module, as HandMod does through a data input port and a mapper graph that emits CC.

Give every mapping a range, a curve and smoothing, and decide what happens when present is false: hold the last value, glide to a rest value, or release.

Ask your agent

Read the Maru docs page "Camera control in a module", then read HandMod's
source with `maru module files hand-mod` and `maru module read hand-mod
src/index.ts`, including its ui folder. Add camera control to my module: the
right hand's height should sweep the filter cutoff with smoothing, and the
filter should glide back to its knob position when the hand leaves the frame.

Hand Chord (hand-chord) is a second reference: it picks chords with recognised gestures and strums twelve virtual strings with the other hand's index finger, sending MIDI notes rather than CC.

Availability

Camera tasks run on website module pages and the module:dev workbench, in browsers with camera access. Native plug-ins, mobile and hardware targets do not offer them; a module there keeps its ordinary controls.