Generate audio in a module docs

A musician types "a soft wooden knock", presses Generate, auditions four takes and keeps one. The kept take plays through your instrument at once. This page shows how to build that flow. It is part of AI tasks in your module: your module asks for the task, and the host decides which model and runtime produce the audio. For choosing when generation belongs in a product at all, read Generation in your instruments and apps.

Choose the capability

Two face capabilities generate audio. They differ in where the result goes.

You want	Capability	Result	Who pays
The take to play in your instrument	`audio.generation/v1` (`face.generation`)	Installed into your Processor's sample slots	Depends on the runtime: on-device generation uses the person's own hardware; hosted generation is metered on their account
A sound saved to the person's library, attributed to your app	`app.generation/v1` (`face.appGeneration`)	A library take; the face receives status, not audio	The person's credits, after consent to your registered app

This page covers audio.generation/v1. For app.generation/v1, see Add generation to an instrument.

1. Declare where takes go

A generation target names a place a take can fill and what fits there. The host derives its limits, its install and its analysis from this declaration.

generation: {
  targets: [
    {
      id: "take",
      audioTypes: ["one-shot", "loop"],
      maxSeconds: 8,
      maxVariations: 8,
      install: { resources: ["takeB", "takeC"], meta: "genMeta" },
      analysis: ["pitch", "silence", "loop"],
      spectrogram: true,
    },
  ],
},
ui: {
  entry: "./ui/index.tsx",
  hostCapabilities: { optional: ["audio.generation/v1"] },
},

audioTypes is any of one-shot, loop, sfx and drum-kit.
maxSeconds is the longest take your slots are sized for (at most 120).
install.resources names exactly two sample resources (f32): the host writes the inactive one and then switches, so the Processor never reads a half-written buffer.
install.meta names an f32 data resource that tells the Processor which slot is active and what the host found in the take.
analysis asks the host for a root note (pitch), trimmed leading silence (silence) and a loop suggestion (loop).

A module may declare up to 16 targets, for example one per drum pad.

2. Read the take in the Processor

Declare the two sample slots and the meta slot as ordinary resources. The host fills them through the same loadResource path as any other resource; your Processor reads them the same way it reads a factory sample.

The meta slot holds 11 f32 values (GENERATED_TAKE_META in @maruaudio/audio-contracts):

Index	Field	Meaning
0	`source`	−1 for a generated take; the factory source index otherwise
1	`version`	Meta layout version (1)
2	`activeSlot`	0 or 1: which of `install.resources` holds the take
3–5	`rootMidi`, `rootCents`, `confidence`	Suggested root key; −1 and 0 for an unpitched take
6	`character`	0 silent, 1 tonal, 2 percussive, 3 textural
7–9	`loopStartFrame`, `loopEndFrame`, `loopScore`	Loop suggestion, −1 when none
10	`onsetFrame`	Leading frames the host trimmed

Frame positions are at the installed clip's own sample rate. Switch to the new slot at a block boundary and fade voices that still read the old one. The sampler-zone DSP leaf does this for you. Because factory takes can reach the same meta slot, one Processor path plays both.

3. Build the face

The face reads face.generation and calls its intent methods. Everything it sees is display-safe: status, progress, peaks, analysis and provenance.

tsx

const generation = face.generation

function onGenerate(prompt: string) {
  if (!generation || generation.state.availability !== "ready") return
  generation.generate({
    target: "take",
    prompt,
    seconds: 2,
    variations: 4,
    audioType: "one-shot",
  })
}

const target = generation?.state.targets.take
// target.phase: "idle" | "generating" | "ready" | "error"
// target.takes: each take's status, progress, peaks and analysis
// generation.selectTake("take", takeId) installs a take into the Processor

Method	Effect
`generate(request)`	Starts takes for one target: `prompt`, `seconds`, `variations`, optional `audioType`, `model` and `negativePrompt`
`selectTake(target, takeId)`	Installs the take; `target.installedTakeId` confirms it
`regenerateTake(takeId)`	Same request, new seed, replaces only that take
`pinTake`, `discardTake`	Keep a take through a regenerate, or drop it
`cancel(target)`	Stops that target's running job
`setConnector(id)`	Switches between the runtimes the host offers
`requestAuth()`	Opens the host's sign-in when a hosted runtime needs it

state.availability is ready, needs-auth, unavailable or error. When face.generation is absent, or unavailable, hide the Generate controls and keep the factory takes playable.

Optional host services — a take library, WAV export and Send to Sampler — appear in state.services. Check that object rather than the method before showing those actions.

The Crea Synth module is the complete reference: its target is in modules/crea-synth/src/index.ts and its Sound tab in modules/crea-synth/src/ui/index.tsx.

Runtimes: on this device or hosted

state.connectors lists the runtimes the host offers for this face. Each one reports a kind, a status, its limits, an optional cost and the models it can run:

Kind	Runs	Status to expect
`local-worker`	In the browser, on the person's GPU (WebGPU)	`needs-download` until the person agrees to download the model, then `available`
`hosted`	On Maru's servers, metered on the person's account	`needs-auth` until they sign in and are admitted
`local-daemon`	A desktop companion app (planned)	—
`mock`	Deterministic test tones in development	`available`

The host asks for consent before any model download and caches the model afterwards. Show the connector's label, status and statusDetail so a person understands why a choice is unavailable, and let them switch with setConnector.

Leave model unset to let the runtime choose for the audio type. Pass a model id only when your sound design depends on one; the ids a runtime accepts are listed in its connector's models, and an id a runtime does not list is refused rather than replaced.

Writing prompts

Describe the sound, not the instrument you are building. Name the source, articulation, space and length: "a dry wooden percussion hit, close microphone", "an evolving glassy pad, slow attack, wide stereo". The runtime adds any preamble its model expects for one-shots or loops, so do not add model-specific tags yourself.

Generate several variations per request and let the person choose. The host assigns seeds and records each take's seed and model in its provenance; regenerateTake tries a new seed for the same request.

Current implementations

Today's runtimes use the Stable Audio 3 family. These details can change as hosts add other models for the same task; your module does not need to change.

Runtime	Model	Audio types	Longest take
This device, WebGPU	Stable Audio 3 Small SFX	one-shot, sfx	8 s
This device, WebGPU	Stable Audio 3 Small Music	loop	30 s
This device, browsers without WebGPU	Stable Audio 3 Small SFX can run on WASM, but website hosts do not offer it yet	—	—
Hosted (preview)	Stable Audio 3 Medium and the Small models	one-shot, loop, sfx	Reported in the connector's `models` and `limits`

The on-device model is a download of about 1.6 GB. Hosted generation needs an account admitted to the preview. Stable Audio output terms are under review, and connectors report the model licence as unknown until that review completes.

Where it works

Host	Generation
Website module pages	On this device and hosted
`module:dev` workbench	Test runtime, this device and a configured hosted proxy
Studio and native plug-ins	Not yet; factory and saved takes play
Mobile and hardware	No

Documentation