AI tasks in your module
Use generation, transcription, separation, rhythm and analysis as host tasks: declare the task, let the host run whichever model and runtime it supports, and keep the module working when a task is unavailable.
A module asks for a task, such as "generate a take", "turn this clip into MIDI" or "split these stems". It never names a model or runs one. The host chooses an implementation it supports — on this device, on Maru's servers, or through a provider — runs it outside the audio callback, and hands the result back through the module's existing contracts.
Because tasks are separate from models, the implementation behind a task can change without changing your module. A faster model, a new provider or a local runtime arrives as another option on the host, not as an SDK migration. The model directory describes the models themselves; AI for music explains why Maru works on these tasks. For what each task is good for, its limits and ideas to build, start with AI tasks in the Help Center.
How a task reaches your module
- Declare it. List the task's host capability as optional in the module's UI options, and declare any resource slots the result is installed into.
- Read it from the face. The host passes a task surface as a face prop: display-safe state (status, progress, takes, notes) and intent methods (generate, select, cancel). The face never sees tokens, URLs or model files.
- Handle "not here". A host without the task omits the prop. Keep the module's ordinary controls, factory sounds and saved results usable.
ui: {
entry: "./ui/index.tsx",
hostCapabilities: { optional: ["audio.generation/v1"] },
},
Inference always runs beside the audio path. A Processor only ever receives ordinary data — a sample buffer, MIDI notes, a parameter — through the same resource and event contracts it already uses.
Tasks
| Task | In a module face | From an app or server | Guide | Current implementations |
|---|---|---|---|---|
| Generate audio | audio.generation/v1 (takes installed into the Processor) or app.generation/v1 (takes saved to the person's library) | generate_audio job | Generate audio in a module | Stable Audio 3 Small, Stable Audio 3 Medium |
| Audio to MIDI | audio.transcription/v1 | transcribe_audio_to_midi with mode monophonic, polyphonic or drums | Audio to MIDI in a module | Basic Pitch (polyphonic), Pitchy (monophonic, no model), Inverse Drum Machine (drums) |
| Separate stems | audio.separation/v1 | separate_audio_stems | Separate stems in a module | Spleeter (device), SCNet (server) |
| Beats and tempo | Beat grid through audio.separation/v1 (trackRhythm); model-free onsets from @maruaudio/audio-sdk/analysis/novelty | analyze_audio with tempo, or rhythm alone for a beat grid | Beats and tempo in a module | Beat This (beats and downbeats) |
| Analyse a clip | audio.clip-analysis/v1 | analyze_audio | Add generative audio to your app | Model-free loudness, sections and quality |
| Camera control | vision.hand-landmarks/v1, vision.face-landmarks/v1, vision.pose-landmarks/v1, vision.hand-gestures/v1 | — | Camera control in a module | MediaPipe landmark models |
| Transcribe speech | — | transcribe_speech | — | None yet: jobs are refused before execution |
The implementation column describes what hosts run today. Inspect the task's
state in the face, or maru.audio.capabilities() from an app, for what a host
actually offers right now; a catalog entry alone does not mean a model is
online.
Where tasks are available
| Host | Generate audio | Audio to MIDI, stems, clip analysis | Camera control |
|---|---|---|---|
| Website module pages | On this device (WebGPU browsers) and hosted, for admitted accounts | Yes | Yes |
module:dev workbench | Test runtime, this device, and hosted through a configured proxy | Not established | Yes |
| Studio preview, desktop Studio | Not yet | Not established | — |
| Native plug-ins (VST3, AU, CLAP) | Not yet: factory and saved takes play | Not yet | — |
| Mobile and hardware targets | No | No | No |
A module built today with an optional task capability runs on every target; the task simply appears where a host supports it.
Licences and attribution
Each implementation carries its own licence. A face's generation state reports each model's licence string, and the model directory records the terms Maru has reviewed. Some open models are licensed for non-commercial use only, and Stable Audio output terms are still under review; check the model page before you sell a module whose factory content was generated.
Further reading
- Generate audio in a module, Audio to MIDI, Separate stems, Beats and tempo and Camera control
- Add generative audio to your app: jobs, sources, recovery, consent and billing.
- Generate audio: the hosted generation architecture.
- Adding a model: what changes when a model becomes a feature.