Audio to MIDI in a module docs

Audio to MIDI turns a performance into notes you can edit, quantise and send to any instrument. It is part of AI tasks in your module. For what the task is good at and where it fails, read Turn audio into MIDI in the Help Center.

There are two shapes of this task:

You want	Use	Runs
Notes from a recorded clip, after the fact	`audio.transcription/v1` (`face.transcription`)	In the host's worker, outside the audio callback
Notes from a live input, while it plays	The `@maruaudio/dsp/analysis/pitch-tracking` leaf in your Processor	Inside your Processor, on every target

Clip transcription

1. Declare the task and a timeline

The host transcribes; your Processor plays the result back as MIDI. Declare the capability, a MIDI output port and a data resource the host fills with the note timeline:

ports: [{ id: "midi-out", label: "MIDI", direction: "output", kind: "midi" }],
implementation: {
  kind: "processor",
  source: "src/dsp/processor.ts",
  maxEventOutputs: 256,
  resources: [
    { id: "timeline", kind: "data", elementType: "f32", maxElements: 500003, loading: "preload" },
  ],
},
ui: {
  entry: "./ui/index.tsx",
  hostCapabilities: { optional: ["audio.transcription/v1"] },
},

The Processor can delegate to the shared @maruaudio/dsp/processors/midi-timeline leaf, which schedules the notes on the sample clock. PitchRead's src/dsp/processor.ts is that leaf and nothing else.

2. Drive it from the face

face.transcription is display-safe state plus intent methods:

tsx

const transcription = face.transcription
if (!transcription) return <p>Transcription is not available here.</p>

const { phase, source, notes, progress } = transcription.state
// phase: "idle" | "loading" | "transcribing" | "ready" | "error"

<Button onClick={transcription.chooseSource}>Choose audio</Button>
<Button disabled={!source || phase === "transcribing"} onClick={transcription.transcribe}>
  Extract notes
</Button>

Member	Purpose
`chooseSource()`, `chooseExample?(id)`, `examples`	Pick a file or a host-supplied example clip
`recording?`	Optional bounded capture from the host's audio input: `start`, `stop`, `cancel`
`transcribe()`, `cancel()`	Run or stop extraction
`state.notes`	`startSeconds`, `durationSeconds`, `midi`, `velocity` (0 to 1) for each note
`state.source`	Label, duration, peaks, optional waveform and spectrogram for display
`settings?`	The models the host offers (`models`, `setModel`) and their tuning parameters (`parameters`, `setParameter`)
`sourcePlayback?`, `audition?`	Play the original, or blend original and MIDI audition
`createMidiFile()`	Host-encoded MIDI bytes for download or drag

settings.changedSinceResult tells you the draft settings differ from the result on screen; changes apply to the next extraction, never to an existing result.

Choosing a mode

The host lists its implementations in settings.models. Today they cover three kinds of source:

Source	Mode	Current implementation
One voice or instrument at a time (singing, humming, a bass line)	Monophonic	Pitchy (McLeod pitch method, no model download)
Chords, piano, guitar	Polyphonic	Basic Pitch
A drum recording	Drums	Inverse Drum Machine (server only)

Show the model's label and description from settings.models instead of hard-coding names; the list can change without a change to your module.

Live pitch to MIDI

For a performer singing or playing into the module, track pitch in the Processor itself. The Pitchy module does this with @maruaudio/dsp/analysis/pitch-tracking: a pure DSP tracker that runs in the browser, native plug-ins and hardware alike, with no host task involved. Read its source with:

maru module files pitchy
maru module read pitchy src/dsp/processor.ts

From an app or server

Apps submit transcribe_audio_to_midi with mode monophonic or polyphonic, and drum transcription with mode: "drums". See Add generative audio to your app.

Limits and availability

Clips are limited to 30 seconds on the device path; the server path processes longer clips in 30-second windows.
An empty result (no_notes) is a valid outcome, not an error.
Clip transcription runs on website module pages. Studio and native plug-ins do not offer it yet; the live pitch tracker works everywhere.

Ask your agent

text

Read the Maru docs page "Audio to MIDI in a module", then read the Pitchy
module's source with `maru module files pitchy` and `maru module read`.
Add a monophonic pitch-to-MIDI input to my module, and keep its existing
controls working when no transcription host is present.

Documentation