# Audio to MIDI in a module

> Turn a recorded clip or a live input into MIDI notes: the host transcription task for clips, the realtime pitch tracker for live playing, and how the notes reach your Processor.

Canonical: https://maru.audio/docs/audio-to-midi-in-a-module
Documentation index: https://maru.audio/llms.txt

Audio to MIDI turns a performance into notes you can edit, quantise and send to
any instrument. It is part of [AI tasks in your module](https://maru.audio/docs/ai-tasks.md).
For what the task is good at and where it fails, read
[Turn audio into MIDI](https://maru.audio/help/turn-audio-into-midi.md) in the Help Center.

There are two shapes of this task:

| You want | Use | Runs |
| --- | --- | --- |
| Notes from a recorded clip, after the fact | `audio.transcription/v1` (`face.transcription`) | In the host's worker, outside the audio callback |
| Notes from a live input, while it plays | The `@maruaudio/dsp/analysis/pitch-tracking` leaf in your Processor | Inside your Processor, on every target |

## Clip transcription

### 1. Declare the task and a timeline

The host transcribes; your Processor plays the result back as MIDI. Declare the
capability, a MIDI output port and a data resource the host fills with the note
timeline:

```ts
ports: [{ id: "midi-out", label: "MIDI", direction: "output", kind: "midi" }],
implementation: {
  kind: "processor",
  source: "src/dsp/processor.ts",
  maxEventOutputs: 256,
  resources: [
    { id: "timeline", kind: "data", elementType: "f32", maxElements: 500003, loading: "preload" },
  ],
},
ui: {
  entry: "./ui/index.tsx",
  hostCapabilities: { optional: ["audio.transcription/v1"] },
},
```

The Processor can delegate to the shared `@maruaudio/dsp/processors/midi-timeline`
leaf, which schedules the notes on the sample clock. PitchRead's
`src/dsp/processor.ts` is that leaf and nothing else.

### 2. Drive it from the face

`face.transcription` is display-safe state plus intent methods:

```tsx
const transcription = face.transcription
if (!transcription) return <p>Transcription is not available here.</p>

const { phase, source, notes, progress } = transcription.state
// phase: "idle" | "loading" | "transcribing" | "ready" | "error"

<Button onClick={transcription.chooseSource}>Choose audio</Button>
<Button disabled={!source || phase === "transcribing"} onClick={transcription.transcribe}>
  Extract notes
</Button>
```

| Member | Purpose |
| --- | --- |
| `chooseSource()`, `chooseExample?(id)`, `examples` | Pick a file or a host-supplied example clip |
| `recording?` | Optional bounded capture from the host's audio input: `start`, `stop`, `cancel` |
| `transcribe()`, `cancel()` | Run or stop extraction |
| `state.notes` | `startSeconds`, `durationSeconds`, `midi`, `velocity` (0 to 1) for each note |
| `state.source` | Label, duration, peaks, optional waveform and spectrogram for display |
| `settings?` | The models the host offers (`models`, `setModel`) and their tuning parameters (`parameters`, `setParameter`) |
| `sourcePlayback?`, `audition?` | Play the original, or blend original and MIDI audition |
| `createMidiFile()` | Host-encoded MIDI bytes for download or drag |

`settings.changedSinceResult` tells you the draft settings differ from the
result on screen; changes apply to the next extraction, never to an existing
result.

### Choosing a mode

The host lists its implementations in `settings.models`. Today they cover three
kinds of source:

| Source | Mode | Current implementation |
| --- | --- | --- |
| One voice or instrument at a time (singing, humming, a bass line) | Monophonic | Pitchy (McLeod pitch method, no model download) |
| Chords, piano, guitar | Polyphonic | [Basic Pitch](https://maru.audio/models/basic-pitch) |
| A drum recording | Drums | Inverse Drum Machine (server only) |

Show the model's label and description from `settings.models` instead of
hard-coding names; the list can change without a change to your module.

## Live pitch to MIDI

For a performer singing or playing into the module, track pitch in the
Processor itself. The Pitchy module does this with
`@maruaudio/dsp/analysis/pitch-tracking`: a pure DSP tracker that runs in the
browser, native plug-ins and hardware alike, with no host task involved. Read
its source with:

```sh
maru module files pitchy
maru module read pitchy src/dsp/processor.ts
```

## From an app or server

Apps submit `transcribe_audio_to_midi` with `mode` `monophonic` or
`polyphonic`, and drum transcription with `mode: "drums"`. See
[Add generative audio to your app](https://maru.audio/docs/generative-features.md#supply-audio-to-a-task).

## Limits and availability

- Clips are limited to 30 seconds on the device path; the server path processes
  longer clips in 30-second windows.
- An empty result (`no_notes`) is a valid outcome, not an error.
- Clip transcription runs on website module pages. Studio and native plug-ins
  do not offer it yet; the live pitch tracker works everywhere.

## Ask your agent

```text
Read the Maru docs page "Audio to MIDI in a module", then read the Pitchy
module's source with `maru module files pitchy` and `maru module read`.
Add a monophonic pitch-to-MIDI input to my module, and keep its existing
controls working when no transcription host is present.
```