Implement an ML task engine docs

A module asks for an AI task by capability and never names a model. This page is for the other side: the people who write the engines behind a task and the hosts that run them. You need it to add a new transcription engine, ship a model a host can verify, or offer a task in your own host. To use a task in a module, start with AI tasks in your module instead.

The four contracts

Tasks are built from four pieces, each with a single owner, so a module, an engine and a host can be written by different people and still fit together. Every task has a row in the task table and a face surface. A task whose implementations you can add also has an engine contract and model sources. Audio to MIDI uses all four.

Contract	Entry point	Says
Task table	`@maruaudio/audio-contracts/ml-task-contracts`	For each task and mode: the capability a module declares, the face prop the host fills, the input the host prepares, the output and how it reaches the module
Face surface	`@maruaudio/audio-contracts/module-face-transcription-live` (live), `@maruaudio/audio-contracts/module-face-transcription` (clip)	The display-safe state and intents a face sees, and the bounded messages that carry them into a face in another frame
Engine	`@maruaudio/gen-sdk/transcription/engine` (`maru-transcription-engine/v1`)	What an implementation provides: its descriptor, clip transcription, live transcription or both
Model source	`@maruaudio/audio-contracts/model-source` (`maru-model-source/v1`)	Which files a model is made of, pinned by size and SHA-256, and where a host may fetch them

A module author touches only the first two. An engine author touches only the last two. A host connects them.

The task table is plain data. ML_TASK_CONTRACTS lists every task, and mlTaskForCapability("audio.transcription.live/v1") returns the task, mode and contract a capability belongs to. For audio to MIDI it reads:

Mode	Capability	Face prop	Input	Delivery
Clip	`audio.transcription/v1`, with the `audio.transcription.pitch/v1` profile	`transcription`	Mono clip, up to 30 seconds per run	A timeline resource, and the face
Live	`audio.transcription.live/v1`	`liveTranscription`	Mono stream from the module's input or a microphone	Notes on the Processor's note input, and the face

Both modes use the same engine contract.

Write a transcription engine

An engine is plain TypeScript. It never touches the page, the audio thread or the network: the host hands it audio, verified model bytes and an inference function, and it hands back notes. It exports a TranscriptionEngineV1:

interface TranscriptionEngineV1 {
  readonly descriptor: TranscriptionEngineDescriptorV1
  transcribeClip?(input: ClipTranscriptionInputV1): Promise<{ readonly notes: readonly TranscriptionNoteV1[] }>
  createLive?(context: LiveTranscriptionContextV1): Promise<LiveTranscriberV1> | LiveTranscriberV1
}

Implement transcribeClip for clip mode, createLive for live mode, or both. Maru's own engines are references for each case: the Pitchy engine (@maruaudio/gen-sdk/transcription/engines/pitchy) runs clips and live input with no model, and the Basic Pitch engine (@maruaudio/gen-sdk/transcription/engines/basic-pitch) runs clips through an ONNX model.

The descriptor

The descriptor is how a host decides whether, and how, to offer your engine.

Field	Meaning
`contract`	Always `TRANSCRIPTION_ENGINE_CONTRACT` (`"maru-transcription-engine/v1"`)
`id`	A stable lowercase id, such as `pitchy` or `basic-pitch/nmp-onnx`. Hosts save it with results and settings
`label`, `description`	What the person sees in the engine picker. Say what the engine is good at and whether it downloads a model
`profile`	`pitch` for pitched notes, `drums` for drum hits
`polyphony`	`mono` when the engine follows one note at a time, `poly` when it reports chords
`modes`	`clip`, `live` or both
`sampleRate`	The rate your engine wants. The host resamples to it; you never receive another rate
`clip.maxSeconds`	Required for clip mode: the longest clip one run accepts
`live.blockFrames`	Required for live mode: the frames in every `push`
`live.latencyFrames`	Required for live mode: the fixed delay, in input frames, from the audio a note is heard in to the frame of its event
`parameters`	Tuning controls: `id`, `label`, `default`, `min`, `max`, `step`, optional `unit` and `description`
`model`	A `maru-model-source/v1` declaration, or absent for a model-free engine
`license`	The licence of your code or algorithm. The model's licence lives in `model.license`

Hosts check an engine with transcriptionEngineErrors(engine) before they offer it. It validates the descriptor (unique parameter ids, finite and ordered ranges, bounded labels and descriptions, at most 32 parameters) and that every declared mode has its method: transcribeClip for clip, createLive for live. Live hosts use liveTranscriptionEngineErrors(engine), which adds the requirement that the engine offers live mode. A host quietly leaves out an engine that fails, so run the same checks in your own tests. Validate the model source with modelSourceErrors(descriptor.model) as well, and keep parameter ids unique: the face carries at most 32 parameters and 16 engines.

Clip mode

transcribeClip receives one finite mono clip, already decoded and resampled: audio (a Float32Array at descriptor.sampleRate), settings (a value for each parameter id), model when you declared one, an optional signal to honour cancellation, and an optional onProgress. It returns notes with startSeconds, durationSeconds, midi and velocity from 0 to 1.

import type {
  ClipTranscriptionInputV1,
  TranscriptionNoteV1,
} from "@maruaudio/gen-sdk/transcription/engine"

const SAMPLE_RATE = 16000
const WINDOW = 512
/** Your decoder: probabilities for one window → notes, in seconds from the clip start. */
declare function decodeNotes(probabilities: Float32Array, offsetSeconds: number, threshold: number): TranscriptionNoteV1[]

async function transcribeClip(input: ClipTranscriptionInputV1): Promise<{ readonly notes: readonly TranscriptionNoteV1[] }> {
  if (input.sampleRate !== SAMPLE_RATE) throw new RangeError("Onset Net input must be resampled to 16 kHz")
  const infer = input.model?.infer
  if (!infer) throw new TypeError("Onset Net needs the host's infer")
  const threshold = input.settings.threshold ?? 0.5
  const total = Math.ceil(input.audio.length / WINDOW)
  const notes: TranscriptionNoteV1[] = []
  for (let index = 0; index < total; index++) {
    input.signal?.throwIfAborted()
    const window = new Float32Array(WINDOW) // zero-padded at the end of the clip
    window.set(input.audio.subarray(index * WINDOW, (index + 1) * WINDOW))
    const out = await infer({ audio: { data: window, dims: [1, WINDOW] } })
    notes.push(...decodeNotes(out.notes!.data, (index * WINDOW) / SAMPLE_RATE, threshold))
    input.onProgress?.({ completed: index + 1, total })
  }
  return { notes }
}

An empty result is a valid outcome. Throw when the input breaks your contract, such as the wrong rate or an unknown parameter id.

Live mode

createLive(context) returns a live transcriber with four methods:

push(block, startFrame) receives a mono block of exactly live.blockFrames frames at your rate. startFrame counts input frames from the start of the stream. The host reuses the buffer, so copy what you keep.
setParameter(id, value) changes a parameter while running. The host has already clamped the value to min and max and snapped it to step. Throw for an id you do not have.
reset(frame) releases every held note, with its note-off at frame, and clears your state. The host calls it when it stops the engine and when the input skips by more than a second.
close() stops any further work.

Report notes through context.emit({ type, midi, velocity, frame }). The frame is in the same domain as startFrame, with your latency already added, so the host can place each note exactly where it was played.

Choosing a latency. latencyFrames is a promise: a note heard at frame n is reported at frame n + latencyFrames. Set it to the look-ahead your analysis needs plus the inference time you expect on a typical device. Pitchy's is 3,456 frames at 22,050 Hz (about 157 ms): nine 256-frame hops to cover its default 100 ms minimum note length, plus 1,152 frames of look-ahead for the analysis window. When a decision comes later than promised, stamp it at the latest frame you have received instead, so no event lands before the audio that decided it, and keep event frames in order. The host shows the latency, plus the measured inference time, to the person.

One inference in flight. A live model must never queue work. If a device is slower than the stream, a queue makes every result later than the last. Wrap your model call in createLatestFrameScheduler from @maruaudio/audio-sdk/model-runtime/latest-frame: it keeps one run in flight and at most one frame waiting, and a newer frame replaces the waiting one. Its stats() feed context.onHealth({ inferredRatio, meanAgeMs }), which the face shows as health; call it a few times a second at most. Report one failed inference with context.onError(error); the session keeps running. An exception from push or setParameter, or a malformed event, stops the session with an error.

Here is a complete model-backed live engine. It infers once per block, keeps one inference in flight and turns note probabilities into note events:

import {
  TRANSCRIPTION_ENGINE_CONTRACT,
  type LiveTranscriberV1,
  type LiveTranscriptionContextV1,
  type TranscriptionEngineDescriptorV1,
  type TranscriptionEngineV1,
} from "@maruaudio/gen-sdk/transcription/engine"
import { createLatestFrameScheduler } from "@maruaudio/audio-sdk/model-runtime/latest-frame"

const SAMPLE_RATE = 16000
/** One analysis frame per push. */
const BLOCK_FRAMES = 512
/** Covers the frame itself plus the inference time seen on target devices. */
const LATENCY_FRAMES = 2048
const LOWEST_NOTE = 40
const NOTE_COUNT = 49

export const onsetNetDescriptor: TranscriptionEngineDescriptorV1 = {
  contract: TRANSCRIPTION_ENGINE_CONTRACT,
  id: "example/onset-net",
  label: "Onset Net",
  description: "Polyphonic notes from a played instrument. Downloads a 1.2 MB model.",
  profile: "pitch",
  polyphony: "poly",
  modes: ["live"],
  sampleRate: SAMPLE_RATE,
  live: { blockFrames: BLOCK_FRAMES, latencyFrames: LATENCY_FRAMES },
  parameters: [
    { id: "threshold", label: "Note threshold", default: 0.5, min: 0.05, max: 0.95, step: 0.01 },
  ],
  model: {
    format: "maru-model-source/v1",
    id: "example/onset-net@1.0.0",
    label: "Onset Net (ONNX)",
    license: { name: "Apache-2.0", spdx: "Apache-2.0", commercialUse: "allowed" },
    files: {
      model: {
        path: "onset-net.onnx",
        bytes: 1213440,
        sha256: "0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0",
      },
    },
    locations: [
      { kind: "module-asset", dir: "models/onset-net" },
      { kind: "https", base: "https://models.example.com/onset-net/1.0.0/" },
    ],
  },
  license: "MIT",
}

interface Frame {
  readonly audio: Float32Array
  endFrame: number
}

function createLive(context: LiveTranscriptionContextV1): LiveTranscriberV1 {
  const infer = context.model?.infer
  if (!infer) throw new TypeError("Onset Net needs the host's infer")
  let threshold = context.settings.threshold ?? 0.5
  const held = new Uint8Array(NOTE_COUNT)
  // One frame running, one waiting and one being filled.
  const pool: Frame[] = [0, 1, 2].map(() => ({ audio: new Float32Array(BLOCK_FRAMES), endFrame: 0 }))
  let running: Frame | null = null
  let waiting: Frame | null = null
  let streamEnd = 0
  let lastFrame = Number.NEGATIVE_INFINITY
  let lastHealth = 0

  const emit = (type: "on" | "off", midi: number, velocity: number, frame: number) => {
    lastFrame = Math.max(lastFrame, frame) // events never go back in time
    context.emit({ type, midi, velocity, frame: lastFrame })
  }

  const makeScheduler = () => {
    const scheduler = createLatestFrameScheduler<Frame, Float32Array>({
      run: async (frame) => {
        running = frame
        if (waiting === frame) waiting = null
        const out = await infer({ audio: { data: frame.audio, dims: [1, BLOCK_FRAMES] } })
        const probabilities = out.notes?.data
        if (!probabilities || probabilities.length !== NOTE_COUNT)
          throw new Error(`Onset Net must return ${NOTE_COUNT} note probabilities`)
        return probabilities
      },
      onResult: (probabilities, info) => {
        running = null
        // A late result lands where the stream is now, never before the audio that decided it.
        const frame = Math.max(info.frame.endFrame + LATENCY_FRAMES, streamEnd)
        for (let i = 0; i < NOTE_COUNT; i++) {
          const on = probabilities[i]! >= threshold
          if (on && !held[i]) {
            held[i] = 1
            emit("on", LOWEST_NOTE + i, probabilities[i]!, frame)
          } else if (!on && held[i]) {
            held[i] = 0
            emit("off", LOWEST_NOTE + i, 0, frame)
          }
        }
        if (context.onHealth && info.completedAt - lastHealth >= 250) {
          lastHealth = info.completedAt
          const stats = scheduler.stats()
          context.onHealth({
            inferredRatio: stats.submitted ? stats.completed / stats.submitted : 1,
            meanAgeMs: stats.meanAgeMs,
          })
        }
      },
      onError: (error) => {
        running = null
        context.onError?.(error)
      },
    })
    return scheduler
  }
  let scheduler = makeScheduler()

  return {
    push(block, startFrame) {
      if (scheduler.closed) return
      streamEnd = startFrame + block.length
      const frame = pool.find((candidate) => candidate !== running && candidate !== waiting)!
      frame.audio.set(block) // copy: the host reuses `block`
      frame.endFrame = streamEnd
      waiting = frame
      scheduler.submit(frame) // replaces an older waiting frame
    },
    setParameter(id, value) {
      if (id !== "threshold") throw new RangeError(`Onset Net has no parameter ${id}`)
      threshold = value
    },
    reset(frame) {
      for (let i = 0; i < NOTE_COUNT; i++) if (held[i]) emit("off", LOWEST_NOTE + i, 0, frame)
      held.fill(0)
      scheduler.close() // ignores the result still in flight
      // The abandoned run may still read its buffer; give the pool a fresh one.
      if (running) pool[pool.indexOf(running)] = { audio: new Float32Array(BLOCK_FRAMES), endFrame: 0 }
      running = waiting = null
      scheduler = makeScheduler()
    },
    close() {
      scheduler.close()
    },
  }
}

export const onsetNetEngine: TranscriptionEngineV1 = { descriptor: onsetNetDescriptor, createLive }

To offer the same engine for clips, replace the export with one that adds clip mode to the descriptor and the transcribeClip above:

export const onsetNetEngine: TranscriptionEngineV1 = {
  descriptor: { ...onsetNetDescriptor, modes: ["clip", "live"], clip: { maxSeconds: 30 } },
  transcribeClip,
  createLive,
}

Declare a model

Declare the files your engine needs with maru-model-source/v1, in descriptor.model. The declaration separates what the model is from where its bytes live, so the same engine works whether Maru hosts the files, your module ships them, or you serve them yourself.

model: {
  format: "maru-model-source/v1",
  id: "example/onset-net@1.0.0",
  label: "Onset Net (ONNX)",
  license: { name: "Apache-2.0", spdx: "Apache-2.0", commercialUse: "allowed" },
  files: {
    model: {
      path: "onset-net.onnx",
      bytes: 1213440,
      sha256: "0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0",
    },
  },
  locations: [
    { kind: "module-asset", dir: "models/onset-net" },
    { kind: "https", base: "https://models.example.com/onset-net/1.0.0/" },
  ],
},

Identity. id is lowercase vendor/name, with an optional @version. Give a new version a new id when its files change.

Files. Each file has a role key, such as model for a single-graph model, license or notice: lowercase letters, digits, _ and -, starting with a letter. Its path is relative to the location (no leading /, no .., \, %, ? or #), with its exact bytes and lowercase hex sha256. A model holds 1 to 32 files. Your engine receives the verified bytes under the same keys, in context.model.files.

Locations. A host walks the list in order and uses the first kind it supports:

Kind	Fields	Where the bytes come from
`maru-ledger`	`set`	A model set Maru hosts. The host fetches it through its own model route, never a path you supply
`module-asset`	`dir`	A directory inside your module's release, so the model works offline and in hosts without network access
`https`	`base`	Your own server. The base must be an `https` URL ending in `/`, without credentials, query or fragment, and the server must allow cross-origin requests from the host. A host fetches from it only if it allows author-hosted models

Hosts that cannot reach one location move on to the next. A location that answers with different files is a refusal, not a reason to try the next one.

Licence. license.name is shown to the person beside the engine. commercialUse says what the weights' terms allow: allowed, non-commercial or unknown. Treat unknown as non-commercial, and only declare allowed when the model's own licence file says so. spdx and an https url are optional.

Integrity. The host checks every file's size and SHA-256 before your engine sees it. A file that does not match is refused, the run stops with an error, and the host never falls back to the unverified bytes. A model can be re-hosted anywhere without changing your engine, and a corrupted or swapped file cannot reach it.

Maru-hosted models are declared from Maru's model ledger: gen-sdk generates each declaration from its ledger entry (pnpm --filter @maruaudio/gen-sdk model-sources:generate), and a test fails while the generated file is stale, so an engine and the hosted files cannot disagree. The Basic Pitch engine's BASIC_PITCH_MODEL_SOURCE is one, with a single maru-ledger location.

Keep heavy dependencies optional

A host should load only what it runs. The engine contract entry, @maruaudio/gen-sdk/transcription/engine, imports no runtime, and each of Maru's engines is its own entry point. A host that offers only Pitchy never loads ONNX Runtime.

Follow the same pattern in your engine:

Take inference from the host. context.model.infer runs your model in the host's model worker, one request at a time, with named float tensors in and out. It also lets the host choose WebGPU or WebAssembly.
If your engine must run its own runtime when a host offers no infer, import it with a dynamic import() on first use, as the Basic Pitch engine loads onnxruntime-web. Importing the engine for its descriptor then costs nothing.

Run engines in a host

@maruaudio/gen-sdk/transcription/live-session does the host work for live transcription in a browser host. createLiveTranscriptionSurface builds the face.liveTranscription surface from your engines and owns its status machine, engine switching and note subscriptions. Your host supplies the input, the model files and the note delivery:

import {
  createLiveTranscriptionSurface,
  transcriptionInferFromHostSession,
  type LiveTranscriptionDeliveryV1,
} from "@maruaudio/gen-sdk/transcription/live-session"
import { pitchyEngine } from "@maruaudio/gen-sdk/transcription/engines/pitchy"
import {
  fetchVerifiedModelFiles,
  resolveModelSource,
} from "@maruaudio/audio-sdk/model-runtime/model-source"
import type { HostModelFilePlan, HostModelInference } from "@maruaudio/audio-sdk/web-prototype-host"
import { onsetNetEngine } from "./onset-net"

// What your host already has.
declare const context: AudioContext
declare const moduleInput: AudioNode
declare const modelInference: HostModelInference
declare const captureWorkletUrl: string
declare function ledgerPlan(set: string): Promise<HostModelFilePlan>
declare function moduleAssetUrl(dir: string, path: string): string
/** Schedules a note on the module Processor's note input at a context time. */
declare const processorNoteInput: LiveTranscriptionDeliveryV1
declare function publishToFace(): void

export const liveTranscription = createLiveTranscriptionSurface({
  engines: [pitchyEngine, onsetNetEngine],
  inputs: [
    { id: "module-input", label: "Module input" },
    { id: "microphone", label: "Microphone" },
  ],
  async acquireInput({ inputId, prompting }) {
    if (inputId === "module-input") return { context, input: moduleInput }
    const permission = await navigator.permissions.query({ name: "microphone" as PermissionName })
    if (permission.state === "prompt") prompting()
    // A refusal rejects with NotAllowedError, and the status becomes "denied".
    const stream = await navigator.mediaDevices.getUserMedia({
      audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false },
    })
    return {
      context,
      input: context.createMediaStreamSource(stream),
      release: () => stream.getTracks().forEach((track) => track.stop()),
    }
  },
  async loadModel(engine, signal) {
    const resolved = await resolveModelSource(engine.descriptor.model!, { ledger: ledgerPlan, moduleAssetUrl }, { signal })
    const files = await fetchVerifiedModelFiles(resolved, { signal }) // refuses a wrong size or sha256
    const session = await modelInference.createSession(new Uint8Array(files.model!), ["webgpu", "wasm"])
    return { files, infer: transcriptionInferFromHostSession(session) }
  },
  deliver: processorNoteInput,
  session: { captureWorkletUrl },
  onChange: () => publishToFace(),
})
// Hand `liveTranscription` to the face as `face.liveTranscription`.

Option	Purpose
`engines`	The engines to offer, in order. Engines that fail `liveTranscriptionEngineErrors`, repeat an id or exceed 16 are left out; with none left, the status is `unavailable`
`inputs`, `inputId`	The inputs the face may choose between. Omit when there is only one
`acquireInput({ inputId, signal, prompting })`	Open the input on every start and return its `AudioContext`, an `AudioNode` and an optional `release`. Call `prompting()` when the browser is about to ask for permission. Reject with `NotAllowedError` or `SecurityError` on a refusal: the status becomes `denied`
`loadModel(engine, signal)`	Return verified `files`, an `infer` and an optional `release` for an engine with a model. It runs on every start; the surface calls `release` after the engine closes on stop, engine switch, failure and dispose
`deliver`	`noteOn(midi, velocity, whenSeconds)` and `noteOff(midi, whenSeconds)` on the `AudioContext` clock. Schedule each note on the module Processor's note input
`settings`	Saved parameter values per engine id
`session`	Fixed options for every run, such as `captureWorkletUrl` (the host-served capture worklet) and `snapshotIntervalMs` (50 by default)
`onChange`	Called with every published state, to re-render or post the snapshot to the face

The handle adds settled(), which resolves when queued start and stop work has finished, and dispose() for when the module goes away.

For each run, createLiveTranscriptionSession captures the input with createStreamCapture in whole render quanta at the context rate, resamples it to the engine rate and re-blocks it to live.blockFrames. Time comes from the capture's frame stamps, never from when a message arrives. A skip of up to a second is filled with silence so the clock never drifts, and a longer one resets the engine. Each event frame becomes a context time, with the engine's latency included, before deliver sees it. A second note-on for a held note sends its note-off first, and a note-off for a note that is not held is dropped. Use the session on its own for one engine on one input without a face.

resolveModelSource and fetchVerifiedModelFiles come from @maruaudio/audio-sdk/model-runtime/model-source. Supply the resolvers your host has: ledger for Maru-hosted sets, moduleAssetUrl for files in the module's release, and allowHttps: true only if you accept author-hosted models. transcriptionInferFromHostSession adapts a host inference session to the engine's infer.

A face that runs in an isolated frame cannot receive the surface itself. Post createAudioModuleFaceLiveTranscriptionSnapshot(surface) as a maru.liveTranscriptionState message, and decode the face's maru.liveTranscriptionIntent messages with decodeAudioModuleFaceLiveTranscriptionIntentMessage before you act on them. Both are bounded and checked on each side. On the face's side, createAudioModuleFaceLiveTranscriptionBridgeSurface rebuilds the surface without subscribeNotes.

Clip engines need less: decode and resample the clip, load the model the same way, call transcribeClip, and hand the notes to the clip task's face and timeline.

Where engines run

An engine runs in any host that implements its task's delivery. Each host's entry in Maru's host table lists the capabilities it carries, and a host fills face.liveTranscription only when its entry lists audio.transcription.live/v1. Browser hosts run live engines through the session above, beside the audio path. A native plug-in offers the task only when it delivers notes into the Processor the same way. Modules declare the capability as optional, so they keep working in every host without it.

Ask your agent

text

Read the Maru docs page "Implement an ML task engine", then read
packages/gen-sdk/src/transcription/engine.ts and engines/pitchy.ts. Write a
maru-transcription-engine/v1 live engine around my ONNX model: declare its
files with maru-model-source/v1 and real SHA-256 hashes, keep one inference in
flight with createLatestFrameScheduler, choose latencyFrames from my model's
look-ahead, and add tests that run transcriptionEngineDescriptorErrors and
modelSourceErrors on the descriptor.

Documentation