Run an audio or MIDI model experiment

Choose one output and one reference implementation before building an integration. This guide connects the Learn projects to external developer tools and a reusable experiment record. The workflows were source-reviewed on October 26, 2026; training, inference and hardware measurements were not run for this guide.

In the Help Center: Generative features explains choosing audio, MIDI or descriptions and which actions Maru offers.

Pick an entry point

Result	Reference workflow	First useful artifact	Setup boundary
Sound tags or retrieval	AST and CLAP examples	Predictions on a fixed clip set	Python, compatible decoder and model downloads; use each checkpoint's processor
A personal sample family	SA3 adapter workflow	Base/adapter WAV comparison	Upstream repository, captioned audio and supported training accelerator; inference support alone does not establish training support
Codec-based audio generation	MusicGen author guide and its linked browser demo	A generated clip, then a separate fine-tuning plan	Demo service access or the documented AudioCraft environment; do not mix its dependency requirements into another trainer
Editable musical events	MidiTok conversion examples	A decoded MIDI round trip	Python and MidiTok; no pretrained composer is needed for conversion
Audio questions and answers	Qwen3-Omni reference	Raw text answers to fixed questions	Exact processor/model pairing and the reference's hardware requirements
Captioning adaptation	Audio-text-to-text training tutorial	Base/adapter captions on held-out recordings	Model-specific Audio Flamingo 3 recipe; do not transfer its target-layer configuration to Qwen by assumption
Common multimodal serving	vLLM-Omni	Same-checkpoint reference/server comparison	Supported model, stage configuration and runtime/hardware combination

Start with the upstream installation instructions in a separate experiment environment. Record the repository revision, Python and package versions, accelerator/runtime versions and exact checkpoint revision. Moving main or latest documentation is an entry point, not a reproducible environment pin. Check code, weights, training data and output terms for your intended use. No account upload is required merely to keep a local experiment record.

Choose what should learn

Method	Trainable part	Comparison needed
Frozen features plus classifier	A small head over existing embeddings	Retrieval or unadapted classifier versus the new labels
Adapter or LoRA	Selected additions to a compatible base	Unchanged base versus adapter checkpoints, with the same input suite
Full fine-tuning	Many or all pretrained parameters	Adapter baseline plus forgetting checks on the base model's tasks
From scratch	A selected new model, such as a small MIDI composer	Untrained and simple musical baselines, plus held-out sequences

PEFT supplies adaptation tooling; it does not establish that a model's modules, targets and saved format are compatible. Accelerate helps manage supported execution configurations. Establish one working batch before distributing the run.

Check one MIDI file before training

Following the documented MidiTok conversion API, this minimal example encodes an existing file and writes a decoded copy. Run it in your external environment with an owned phrase.mid; preserve any existing round-trip.mid first. The snippet is source-reviewed, not locally execution-verified.

from pathlib import Path
from miditok import REMI

events = REMI()
encoded = events(Path("phrase.mid"))
events(encoded).dump_midi(Path("round-trip.mid"))

Compare the two files in your MIDI editor, including tempo, note starts, durations, velocities and track behavior. Defaults are a teaching start; freeze a deliberate tokenizer configuration for training. Tokenizer learning alone does not train a composer.

For the next stage, use MidiTok's PyTorch data guide and a small causal-model implementation. The causal training tutorial explains the objective; its text preprocessing must be replaced with musical event batches and a matching vocabulary. Before a longer run, verify:

• Input and labels: a prefix predicts its next event. Some model classes shift labels internally; shift exactly once across your collator/model/loss.
• Padding: attention masks exclude padding and padded labels are ignored by the loss. BOS, EOS and PAD have distinct intended roles.
• Vocabulary: embedding/output dimensions, event IDs and special tokens agree with the saved tokenizer. A pretrained musical checkpoint keeps its original tokenizer unless its model is deliberately adapted too.
• Decoding: generated IDs are valid for the event grammar. Log rejected or repaired generations, rather than silently dropping failures from evaluation.

Whole compositions and related arrangements stay in one source split. Make crops, overlapping windows and transpositions afterward. Fit learned tokenizer compression only on training data. Select hyperparameters on validation, then evaluate the selected checkpoint on an untouched test split.

Preserve an experiment record

Copy this Markdown into your experiment folder and fill in actual values. It is a lab note template, not a Maru manifest or API schema.

# Experiment
Question and proposed success criterion:
Reference implementation and source revision:
Environment lock/export and accelerator:
Base checkpoint revision/hash:
Adapter or trained checkpoint identity:
Processor/tokenizer configuration and hash:
Dataset manifest, source groups and permissions record:
Train / validation / final-test assignments:
Prompts or phrase prefixes, seeds and generation settings:
Commands and checkpoint-selection rule:
Raw output paths, failures and any post-processing:
Measurements, listening review and unresolved limits:
Decision and next experiment:

For audio generation, preserve original WAVs and separate level-matched listening copies. Record silence, clipping, onset/tail behavior, duplication, useful variation and listening judgments separately. A matching seed helps a within-model comparison; it does not make different architectures generate equivalent content or ensure byte-identical results across runtimes.

For MIDI, retain invalid outputs, raw and edited files, event statistics and auditions through the same sound source. For captions, record correct, missed and unsupported claims against independently written references. Compare a specialist and a language model only on a shared question, not by treating their raw scores as interchangeable.

Compare shared serving after reference inference

Use the same checkpoint, processor, clip bytes and output settings in the reference runtime and candidate server. Measure model loading separately from warm request execution. Record time to first usable output, completion time, peak memory, concurrency and failures; include request count and measurement method beside every summary.

Test cancellation, one failed stage, a queued request and recovery without restarting unrelated work. One endpoint may coordinate several processes or accelerators. A smaller active-expert count does not mean only those weights need to be stored. Shared serving and model alignment are separate decisions.

Bring a useful result into Maru

Keep training and inference outside the realtime Processor. Integrate through the existing AI task contracts, returning ordinary PCM assets, musical events or display-safe results through their supported host paths. An external checkpoint needs a compatible implementation; these exercises do not add a general model-upload or audio-chat capability to Maru.

Use the matching Learn project for the complete comparison: sound taggers, sample adaptation, MIDI composition or audio-language answers.