# Run an audio or MIDI model experiment

> Choose a reference workflow, preserve its inputs and compare training or serving changes.

Canonical: https://maru.audio/docs/audio-model-experiments
Documentation index: https://maru.audio/llms.txt

Choose one output and one reference implementation before building an
integration. This guide connects the Learn projects to external developer tools
and a reusable experiment record. The workflows were source-reviewed on
October 26, 2026; training, inference and hardware measurements were not run
for this guide.

> **In the Help Center:** [Generative features](https://maru.audio/help/generative-features.md)
> explains choosing audio, MIDI or descriptions and which actions Maru offers.

## Pick an entry point

| Result | Reference workflow | First useful artifact | Setup boundary |
| --- | --- | --- | --- |
| Sound tags or retrieval | [AST](https://huggingface.co/docs/transformers/en/model_doc/audio-spectrogram-transformer) and [CLAP](https://huggingface.co/docs/transformers/en/model_doc/clap) examples | Predictions on a fixed clip set | Python, compatible decoder and model downloads; use each checkpoint's processor |
| A personal sample family | [SA3 adapter workflow](https://github.com/Stability-AI/stable-audio-3/blob/main/docs/workflows/lora.md) | Base/adapter WAV comparison | Upstream repository, captioned audio and supported training accelerator; inference support alone does not establish training support |
| Codec-based audio generation | [MusicGen author guide](https://facebookresearch.github.io/audiocraft/docs/MUSICGEN.html) and its linked [browser demo](https://huggingface.co/spaces/facebook/MusicGen) | A generated clip, then a separate fine-tuning plan | Demo service access or the documented AudioCraft environment; do not mix its dependency requirements into another trainer |
| Editable musical events | [MidiTok conversion examples](https://miditok.readthedocs.io/en/latest/examples.html) | A decoded MIDI round trip | Python and MidiTok; no pretrained composer is needed for conversion |
| Audio questions and answers | [Qwen3-Omni reference](https://huggingface.co/docs/transformers/en/model_doc/qwen3_omni_moe) | Raw text answers to fixed questions | Exact processor/model pairing and the reference's hardware requirements |
| Captioning adaptation | [Audio-text-to-text training tutorial](https://huggingface.co/docs/transformers/en/tasks/audio_text_to_text) | Base/adapter captions on held-out recordings | Model-specific Audio Flamingo 3 recipe; do not transfer its target-layer configuration to Qwen by assumption |
| Common multimodal serving | [vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni/en/latest/) | Same-checkpoint reference/server comparison | Supported model, stage configuration and runtime/hardware combination |

Start with the upstream installation instructions in a separate experiment
environment. Record the repository revision, Python and package versions,
accelerator/runtime versions and exact checkpoint revision. Moving `main` or
`latest` documentation is an entry point, not a reproducible environment pin.
Check code, weights, training data and output terms for your intended use.
No account upload is required merely to keep a local experiment record.

## Choose what should learn

| Method | Trainable part | Comparison needed |
| --- | --- | --- |
| Frozen features plus classifier | A small head over existing embeddings | Retrieval or unadapted classifier versus the new labels |
| Adapter or LoRA | Selected additions to a compatible base | Unchanged base versus adapter checkpoints, with the same input suite |
| Full fine-tuning | Many or all pretrained parameters | Adapter baseline plus forgetting checks on the base model's tasks |
| From scratch | A selected new model, such as a small MIDI composer | Untrained and simple musical baselines, plus held-out sequences |

[PEFT](https://huggingface.co/docs/peft/en/index) supplies adaptation tooling;
it does not establish that a model's modules, targets and saved format are
compatible. [Accelerate](https://huggingface.co/docs/accelerate/en/index)
helps manage supported execution configurations. Establish one working batch
before distributing the run.

## Check one MIDI file before training

Following the documented MidiTok conversion API, this minimal example encodes
an existing file and writes a decoded copy. Run it in your external environment
with an owned `phrase.mid`; preserve any existing `round-trip.mid` first. The
snippet is source-reviewed, not locally execution-verified.

```python
from pathlib import Path
from miditok import REMI

events = REMI()
encoded = events(Path("phrase.mid"))
events(encoded).dump_midi(Path("round-trip.mid"))
```

Compare the two files in your MIDI editor, including tempo, note starts,
durations, velocities and track behavior. Defaults are a teaching start;
freeze a deliberate tokenizer configuration for training. Tokenizer learning
alone does not train a composer.

For the next stage, use [MidiTok's PyTorch data guide](https://miditok.readthedocs.io/en/latest/pytorch_data.html)
and a small causal-model implementation. The
[causal training tutorial](https://huggingface.co/docs/transformers/en/tasks/language_modeling)
explains the objective; its text preprocessing must be replaced with musical
event batches and a matching vocabulary. Before a longer run, verify:

- **Input and labels:** a prefix predicts its next event. Some model classes
  shift labels internally; shift exactly once across your collator/model/loss.
- **Padding:** attention masks exclude padding and padded labels are ignored
  by the loss. BOS, EOS and PAD have distinct intended roles.
- **Vocabulary:** embedding/output dimensions, event IDs and special tokens
  agree with the saved tokenizer. A pretrained musical checkpoint keeps its
  original tokenizer unless its model is deliberately adapted too.
- **Decoding:** generated IDs are valid for the event grammar. Log rejected or
  repaired generations, rather than silently dropping failures from evaluation.

Whole compositions and related arrangements stay in one source split. Make
crops, overlapping windows and transpositions afterward. Fit learned tokenizer
compression only on training data. Select hyperparameters on validation, then
evaluate the selected checkpoint on an untouched test split.

## Preserve an experiment record

Copy this Markdown into your experiment folder and fill in actual values.
It is a lab note template, not a Maru manifest or API schema.

```markdown
# Experiment
Question and proposed success criterion:
Reference implementation and source revision:
Environment lock/export and accelerator:
Base checkpoint revision/hash:
Adapter or trained checkpoint identity:
Processor/tokenizer configuration and hash:
Dataset manifest, source groups and permissions record:
Train / validation / final-test assignments:
Prompts or phrase prefixes, seeds and generation settings:
Commands and checkpoint-selection rule:
Raw output paths, failures and any post-processing:
Measurements, listening review and unresolved limits:
Decision and next experiment:
```

For audio generation, preserve original WAVs and separate level-matched
listening copies. Record silence, clipping, onset/tail behavior, duplication,
useful variation and listening judgments separately. A matching seed helps a
within-model comparison; it does not make different architectures generate
equivalent content or ensure byte-identical results across runtimes.

For MIDI, retain invalid outputs, raw and edited files, event statistics and
auditions through the same sound source. For captions, record correct, missed
and unsupported claims against independently written references. Compare a
specialist and a language model only on a shared question, not by treating
their raw scores as interchangeable.

## Compare shared serving after reference inference

Use the same checkpoint, processor, clip bytes and output settings in the
reference runtime and candidate server. Measure model loading separately from
warm request execution. Record time to first usable output, completion time,
peak memory, concurrency and failures; include request count and measurement
method beside every summary.

Test cancellation, one failed stage, a queued request and recovery without
restarting unrelated work. One endpoint may coordinate several processes or
accelerators. A smaller active-expert count does not mean only those weights
need to be stored. Shared serving and model alignment are separate decisions.

## Bring a useful result into Maru

Keep training and inference outside the realtime Processor. Integrate through
the existing [AI task contracts](https://maru.audio/docs/ai-tasks.md), returning ordinary
PCM assets, musical events or display-safe results through their supported
host paths. An external checkpoint needs a compatible implementation; these
exercises do not add a general model-upload or audio-chat capability to Maru.

Use the matching Learn project for the complete comparison:
[sound taggers](https://maru.audio/learn/compare-audio-classifiers.md),
[sample adaptation](https://maru.audio/learn/adapt-a-sample-generator.md),
[MIDI composition](https://maru.audio/learn/train-a-midi-model.md) or
[audio-language answers](https://maru.audio/learn/test-an-audio-language-model.md).