What an audio model learns

Here are three short recordings: a piano, a bass guitar and an electric guitar. Imagine handing each one to a model that writes down the notes it hears, the kind of tool that turns a recording into MIDI.

Make your guess before you read on. If you picked the guitar, that's a good instinct, and the reason behind it is worth knowing. It isn't that guitars are "harder" in some vague way. It's that a model has learned habits from the examples it was shown, and some sounds pull those habits in the wrong direction. Once you can hear why, you can predict a model's mistakes before it makes them.

Where do the ghost notes come from?

Every pitched note is really a stack of tones. Pluck a low E on a guitar and you hear the note itself at about 82 Hz, plus quieter copies at two, three, four times that frequency and up. Those upper tones, the harmonics, are why a guitar sounds like a guitar and not a flute. A piano has them too, but a bright, driven guitar has strong ones, sometimes nearly as loud as the note underneath.

Now picture what a note-reading model is doing. It's looking for signs of pitches. A loud harmonic an octave up looks a lot like a quieter note an octave up. Push the guitar through some drive and you give it more of exactly those signs.

Watch the upper peaks climb as you raise Amount. To your ear it's still one note, just fiercer. To a model, the picture now holds strong signs of notes that nobody played.

Here's what that failure tends to look like on the page. This is a drawing of the pattern, not a recording of one model run:

Notice where the extra notes sit. Not scattered at random, but an octave or an octave-and-a-fifth above real notes, right where the strongest harmonics live. That's the clue that the model isn't broken. It's doing exactly what it learned, on a sound that tricks it.

What did the model learn, and when?

A model gets its habits in two very different moments.

Training happens once, long before you meet the model. It's shown thousands of examples: recordings paired with the notes that were actually played. Each time it guesses wrong, its internal numbers (called weights) are nudged so the next guess is a little better. Over millions of nudges, it learns what a note tends to look like.

Inference is what happens when you press the button. Your recording goes through those finished weights and out comes an estimate. The weights don't change. Run the same clip ten times and you get the same calculation ten times, not a model that knows your song better.

That's the most common misunderstanding about these tools: "it'll learn my playing if I keep using it." An ordinary run doesn't learn anything. Some apps do keep recordings or use them to train later versions, but that's a choice the app makes about your audio, and it's worth knowing where your recording goes before you upload anything you care about. The model itself, on a normal run, is a fixed set of habits.

Where do the habits come from?

The examples a model trains on decide almost everything about how it behaves, so it's worth knowing the three ways they're usually gathered.

Labelled examples. The note reader above learned from recordings paired with the right answers: this clip, these notes. The field calls this supervised learning. It's the most direct way to teach a job, and the most expensive, because someone has to write down every answer. That's also why its habits follow its answer sheet so closely. If the sheets were mostly piano, it learned piano.

Unlabelled audio. Some models first learn from huge amounts of audio that nobody labelled at all. They set themselves puzzles: hide a slice of the sound and guess what was there, or squeeze a clip down and try to rebuild it. Solving millions of those puzzles teaches a model what music tends to sound like, with no answers needed. The field calls this self-supervised learning. On its own it doesn't do a musical job, but it gives a later job a head start.

Fine-tuning. Start from a model that already knows a lot in general, then train it a little more on a smaller set of examples for a narrower job. It's a bit like a session player with years of general listening learning your band's songs in a week. The new habits can push some old ones aside. Training on your own sounds covers when it's worth it and what it costs.

Whichever way the habits arrived, they're fixed by the time you press the button. The model you run is the result of all that training, not a student still in the room.

What was it trained to answer?

A model has one job, and everything else falls away. A note-reading model learned to answer "which pitches start and stop when?" It never learned to keep the player's tone, the room or the way a bend sings into the note. A stem splitter learned a different job: "which parts of this sound belong to the voice?" Same input, different question, different blind spots. Those are two of the eight jobs laid out in What AI can do in music, and every one of them has blind spots set by the question it was trained on.

A clear solo piano line gives a note-reading model fewer overlapping sounds to untangle. A driven guitar riff with ringing strings adds harmonics and overlapping tails that can be mistaken for extra notes. How well a particular model handles either recording depends on its training and the task it learned.

Same idea, different place: your ear has trained habits too. Musicians are often sure of a note's name and wrong about its octave, especially on an unfamiliar, bright timbre. Ear training builds those habits deliberately, and they slip in the same predictable places.

How do you check an estimate?

The trap with model output is that it always looks finished. Every note gets the same clean rectangle whether the model was sure or guessing. So check it in three passes, each tied to what you'll do with the result:

1. Pitch. Are the notes in the right octave? Any short notes sitting exactly an octave or a twelfth above a real one? Those are harmonics, and they're the first to delete.
2. Starts. Did one strum become three notes because the strings hit at slightly different times? If you need the rhythm, this matters most.
3. Ends. Did a ringing note become a held note that runs into the next one? For a bass line you're about to play on a synth, long tails make the part sound mushy.

When you listen back, use a plain, short sound. A long pad will blur note ends and hide exactly the errors you're looking for.

Where tidy estimates go wrong

Ghost harmonics are one pattern. A few others turn up whenever the input drifts from what the model learned:

• Reverb carries the last note into the next, so the model hears two notes held at once.
• Chords pile harmonics on top of each other, and some of them land on real chord tones.
• Noisy attacks, like a slapped bass or a scraped string, can read as a burst of tiny notes.

None of these produce a warning. The output looks just as confident as when it's right. So treat any surprising note as a question: listen to that moment, check whether it's an octave above something real, and decide. Sometimes a wrong note is a happy accident you'll keep. Just know that it's your choice, not what was played.

Make it yours

Record ten seconds of something you play or sing, a riff, a hummed melody, a whistled hook. Before you run any note-reading tool on it, write down one prediction: "the low note will come back an octave too high," or "the last note will ring on too long." Then run it, fix only the errors that matter for what you want to do next, and save the corrected MIDI with a one-line note of what the model got wrong. After two or three of these you'll know your tool's habits better than its manual does.

The same prediction game works on any model. Try it with StemSep: choose a separation method, then guess where sounds will leak between the stems before running it. Local methods keep the audio on your device; server methods send it away for processing. Check the method before using an unreleased recording.

Quick check

1. You run the same guitar clip through the model five times and the ghost notes never go away. Is the model failing to learn? It isn't trying to learn. An ordinary run uses fixed weights, so the same input gives the same estimate. To change the result, change the input (less drive, a cleaner take) or the tool.
2. A transcription of your piano line has an extra short note exactly an octave above the bass note on beat one. Played, or not? Probably not. A short note sitting exactly on a harmonic of a real note is the classic ghost. Listen to that moment: if you hear one note, delete it.
3. You need a bass line and a vocal from the same song. Can one model give you both? Not from a note reader. It answers "which pitches when", not "what does the voice sound like". You'd want a separator for the vocal audio, and a note reader on the separated bass if you want MIDI. Each step adds its own estimate.

Next steps

• Go deeper: Stem separation. If a model can be fooled by one guitar's harmonics, how does it pull a whole voice out of a finished mix?
• Preparation: How models see audio. The picture a model works from decides which mistakes it can make.
• Use it: Audio to MIDI. Turn the three-pass check into a fast workflow for cleaning up transcriptions.