Stem separation

Here's a finished song with a singer on top. Your job: take the singer out, using one band of EQ. The band below starts as a deep, wide cut right where a voice is clearest. Press play, then hunt for a setting that makes the voice vanish and leaves the band alone.

You can make the voice quieter. You can't make it go away. Every setting that hurts the voice hurts the guitars and cymbals sitting in the same frequencies, because an EQ treats the whole song the same way, all the time.

And yet stem splitters do something that looks a lot like the impossible. In 2023 the Beatles released "Now and Then", built around John Lennon's voice lifted from a 1970s home demo where it was tangled up with his piano. The voice was pulled free with machine learning, using separation tools Peter Jackson's team had developed for the Get Back documentary. Producers now use the same kind of tool every day to pull acapellas for remixes, mute the bass to practise along, or sample a drum break that never came as a separate track.

How, if an EQ can't?

Why can't you just un-mix a song?

Mixing is adding. At every instant, the voice, guitar, bass and drums each push the speaker a little, and the mix is the sum of those pushes. A sum doesn't remember what was added.

Try it with numbers. Two numbers add up to 10. Were they 6 and 4? 9 and 1? 15 and minus 5? The 10 alone can't tell you. A song is that puzzle at 44,100 or 48,000 moments a second, in two channels.

So a separator doesn't find hidden tracks. It makes an educated guess about how to split the sum, using what it learned from thousands of songs where the separate parts were available. It knows what voices tend to look like, what kick drums tend to look like, and it splits the mix the way those habits suggest. What comes out is an estimate of each part. It can be very good. It's never the original.

An EQ that changes a thousand times a second

Here's the trick that makes it work where your EQ failed. Many separators, including Spleeter, which Deezer's research team released as open source in 2019, look at the song as a spectrogram: the sound chopped into short slices, each split into frequencies.

For every slice and every frequency, the model decides how much belongs to the voice. Near 1 means "this is voice, keep it"; near 0 means "this is something else". That grid of decisions is called a mask, and it's applied to the mix to make the vocal estimate. Another mask makes the drums, another the bass.

Think of it as an EQ with thousands of bands that resets itself many times a second. When the singer holds a note at 400 Hz, the voice mask opens at 400 Hz for exactly that moment. When the guitar takes over that frequency a beat later, the mask closes. Your EQ had to make one choice for the whole song. The mask makes a fresh choice for every tiny tile of time and pitch.

Look at the mix as a picture and you can see both why this works and where it's hard.

Where the voice has a tile to itself, the mask can be confident. Where the voice and a cymbal share the same tile, the model has to divide one patch of energy between two owners, and that's where the trouble starts.

What does a separated vocal sound like?

Once you know the mechanism, the artefacts stop being mysterious. Each one is the mask meeting a tile it can't cleanly assign:

• Leakage. Listen in the gaps between sung lines. Faint hi-hats or guitar chords often hang there, the bits the mask let through.
• Smeared attacks. Drum hits and plucked notes lose their snap. The mask works on slices of time, so a sudden edge gets spread across a slice. (Why the slices blur time is the subject of how models see audio.)
• Watery, chirpy sheen. When the mask flickers between keep and drop on neighbouring tiles, held notes start to bubble, like they're being heard through water.
• Reverb in odd places. A reverb tail mixes the voice with everything else, so part of it ends up in the vocal and part in "other".
• "Other" as a catch-all. A four-stem split gives you vocals, drums, bass and other. Piano, guitars, synths and strings all land in other together.

And here's the surprising one. Add the four estimates back together and you often get something very close to the original mix. That feels like proof the split was right, but it isn't. Back to the numbers: 6 + 4 and 9 + 1 both make 10. If the model put some guitar in the vocal and took the same amount out of "other", the sum is perfect and both stems are wrong.

Not every separator works this way. Some work directly on the waveform, and some combine both, and they make different mistakes. But every one of them is estimating.

When is a stem good enough?

That depends entirely on what you're making. A bass stem with a bit of kick bleeding through is perfect for learning the line by ear. A vocal with faint cymbals is fine under a new beat that has its own cymbals. The same vocal, exposed alone at the quiet end of a remix, will give itself away.

Our take: check every stem twice. First solo, to find the flaws. Then in the track you're actually making, to find out whether they matter. Most leaks that sound awful alone disappear once the new drums come in.

Clean-up only goes so far. EQ can shave off a leaking hi-hat, but it takes the "s" of the singer with it. A gate can silence the gaps, but it can't bring back a consonant the mask threw away. If the stem is missing what your edit needs, change the section you use, not the plug-in chain. Learned repair tools can go further than EQ and gates, but they rebuild what they remove rather than uncovering it; Repair and restoration explains how to hear what they invented.

Stems to remix puts that judgment to work: choose a short section you're allowed to use, mark the stems' weak spots, then build new parts that cover those spots.

Make it yours

Pick a song you'd love to remix or play along with, and choose a section up to three minutes long. Before you separate it, write down what you want ("an acapella for the chorus", "the bass line to learn") and one prediction about where it'll break ("the cymbals will leak into the vocal on the loud chorus").

Run it through StemSep. Its default methods work inside your browser, so the song never leaves your computer; if you switch to the server or paid-provider option, the clip is uploaded to be processed elsewhere. Listen to each stem solo, mark one place it breaks, then try your chosen stem in a quick edit. Export the stems as WAV files, which stay perfectly lined up with the original, and keep your note of what broke. The next time you pick a song to separate, you'll know which ones are worth the trouble.

Quick check

1. You add the four stems back together and it sounds exactly like the original. Can you trust the vocal stem? Not from that alone. The sum only proves nothing was lost overall. Some guitar could have been put in the vocal and taken out of "other", and the sum would still match. Solo the vocal and listen in the gaps.
2. You want a piano sample from a song with piano, guitar and synth pads. Will a four-stem split give it to you? Probably not cleanly. A four-stem split puts piano, guitar and pads together in "other". You'd need a separator trained for piano, or a passage where the piano plays alone.
3. The separated drums sound soft, as if someone rounded off every hit. Is that a mixing problem in the original song? Unlikely. A mask works on short slices of time, so sharp edges get spread across a slice. Expect it most on snappy drums and plucked notes, and check how much it matters once the stem is back in a mix.

Next steps

• Use it: Making room with EQ. Fit a stem into a new mix so its leaks stop mattering.
• Build it: Adding a model to your app. What it takes to run a separator like this inside your own app: downloads, progress, and a Cancel button that really cancels.