Neural instruments and timbre transfer

Press play and listen to a singer whose voice has been taken apart. The words are still there. The tune is still there. But the throat is gone, replaced by a buzzing chord of saw waves that somehow pronounces every vowel. This is a vocoder, and it has been splitting "what is being sung" from "what it sounds like" since Bell Labs built one in the 1930s to squeeze speech down phone lines. Kraftwerk and Daft Punk turned that split into a sound people still chase.

Before you touch Bands: which part of the performance do you think is most fragile? Hold on to your guess.

A neural instrument makes the same trade with a very different tool. Instead of a fixed bank of filters, it uses a model that has listened to hours of one kind of sound (birdsong, a cello, a strange piano) and learned to describe any input in that sound's terms. Sing into a birdsong model and, when it works, the birds sing your phrase back. When it doesn't, you get warbling mush. This guide is about hearing where that line is, and learning to play right up against it.

What survived?

If you lowered Bands, you probably heard the words go first. Consonants and vowels live in fine detail across the spectrum, and 8 bands are too coarse to draw them. The rhythm held, because every band still rises and falls with the singer's syllables. The tune held, because the synth underneath is following the singer's pitch on its own.

That's the question to carry into any timbre transfer: which parts of my performance does this system actually carry? It's never "all of them". A vocoder carries the shape of the spectrum over time and throws away the voice's own buzz. A learned model carries whatever its internal description can hold, and that description was shaped by the recordings it trained on, not by your intentions.

Timbre is what distinguishes two sounds at the same pitch and loudness: the balance of overtones, the breath, the scrape of a bow, the way a note blooms and fades. Timbre transfer tries to keep your pitch and timing while swapping that character. It usually loses some detail, so listen for what it drops. Point the same idea at a model trained on one person's singing and you're converting your voice into theirs, which raises a question this guide leaves aside: whose voice is it? Voices, cloning and consent takes that up.

Inside a learned instrument

The learned instrument on this page, LiveTransfer, is built on RAVE, a model family from the IRCAM research institute in Paris. It came after research demos such as Magenta's Tone Transfer (2020), which turned singing into violin, flute or trumpet in a browser and made a lot of musicians curious about all this. RAVE works in two halves.

The encoder listens to a short slice of your audio and writes down a few numbers that describe it. For the birdsong model that's 8 numbers, refreshed roughly every 11 milliseconds. This list is called the latent code: a compressed description, a bit like a chord chart is a compressed description of a song. The decoder reads the code and turns it back into audio, but the only sounds it ever learned to make are birds. So whatever you feed in comes out as the decoder's best bird-shaped guess at it.

That makes every output an estimate: the model's reading of your phrase, through the only vocabulary it has. Where your sound resembles something in the training recordings, the estimate follows you closely. Where it doesn't, the model fills the gap with what it knows.

If you want to see how other models squeeze audio into codes like this one, How models see audio follows that idea in depth.

Play the real thing

When you open it, the model downloads into your browser first, about 50 MB for the birdsong model and about 125 MB for the piano one, so give it a moment on a slow connection. After that it runs on your own machine: the audio you feed it is processed in the page and isn't sent anywhere.

Feed it a voice or an instrument and start with every Z slider at zero. Amount blends between your input and the model's version, so turn it down to hear what went in, then back up to hear what came out. Listen for the same three things as before: does the rhythm survive, does the rise and fall survive, and what happened to the attacks?

Now move Z1 a little. Each Z slider adds a fixed amount to one of the 8 numbers in the code. Zero doesn't mean silence; it means "leave this part of the description alone". A small offset is like asking the birds to sing your phrase from a slightly different branch. A big one asks them for something they've never sung.

Why a still code goes quiet

Here's the surprise that shapes how these instruments are designed. You might expect that if you stop singing and just set the 8 numbers to an interesting position, the decoder would hold that sound like a synth holds a note. On this path, it mostly goes quiet.

The decoder learned from codes that were always moving, because real recordings are always moving: birds flutter, bows wobble, breath shifts. A frozen code doesn't look like anything it heard in training, so its best estimate is close to nothing. That's why LiveTransfer needs a live input to steer, and why its sibling, Terrain RAVE, quietly encodes a short seed clip and moves around on top of that. The sliders are a way of bending a performance, not of building a sound from nothing.

Our take: that's what makes these instruments feel alive rather than like a preset. You're never playing the model alone. You're playing a duet with your own input.

Same idea, different place: the vocoder at the top makes the same split between "what's happening" and "what it sounds like" with filters you can count. And the same rule applies to both: the carrier has to be rich enough to draw on. A thin input gives a thin result. Vocoders shows why a buzzy saw makes clearer words than a sine.

Where it breaks

The most useful thing you can find with a neural instrument is its edge. Try these, one at a time:

• Drums into the bird model. A model that learned sustained chirps has little idea what to do with a snare crack. You'll likely hear the hits smeared into rustling, with the rhythm blurred. The model isn't broken; you've asked it about something outside everything it heard.
• A very low or very high note. Birdsong lives high. Sing low and the pitch tends to drift or get pulled into the range the model knows.
• Big Z offsets. Push several sliders toward their ends (the range is minus 3 to plus 3) and the texture dissolves into noise. The slider range tells you what numbers the control accepts, not which ones are musical.
• Fast playing. The model works on slices of audio, and your sound has to fill a slice before it can be encoded, decoded and played. The birdsong model's slices are short; the piano model's are about four times longer, so it feels slower and suits held notes better than quick runs.

If you're curious why models fail in such predictable directions, What an audio model learns traces it back to the training recordings.

Make it yours

Find the edge with your own sound. Pick an input that's new to you here: your voice humming, a guitar, a recorded loop. Before you play it, guess which of rhythm, contour and attacks will survive. Then:

1. Set every Z to zero and Amount to full, and listen to the plain transfer.
2. Move one Z slider in small steps until the sound starts to lose your phrase, then back off a little.
3. Try a second slider only once you can describe what the first one does.

The sweet spot is usually just before the sound tips over: clearly the model's character, still clearly your phrase. Press Save on the instrument above, and name the preset after the input that makes it work ("hummed lines", "slow cello"), because the same settings can sound wonderful on one source and like static on another. Both models here are shared for non-commercial use, so keep what you make with them to sketches and personal pieces unless you get permission.

Quick check

1. You sing a fast melody into the birdsong model. The output sounds unmistakably like birds, but you can't hear your rhythm any more. Did it work? That depends on what you wanted. As a texture, maybe yes. As a way to perform your phrase, no. The target sound arriving tells you nothing about whether your performance arrived. Judge it by the part you care about, and try a slower, more sustained phrase to see whether the rhythm comes back.
2. You set all eight Z sliders to zero. Will the output be silent? No. Zero means "don't nudge the code", so you hear the model's plain reading of your input. Silence is what you'd tend to get from the opposite situation: a frozen code with no moving input underneath it.
3. A friend says the piano model is "laggy" and the bird model isn't. Is the piano model just slower to compute? Not necessarily. The piano model works on slices about four times longer, so it has to wait for more of your sound before it can answer. That waiting is built into its design. Held notes and slow swells hide it; fast runs expose it.

Next steps

• Go deeper: Training on your own sounds. The birds and the piano were someone else's recordings. What would it take to train a model on yours, and how much would you need to record?
• Use it: Camera and gesture control. Eight sliders are a lot to move with a mouse while you sing. What if your hand were the controller?
• Build it: Adding a model to your app. That 50 MB download, the progress bar and the Cancel button all had to be designed. Here's how.