Unmixing software

[Guest]

I keep hearing software can now unmix mixes. I heard Giles Martin talking about one of his Beatles release was done this way where there were no multitracks but the software could extract the individual instruments so you could remix them.
I assume it's similar to Melodyn's DNA technology.
I'm sure the results are far from perfect but still, I have some mono/stereo recording that could do with some tweaking so I was wondering who makes this type of software? Is there a market leader? Have SOS reviewed any?

[The Elf]

Steinberg's Spectralayers can do some remarkable things - it's easily the best I've tried. I've used it to undertake some subtle re-balancing on old tracks where the multi-tracks are no longer around.

But don't expect miracles.

[Exalted Wombat]

It's coming. More applicable to re-balancing than actual extraction as yet.

[Sam Inglis]

I've reviewed several of these packages (though I think they have all been updated since the most recent reviews):

RipX DeepAudio was perhaps the most impressive, but has a slightly baffling user interface.

DeMix Pro from Audiosourcere can do some pretty remarkable things.

Xtrax Stems from Audionamix is a simplified version of their 'full fat' unmixing software.

Note that some of them require audio to be uploaded to a server for processing, which can get annoying. And I don't think anyone has quite come up with a really streamlined user interface for this kind of stuff yet.

As Exalted Wombat says, they're all currently capable of rebalancing a mix pretty much without any obvious side effects, but extracting an a cappella vocal or removing one from a track won't be free from artifacts.

[Mike Stranks]

I've rebalanced some tracks in iZotope RX with some success. This has been things like bringing vocal forward, taming enthusiastic drums etc. But all fairly modest stuff.

It's worked fine for my applications - mixing into a podcast. However, if I wanted to do this regularly and as cleanly as possible, I'd be taking advice from The Elf...

[MOF]

but extracting an a cappella vocal or removing one from a track won't be free from artifacts.

This Giles Martin interview suggests it’s artefact free.

https://variety.com/2022/music/news/bea ... 62815/amp/

[James Perrett]

Things have come on in leaps and bounds over the last couple of years. Firstly it should be pointed out that most commercial software is just a repackage of something that is available for free if you are comfortable using command lines.

RX's Music Rebalance and Acon use Spleeter from Deezer while the others seem to perform in a very similar way to Demucs which was developed by Meta.

I think that the Abbey Road people are probably basing their technology on one of these while using their extensive library of multitracks to train the software further. The fact that they can separate Paul and John's voices in the Beatles film is very impressive. Training the software is something that I would really like to learn to do but haven't had a chance to dive deep enough into the documentation to work out how to do it.

Spleeter tends to leave a few artefacts in the individual tracks but is the only one that can extract piano in its standard version and the artefacts cancel out when the tracks are recombined. Demucs leaves fewer artefacts with the drums, bass and vocals but the 'other' track can be less convincing - especially when the 'other' is only guitars.

Music Rebalance is all over this

https://www.youtube.com/watch?v=MyXQEip ... c&index=10

And this

https://music.youtube.com/watch?v=2r_bM ... EeAh1Jjt8M

which are both examples of a less than perfect stereo recording (from an old cassette in the first case) being transformed by the magic of source separation. Unfortunately these were done before I discovered Demucs and none of the work I've done using Demucs has been released yet.

PS - Martin, that's Carol on keyboards on the first one.

[Gone To Lunch]

I have just had this marketing hit from just such an extraction service:

https://www.lalal.ai

[MarkOne]

I have just had this marketing hit from just such an extraction service:

https://www.lalal.ai

I’ve used that before, and as others have said, it is not artefact free, but for the purposes creating a new mix it’s pretty good. And I suspect all these services will improve as the AIs gets more data and the learning algorithms improve, it can only get better.

[Tim Gillett]

... The fact that they can separate Paul and John's voices in the Beatles film is very impressive...

I must have missed that. I watched a lot of of the many pre release YT promos for the film and apart from one very brief sound clip, there seemed just a lot of talk about how great it was. If the results are so apparently great, such as in the Beatles film soundtrack, why not let the before and after audio samples do the talking, selling themselves?

[James Perrett]

If the results are so apparently great, such as in the Beatles film soundtrack, why not let the before and after audio samples do the talking, selling themselves?

I'm fairly sure that a before and after was exactly what I found - possibly from a link on this forum. There's a before and after for Satisfaction on the Abbey Road website.

However, I've just seen a Wired article about James Clarke which implies that much of the Abbey Road process is manual spectral editing.

[Tim Gillett]

I'm fairly sure that a before and after was exactly what I found - possibly from a link on this forum. There's a before and after for Satisfaction on the Abbey Road website.

Yes about 15 seconds of Satisfaction (4:44) and with the most basic error: failing to level match the "before" and "after" samples.

However, I've just seen a Wired article about James Clarke which implies that much of the Abbey Road process is manual spectral editing.

What do you think that suggests or implies?

[MarkOne]

What do you think that suggests or implies?

That the technology is still relatively in it's infancy. And will almost certainly get better and better.

Look at Midjourney and Dal.e now. Around a year ago if you typed in elephant, you'd get a blurry thing that if you held your head the right way and squinted you might just make out said pachyderm.

Now you can type "Elephant on a wide savanna with it's baby, in the style of Van Gogh" and you'll get something that, while it wouldn't fool any Van Gogh scholars as a lost work from his unknown African period, most people would go, "Hey yeah, that elephants as if painted by Van Gogh"

[Drew Stephenson]

I'd suggest that Abbey Road also know that with the Beatles they have a pretty guaranteed audience so it's worth spending the time on it manually.
They might be using the manual files as training for the AI as well.

[Tim Gillett]

What do you think that suggests or implies?

That the technology is still relatively in it's infancy. And will almost certainly get better and better.

Look at Midjourney and Dal.e now. Around a year ago if you typed in elephant, you'd get a blurry thing that if you held your head the right way and squinted you might just make out said pachyderm.

Now you can type "Elephant on a wide savanna with it's baby, in the style of Van Gogh" and you'll get something that, while it wouldn't fool any Van Gogh scholars as a lost work from his unknown African period, most people would go, "Hey yeah, that elephants as if painted by Van Gogh"

Regardless of the stage of development, actual before and after audio examples are the real deal. I suggest that the natural limitation on progress in audio source separation is the well known phenomenon of masking.

[MarkOne]

What do you think that suggests or implies?

That the technology is still relatively in it's infancy. And will almost certainly get better and better.

Look at Midjourney and Dal.e now. Around a year ago if you typed in elephant, you'd get a blurry thing that if you held your head the right way and squinted you might just make out said pachyderm.

Now you can type "Elephant on a wide savanna with it's baby, in the style of Van Gogh" and you'll get something that, while it wouldn't fool any Van Gogh scholars as a lost work from his unknown African period, most people would go, "Hey yeah, that elephants as if painted by Van Gogh"

Regardless of the stage of development, actual before and after audio examples are the real deal. I suggest that the natural limitation on progress in audio source separation is the well known phenomenon of masking.

Yes, Tim, I get that. There may well not be enough actual data to definitively pull out a clean part from the mix just through spectral analysis, but there are over 100 million digitised examples of songs for an AI to train on, so my expectation is that once it gets bits it can’t cleanly extract, it can make up what it thinks should be there. Will it be accurate? Who cares providing the result is musically consistent when compared to the mixed track.

Going back to image manipulation, have you seen what Adobe’s Sensei engine can do in apps like photoshop to fill in a background if you delete a foreground object? Not 100% every time, but improving with every release.

[Hugh Robjohns]

I suggest that the natural limitation on progress in audio source separation is the well known phenomenon of masking.

Masking is a human failing. It doesn't affect a computer DSP code's ability to find and extract specific streams of audio... although it could be used creatively to help hide processing artefacts.

Obviously, audio source separation is still in its relative infancy, but the progress made over the last five years has been incredible and that is bound to continue for quite a while yet.

[MarkOne]

I suggest that the natural limitation on progress in audio source separation is the well known phenomenon of masking.

Masking is a human failing. It doesn't affect a computer DSP code's ability to find and extract specific streams of audio... although it could be used creatively to help hide processing artefacts.

Obviously, audio source separation is still in its relative infancy, but the progress made over the last five years has been incredible and that is bound to continue for quite a while yet.

You’re right Hugh (and by the way that was Tim’s quote, not mine )

[ManFromGlass]

On the practical level I am curious of applications beyond repair of an unacceptable mix. I can see potential copyright issues as somebody creates, say, a duet between Adele and Prince with Ringo and Bono and everyone else who only have first names singing the choruses. But if there is money to be made then the record companies would sort it out.
I can also see forensic usefulness to remove background noise, so spies could no longer have conversations by water fountains. But what else?

[Tim Gillett]

If the results are so apparently great, such as in the Beatles film soundtrack, why not let the before and after audio samples do the talking, selling themselves?

I'm fairly sure that a before and after was exactly what I found - possibly from a link on this forum...

In one pre release promo of the "Get Back" movie, Peter Jackson does specifically state: "...we've even taught the computer what John's voice sounds like and what Paul's voice sounds like..."
We might think that would be the obvious place to demo the audio improvement from where John's voice had previously been overpowered by Paul's voice and vice versa, or from the band instruments. Sadly, no such demo.

https://youtu.be/qQFPnA8OgLo

Another YT upload from New Scientist does again have an audio demo, and thankfully longer than the PJ clip, but no evidence of making intelligible, band members' voices unintelligible in the original recording - which is what PJ and New Scientist seemed to claim could be done now, not some time in the future. It seems Pauls' voice is always intelligible even in the original mix. But if it's not intelligible in the original mix why not demo that and then the intelligibility restoration after processing?

https://youtu.be/sBR5VNWJZmw

Does anyone else notice such disparity between the spoken claims and the presented evidence?

[James Perrett]

Does anyone else notice such disparity between the spoken claims and the presented evidence?

I'm not particularly bothered by other people's claims - I just get on an use the software to create releasable material from sources that otherwise could not be used. One of my more technically impressive uses of it was the song "Move" on the "Waiting for the Weekend" box set by The Vapors from last year. The only vocal track was a guide vocal which was almost buried by spill from the other instruments. Music Rebalance brought this out enough to make it usable. Unfortunately you have to buy the box set if you want to hear it as it isn't available online.

[Hugh Robjohns]

You’re right Hugh (and by the way that was Tim’s quote, not mine )

Apologies. Fixed!

[Tim Gillett]

I suggest that the natural limitation on progress in audio source separation is the well known phenomenon of masking.

Masking is a human failing...

So signs of likely masking wouldnt show up on any measuring/analysis tools?

[resistorman]

I'm thinking that the future of noise reduction and audio reconstruction will be the re-synthesis of the original material, not removing offending artifacts.

[Wonks]

I suggest that the natural limitation on progress in audio source separation is the well known phenomenon of masking.

Masking is a human failing...

So signs of likely masking wouldnt show up on any measuring/analysis tools?

No. Masking is a by-product of the way our brains interpret sounds and limit the amount of information that they need to process. The sonic information is still there for software to work with on full fidelity, non-data compressed recordings.

As for sound demos, don't forget that this is The Beatles we are talking about. Extremely reticent to put things online, especially when the songs themselves are involved.