Immersive audio recording for grand piano?

[DC-Choppah]

I would like to try to record my grand piano using immersive audio techniques, to be played back on Apple AirPod Max headphones.

I would like to try one of the DPA immersive audio captures
https://www.dpamicrophones.com/immersive

... to try to hear the piano from my position as the player, where it feels so wonderful. Hard to explain really. I want YOU to experience it so I don't have to explain.

Playback is with Apple AirPod max headphones which when used from an ipod, anchor the sound to the screen as it moves. So the idea here is to have the experience that I have when I play and I make it sound magical.

Any thoughts on this technique?

What I am unclear on is which mic system to use for this, and what this technique will sound like. But I do want to give it a try.

My thought is to use the head mounted mics:

https://www.dpamicrophones.com/immersiv ... microphone

https://www.soundonsound.com/news/namm- ... 0-binaural

to hear the piano from my playing position and capture.

I will take all discussion to heart.

Of course, the worse case scenario is that I spent time and money here, it sounds no better than what I can capture with my nice stereo pencil mics

[Aled Hughes]

I think you might need to be careful here and ensure that any binaural effect is not being applies twice.

I think, the way you're explaining, that the Apple iPad/Airpods apply their own binaural processing, so in that case you wouldn't want to use a binaural mic setup alongside (ie head-mounted mics, or a dummy head).

If you're going for head-mounted mics, then you'll need a 'normal' stereo playback system that doesn't apply any further processing of its own. If such an option is available, I guess you could apply the head-tracking for left/right movement.

To get the Airpods headtracking etc working, I assume you'd need to use a surround/3D mic array, which the Apple tech then decodes into binaural stereo with head tracking (at least that's what my Audeze Mobius does - it takes a 7.1 mix, applies Waves binaural processing, takes the surround position information from the 7.1 mix and applies it to the head tracker. No good comes from feeding material that is already binaural into the processing!). It's still two-channel playback, but with the binaural processing and head-tracking.

I've been thinking about this - so much of 3D audio over headphones rely on binaural processing, and it's becoming more and more commonplace - I wonder what future binaural recording has anymore: unless you can be certain that the consumer playback systems doesn't apply further processing, the results could be quite unexpected.

[tacitus]

So I don’t have to walk around with tie-clip mikes on my glasses any more? I’ll buy that.

You can probably tell I’ve not had a lot to do with immersive audio. On the other hand, confronted by a new technique, my first instinct is to get somewhere near with the kit and methods I have already. In your case, record from your desired position with static mikes and establish a bottom line, then fix your pencil mikes to your head and see what you get. You’ll probably have to ignore “noises off” but if it’s with video you’ll get some notion of the sense or stupidity of it. I think.

[Sam Inglis]

Aled's right, if the capture format is binaural then the recorded audio should be played back as it was recorded, not sent through additional binaural processing on the way out. If you record with in-ear mics then the recorded audio will have the movement of your own head baked in. It won't capture sufficient 3D directional information for head tracking from the listener's point of view to be added afterwards.

There are various immersive / 3D mic arrays around that you could try. The most obvious starting point would be an Ambisonic mic like the Soundfield. A second-order Ambisonic mic like the Spatial Mic would deliver more precise localisation / directionality. Assuming you want the head tracking to follow the listener's movements rather than the player's, you would want the mic to be mounted in a fixed position, not on your head.

[Hugh Robjohns]

Seconded!

DO NOT use head-mounted mics. Every slight movement as you play will disturb the soundfield in a very unpleasant and annoying way.

If I'm reading you right, you want to capture the sound of the piano in your room as heard near the piano keyboard, as an immersive experience — meaning you want a surround sound mic array of some sort.

Like Sam, I'd go down the B-format ambisonic route as a first attempt because you can hire a suitable mic for the experiment, it's easy to rig and position, and post-processing is flexible and straightforward.

I'd position it on a boom stand behind you, hanging just over your head...

[Wonks]

Like Sam, I'd go down the B-format ambisonic route as a first attempt because you can hire a suitable mic for the experiment, it's easy to rig and position, and post-processing is flexible and straightforward.

I'd position it on a boom stand behind you, hanging just over your head...

Try Damocles Hire.

[tacitus]

Damocles? You’d leave a thread hanging, would you?

[The Red Bladder]

1. Pretty much what Hugh said. And definitely not any head-mounting - unless you play piano without moving or breathing! I get to do a great deal of classical piano recording as we have a Bösendorfer 275 and everyone who bangs away on that thing moves about and breathes. I've seen 'em at it!

2. Headphones implies a kunstkopf recording and those 4061s are ideal for that - I have a pair just for that purpose. Well, that and PZM recording and film dubbing (ADR). You can DIY a kunstkopf - you do not have to stump up the readies for some magic mic. DPA do a nice little rubber PZM mounting pad for the 4061 and at £18 each, well worth getting.

[Wonks]

[DC-Choppah]

These responses are incredibly helpful to me and I am very grateful. Know that I take it all to heart. I have never 'gone past' stereo, so this is new to me. But I do know how to follow my ears!

What I am trying to do is let you share the experience of what a beautiful piano, fully exposed with nothing between my ears and the piano sounds like when I play. There is a magic to it that does not come across in most recordings, and I am trying to see if it is possible to capture that inspirational magic, so that you can share the experience too.

So far I gather:

• Since I will be moving around (I can't help it), do not use head mounted mics to record.
• Since there will be fixed video showing the hammers/string from a fixed view, and since I want to play back on Apple AirPod Max that have the ability to fix sound to the screen and let you move your head, I can use one of the immersive recording techniques that produces B-Format Ambisonics. Thank you.

[The Red Bladder]

With an immersive audio, may I suggest an immersive video - i.e. put a few GoPros around the piano, so one on the keys, one mounted on your head and one on the fingers. You can open the sequence with a reference shot of you at the piano starting to play the piece recorded at a separate time without cameras all over the place and then cross-fade to the actual recording.

We do this sort of thing all the time BTW! It's easier than it might at first appear, though I've never tried a head-mounted GoPro yet.

[DC-Choppah]

With an immersive audio, may I suggest an immersive video - i.e. put a few GoPros around the piano, so one on the keys, one mounted on your head and one on the fingers. You can open the sequence with a reference shot of you at the piano starting to play the piece recorded at a separate time without cameras all over the place and then cross-fade to the actual recording.

We do this sort of thing all the time BTW! It's easier than it might at first appear, though I've never tried a head-mounted GoPro yet.

I love this Red, let me try to wrap my head around it.

So in the end we end up a an immersive video that can cut between some perspectives. One view shows me at the piano, and another shows what I see looking into the strings (my view from my head) another shows the perspective from the keys being pressed. And maybe one more showing the felt hit the strings

So if we edit together views that move between those perspectives, how does the immersive audio relate to that?

[The Red Bladder]

So you record the audio as a stereo signal using a binaural kunstkopf mounted above your head - two DPA 4061 mics side mounted using those rubber pad things mounted onto a block of shaped Styrofoam or similar stuff. The audio signal that will be used on the final product, instead of going to the regular multitrack, is sent to the video recorder at 48 kHz - 16 bit (the audio standard for video).

The main camera (in our place) is a 6K film camera, the rest are 4K DSLRs or a GoPro or whatever suits. Each records its own on-camera audio and we use that audio to auto-sync the images. So now we have let's say 3 or 4 video feeds and each one is later placed on the timeline in the video editor (DaVinci Resolve or Vegas or Premiere - I use Vegas) and you cross-fade and zoom in or out to fit the music.

My wacky suggestion is one head-cam, one film camera on the fingers, one possibly inside the piano on the felts close-up and one looking down from above, taking in fingers and felts.

Then add to all that (recorded separately of course!) a wide shot of you at the full grand doing the opening and closing pieces and one could also do a few crane-shots traveling across the piano - that always looks the dog's bo-locks!

Now that I think about it, a head-cam sounds awkward and likely to fail spectacularly, but perhaps a small cam on your chest might do the job.

The trick is to do the editing in sympathy with the music, so romantic stuff like Chopin could be a series of four-bar crossfades, whereas Mussorgsky could be sharp switches to closeups of fingers picking out just a bar here or there - depending on which pieces of course! Some of Chopin's stuff can be quite funky! I used that camera arrangement (without the head or chest cam) a few weeks ago for a sonata in B minor from old Fryderyk that sounded as if it were someone who lived 100 or 150 years later. That was for Korean TV (no, I don't know why either!)

Anyway, it's coming up to beer-o'clock, so I'll knock off for now!

[tea for two]

My wacky suggestion is ...one film camera on the fingers,

Dear Vangelis had a camera strapped to each of his wrists for a concert.
DC-Choppah wouldn't harm to do the Vangelis way. Were the camera darker in colour would blend better with a dark dinner jacket, blazer.

Also I wouldn't say head mounted it could be quite unwatchable with constant head bobbing up and down lolling aboot side to side lol lurching back n forth so as TRB said earlier chest mounted pointing down there's straps for these.

Now that I think about it, a head-cam sounds awkward and likely to fail spectacularly, but perhaps a small cam on your chest might do the job.

A contact mic preehaps near the harp.

A possible camera to consider is insta360 because there's an "invisible" stick for it, which insta software makes "disappear" so a person filming you can get as close far to you and the Piano from above close up lots of angles yet it looks like there was no camera in the video.
There's quite a following for this with people posting their wild n weirrrd disappearing camera videos.

[Arpangel]

I’m loving just recording to Garage Band on my MacBook Air, the sound of the playback through the internal speakers is heavenly, with all that spatial stuff going on, if only that could translate to any system, everything else just sounds so flat and boring, forget the latest monitors, we need speakers like this.

[DC-Choppah]

A possible camera to consider is insta360 because there's an "invisible" stick for it, which insta software makes "disappear" so a person filming you can get as close far to you and the Piano from above close up lots of angles yet it looks like there was no camera in the video.
There's quite a following for this with people posting their wild n weirrrd disappearing camera videos.

Tea, the insta360 and all 360 cameras appears to have a blind spot below the camera, so no video is captured along the line from the bottom of the camera?

[tea for two]

https://m.youtube.com/watch?v=awhRm5ruOuo
At 2min40sec is what I was kind of imagining for the Piano shots with an insta360.

[DC-Choppah]

https://m.youtube.com/watch?v=awhRm5ruOuo
At 2min40sec is what I was kind of imagining for the Piano shots with an insta360.

I am getting this. The 360 camera allows you to frame any kind of shot and shot variations in post. You can move the frame around, center the subject, or add motion at will in post. There is no frame of the original shot since you have 360.

These creative dynamic moves give the immersive effect since they seem impossible, like you are a ghost, or spirit that can inhabit the space at will. Very intimate with the space.

In the video they are not capturing audio, just video and have a soundtrack laid on top.

I want the audio to be the key focus so that you experience the intimate view sonically as well as visually.

I am not sure how to relate audio to these post-production views from the 360?

[DC-Choppah]

I’m loving just recording to Garage Band on my MacBook Air, the sound of the playback through the internal speakers is heavenly, with all that spatial stuff going on, if only that could translate to any system, everything else just sounds so flat and boring, forget the latest monitors, we need speakers like this.

Agreed the new Apple speakers on the iPad Pro, Macbook Air are really good and pleasant.

But I love the sound from the AirPod Max headphones. And I am targeting using those for those that want the full experience. They have spatial information that I am not hearing on anything else. Still trying to wrap my head around what they do and relation to audio production. I have not seen a SoS article looking at these Apple AirPod Max headphones?

[Arpangel]

I’m loving just recording to Garage Band on my MacBook Air, the sound of the playback through the internal speakers is heavenly, with all that spatial stuff going on, if only that could translate to any system, everything else just sounds so flat and boring, forget the latest monitors, we need speakers like this.

Agreed the new Apple speakers on the iPad Pro, Macbook Air are really good and pleasant.

But I love the sound from the AirPod Max headphones. And I am targeting using those for those that want the full experience. They have spatial information that I am not hearing on anything else. Still trying to wrap my head around what they do and relation to audio production. I have not seen a SoS article looking at these Apple AirPod Max headphones?

Yes I’m not a big fan of surround or "spatial" in general, but this Apple system is very good indeed, I’d like to investigate further too.

[tea for two]

https://m.youtube.com/watch?v=awhRm5ruOuo
At 2min40sec is what I was kind of imagining for the Piano shots with an insta360.

I am getting this. The 360 camera allows you to frame any kind of shot and shot variations in post. You can move the frame around, center the subject, or add motion at will in post. There is no frame of the original shot since you have 360.

These creative dynamic moves give the immersive effect since they seem impossible, like you are a ghost, or spirit that can inhabit the space at will. Very intimate with the space.

In the video they are not capturing audio, just video and have a soundtrack laid on top.

I want the audio to be the key focus so that you experience the intimate view sonically as well as visually.

I am not sure how to relate audio to these post-production views from the 360?

I love how you described "like you are a ghost, or spirit that can inhabit the space at will. Very intimate with the space. "

I don't know either how to relate quality audio recording of Piano to views from 360. Even if the 360 had a built in audio recorder it wouldn't be the quality and then there would be movement noise from 360 that would be recorded.

[DC-Choppah]

There are various immersive / 3D mic arrays around that you could try. The most obvious starting point would be an Ambisonic mic like the Soundfield. A second-order Ambisonic mic like the Spatial Mic would deliver more precise localisation / directionality. Assuming you want the head tracking to follow the listener's movements rather than the player's, you would want the mic to be mounted in a fixed position, not on your head.

Sam I am just now tracking what you said here. I am slow, but I always get there

My goal is the unique intimate feeling that you get by actually being there in that magical place, as a human being. The connection to the person listener to allow them to experience what I experience is the key new thing.

Therefore, I want THEM to be there. This means that as they move their orientation, the scene moves on the screen - the automatic view change from post processing of the 3D visuals. But also the audio changes to track.

The 3D cameras capture the whole seen and allow us to move the orientation in post automatically, and in real time using Airpod Max that track your orientation and apply the post automatically. You can't move your position, but you can change your orientation.

I want the audio to track the same way so that they are lined together properly. The goal is to simulate what you would have heard from that position if you moved your orientation in the same way.

It is a magical experience, and I am just trying to share the love.

You could sit still and just enjoy. But if you turn your head, the experience matches reality so it tricks you to make you feel like you are there.

I want to get good at this so that I can do it during concerts which may end up being at some of the most beautiful venues and beautiful pianos. I will have limited time and access, but I want to capture MY experience for my fans. I will have a tech to help me, but I need to experiment at my project studio first to see if I like this and show them what to do.

And all of my fans use Apple stuff to listen to With Apple Airpod Max as the most dedicated! So I want to give them some special stuff

[Sam Inglis]

So you'd be looking at putting up a single 3D camera and allowing the user to choose and vary the viewing angle from that camera in real time? If so then an Ambisonic mic would be the perfect audio counterpart to it. I'd probably try to place it directly above the camera on the assumption not many people would want to look straight upwards while listening. Then both the sound and the visuals can simultaneously follow the viewer's head-tracking.

[DC-Choppah]

So you'd be looking at putting up a single 3D camera and allowing the user to choose and vary the viewing angle from that camera in real time? If so then an Ambisonic mic would be the perfect audio counterpart to it. I'd probably try to place it directly above the camera on the assumption not many people would want to look straight upwards while listening. Then both the sound and the visuals can simultaneously follow the viewer's head-tracking.

Yes Sam THIS is it.

So imagine I am at a beautiful piano at a venue.

Behind me is a 3D camera and an ambisonic mic. I capture the show. Now time to edit later.

So we make a video and the user gets to view the video/audio. As they watch, in real time, the view on screen changes and the audio tracks as if they were really there.

The idea is to give the illusion that you were there. The realistic feeling of sitting where I was.

Now for a song that was really fun to play for me, YOU can experience it too. You get to be me.

[DC-Choppah]

I really want to try to make the audio and video track correctly together as you (the observer) moves. I feel like the correct tracking (even if it seems subtle) is actually where the magic is.

Sometimes when I try immersive stuff I become aware of the mismatch between the video and audio, so it destroys the sense of realism. Or the mismatch between my motion and the view, if there is a lag, or something you lose the magic.

But if the tracking is correct, perhaps the mind is fooled and thinks it is actually there?

Please understand, it is about translating the magical experience.

Magic is fragile, but not impossible.