Audio processing in immersive audio services

Patent metadata

Field	Value
Publication number	US-12167219-B2
Application number	US-201917292457-A
Country	US
Kind code	B2
Filing date	Nov 12, 2019
Priority date	Nov 13, 2018
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

The disclosure herein generally relates to capturing, acoustic pre-processing, encoding, decoding, and rendering of directional audio of an audio scene. In particular, it relates to a device adapted to modify a directional property of a captured directional audio in response to spatial data of a microphone system capturing the directional audio. The disclosure further relates to a rendering device configured to modify a directional property of a received directional audio in response to received spatial data.

First claim

The invention claimed is:

1. A device comprising or connected to a microphone system comprising one or more microphones for capturing audio, the device comprising:
a receiving unit configured to:
receive directional audio comprising one or more directional source signals captured by the microphone system;
receive metadata associated with the microphone system, the metadata comprising spatial data of the microphone system, the spatial data being indicative of at least a spatial orientation and/or a spatial position of the microphone system and comprising at least one of a yaw or azimuth, pitch, roll angle(s), or spatial coordinates of the microphone system;

a computing unit configured to:
define a rotation/translation matrix based on the spatial data;
modify at least some of the directional audio by multiplying the directional audio with the rotation/translation matrix to produce modified directional audio, wherein a directional property of the directional audio is modified in response to the spatial data of the microphone system;
encode the modified directional audio into digital audio data; and

a transmitting unit configured to transmit the digital audio data.

2. The device according to claim 1 , wherein the spatial orientation of the microphone system is represented with parameters describing rotational movement/orientation with one degree of freedom (DoF) in the spatial data.

3. The device according to claim 1 , wherein the spatial orientation of the microphone system is represented with parameters describing rotational movement/orientation with three DoF in the spatial data.

4. The device according to claim 1 , wherein the spatial data of the microphone system is represented in six DoF.

5. The device according to claim 1 , wherein the received directional audio comprises audio comprising directional metadata.

6. The device according to claim 1 , wherein the computing unit is further configured to encode at least parts of the metadata comprising the spatial data of the microphone system into said digital audio data;
wherein the transmitting unit is configured to transmit the digital audio data comprising the metadata.

7. The device according to claim 6 , wherein the receiving unit is further configured to receive first instructions indicating to the computing unit whether to include said at least parts of the metadata comprising the spatial data of the microphone system into said digital audio data, whereby the computing unit acts accordingly.

8. The device according to claim 6 , wherein the receiving unit is further configured to receive second instructions indicating to the computing unit which parameter or parameters of the spatial data of the microphone system to include in the digital audio data, whereby the computing unit acts accordingly.

9. The device according to claim 7 , wherein the transmitting unit is configured to transmit the digital audio data to a further device, wherein indications about first and/or second instructions are received from said further device.

10. The device according to claim 1 , wherein the receiving unit is further configured to receive the metadata comprising a time stamp indicating a capturing time of the directional audio, wherein the computing unit is configured to encode said time stamp into said digital audio data.

11. The device according to claim 1 , wherein the computing unit is further configured to:
downmix the modified directional audio based on the spatial data of the microphone system using a downmix matrix; and
encoding the downmixed directional audio and the downmix matrix into the digital audio data,
wherein the downmixing comprises adjusting a beamforming operation of the modified directional audio based on the spatial data of the microphone system.

12. The device according to claim 1 , wherein the device is implemented in a virtual reality (VR) gear or an augmented reality (AR) gear comprising the microphone system and a head-tracking device configured to determine the spatial data of the device in 3-6 DoF.

13. A device for rendering audio signals, the device comprising:
a receiving unit configured to receive digital audio data;
a decoding unit configured to:
decode the received digital audio data into directional audio and metadata, the metadata comprising spatial data indicative of at least a spatial orientation and/or a spatial position of a microphone system; and

a rendering unit configured to:
define a rotation/translation matrix based on the spatial data;
modify a directional property of the directional audio by multiplying the directional audio with the rotation/translation matrix; and
render the modified directional audio.

14. The device according to claim 13 , wherein the microphone system comprising one or more microphones capturing the directional audio, wherein the rendering unit modifies the directional property of the directional audio to at least partly reproduce an audio environment of the microphone system.

15. The device according to claim 13 , wherein the spatial data comprises parameters describing rotational movement/orientation with one degree of freedom; (DoF).

16. The device according to claim 13 , wherein the spatial data comprises parameters describing rotational movement/orientation with three DoF.

17. The device according to claim 13 , wherein the directional audio comprises audio comprising directional metadata.

18. The device according to claim 13 , further comprising a transmitting unit configured to transmit instructions to a further device from which the digital audio data is received, the instructions indicating to the further device which parameter or parameters rotational data should comprise.

19. The device according to claim 13 , wherein the decoding unit is further configured to extract a time stamp indicating a capturing time of the directional audio from the digital audio data.

20. The device according to claim 13 , wherein the spatial data includes spatial coordinates and wherein the rendering unit is further configured to adjust a volume of the rendered audio based on the spatial coordinates.

21. The device according to claim 13 , being implemented in a virtual reality; (VR) gear or an augmented reality (AR) gear comprising a head-tracking device configured to measure the spatial orientation and the spatial position of the device in six DoF.

22. The device according to claim 13 , wherein the rendering unit is configured for binaural audio rendering.

23. A system comprising:
a first device according to claim 1 configured to transmit digital audio data to a second device according to claim 13 , wherein the system is configured for audio and/or video conferencing.

24. The system according to claim 23 , wherein the first device further comprises a video recording unit and being configured to encode recorded video into digital video data and transmit the digital video data to the second device, wherein the second device further comprises a display for displaying decoded digital video data.

25. A system comprising a first device according to claim 1 configured to transmit digital audio data to a second device, the second device comprising:
a receiving unit configured to receive the digital audio data,
a decoding unit configured to:
decode the received digital audio data into directional audio and into metadata, the metadata comprising spatial data comprising at least one from the list of: a ya

Assignees

• Dolby Laboratories Licensing Corp
• Dolby Int Ab

Inventors

• Bruhn Stefan
• Torres Juan Felix
• Mcgrath David S
• Lee Brian B

Classifications

• H04S7/30 Primary

  Control circuits for electronic adaptation of the sound field · CPC title

• H04S2400/03

  Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1 (H04S2400/01 takes precedence) · CPC title

• H04R2499/11

  Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's · CPC title

• H04R5/027

  Spatial or constructional arrangements of microphones, e.g. in dummy heads · CPC title

• H04R1/326

  for microphones (H04R1/34 and H04R1/40 take precedence) · CPC title