Audio capture and rendering for extended reality experiences

Patent metadata

Field	Value
Publication number	US-11429340-B2
Application number	US-202016918441-A
Country	US
Kind code	B2
Filing date	Jul 1, 2020
Priority date	Jul 3, 2019
Publication date	Aug 30, 2022
Grant date	Aug 30, 2022

Abstract

In some examples, a content consumer device configured to play one or more of a plurality of audio streams includes a memory configured to store the plurality of audio streams and audio location information associated with the plurality of audio streams and representative of audio stream coordinates in an acoustical space where an audio stream was captured or synthesized or both. Each of the audio streams is representative of a soundfield. The content consumer device also includes one or more processors coupled to the memory, and configured to determine device location information representative of device coordinates of the content consumer device in the acoustical space. The one or more processors are configured to select, based on the device location information and the audio location information, a subset of the plurality of audio streams, and output, based on the subset of the plurality of audio streams, one or more speaker feeds.

First claim

What is claimed is:

1. A content consumer device configured to play one or more of a plurality of audio streams, the content consumer device comprising:
a memory configured to store the plurality of audio streams and audio location information associated with the plurality of audio streams and representative of audio stream coordinates in an acoustical space where an audio stream was captured or audio stream coordinates in a virtual acoustical space where an audio stream was synthesized or both, each of the audio streams representative of a soundfield; and
one or more processors coupled to the memory, and configured to:
determine device location information representative of device coordinates of the content consumer device in the acoustical space;
determine an audio source distance based on the audio location information as a distance between an audio source in the acoustical space and the device coordinates;
compare the audio source distance to an audio source distance threshold, the audio source distance threshold being definable by a user of the content consumer device via a user interface;
select, when the audio source distance is greater than the audio source distance threshold, a single audio stream of the plurality of audio streams as a subset of the plurality of audio stream; and
output, based on the subset of the plurality of audio streams, one or more speaker feeds.

2. The content consumer device of claim 1 , wherein the one or more processors are further configured to:
obtain a new audio stream and corresponding new audio location information; and
update the subset of the plurality of audio streams to include the new audio stream.

3. The content consumer device of claim 1 , wherein the one or more processors are further configured to:
determine, based on the plurality of audio streams, an energy map representative of an energy of a common soundfield represented by the plurality of audio streams,
wherein the selection of the single audio stream is further based on the energy map.

4. The content consumer device of claim 3 , wherein the one or more processors are further configured to:
analyze the energy map to determine the audio source location information.

5. The content consumer device of claim 1 , wherein the one or more processors are configured to:
select, when the audio source distance is less than or equal to the audio source distance threshold, multiple audio streams of the plurality of audio streams as the subset of the plurality of audio streams, the multiple audio streams being the subset of the plurality of audio streams with the audio stream coordinates surrounding the device coordinates.

6. A content consumer device configured to play one or more of a plurality of audio streams, the content consumer device comprising:
a memory configured to store the plurality of audio streams and audio location information associated with the plurality of audio streams and representative of audio stream coordinates in an acoustical space where an audio stream was captured or audio stream coordinates in a virtual acoustical space where an audio stream was synthesized or both, each of the audio streams representative of a soundfield; and
one or more processors coupled to the memory, and configured to:
determine device location information representative of device coordinates of the content consumer device in the acoustical space;
determine first audio stream coordinates for a first audio stream based on the audio location information;
determine a first audio source distance as a distance between the first audio stream coordinates and the device coordinates;
compare the first audio source distance to a first audio source distance threshold the first audio source distance threshold being moveable by a user of the content consumer device via a user interface;
select, when the first audio source distance is less than or equal to the first audio source distance threshold, the first audio stream of the plurality of audio streams; and
output, based on the first audio stream, one or more speaker feeds,

wherein the first audio stream is an only audio stream selected.

7. The content consumer device of claim 6 , wherein the one or more processors are further configured to:
determine a second audio source distance as a distance between second audio stream coordinates for a second audio stream and the device coordinates;
compare the second audio source distance to a second audio source distance threshold;
select, when both the first audio source distance is greater than the first audio source distance threshold and the second audio source distance is greater than the second audio source distance threshold, the first audio stream of the plurality of audio streams and the second audio stream of the plurality of audio streams; and
output, based on the first audio stream and the second audio stream, one or more speaker feeds.

8. The content consumer device of claim 7 , wherein the one or more processors are configured to combine the first audio stream and the second audio stream by at least one of adaptive mixing the first audio stream and the second audio stream or interpolating a third audio stream based on the first audio stream and the second audio stream.

9. The content consumer device of claim 8 , wherein the one or more processors are configured to combine the first audio stream and the second audio stream by applying a function F(x) to the first audio stream and the second audio stream.

10. The content consumer device of claim 7 , wherein the device location information is first device location information and the device coordinates are first device coordinates, and wherein the one or more processors are further configured to:
determine second device location information representative of second device coordinates of the content consumer device in the acoustical space the second device coordinates being different than the first device coordinates due to movement of the user of the content consumer device;
determine a third audio source distance as a distance between the first audio stream coordinates and the second device coordinates;
determine a fourth audio source distance as a distance between the second audio stream coordinates and the second device coordinates;
compare the third audio source distance to the first audio source distance threshold;
compare the fourth audio source distance to the second audio source distance threshold;
determine whether the second device coordinates have been steady relative to the first audio source distance threshold and the second audio source distance threshold for a predetermined period of time;
based on the second device coordinates being steady relative to the first audio source distance threshold and the second audio source distance threshold for a predetermined period of time, the comparison of the third audio source distance to the first audio source distance threshold, and the comparison of the fourth audio source distance to the second audio source distance threshold, select the first audio stream or the second audio stream; and
output, based on the first audio stream or the second audio stream, one or more speaker feeds.

11. The content consumer device of claim 7 , wherein the one or more processors are further configured to:
select, when the second audio source distance is less than or equal to the second audio source threshold, the second audio stream of the plurality of audio streams; and
output, based on the second audio stream, one or more speaker feeds,
wherein the second audio stream is an only audio stream selected.

12. The content consumer device of claim 7 , wherein the

Assignees

• Qualcomm Inc

Inventors

• Munoz Isaac Garcia
• Swaminathan Siddhartha Goutham
• Salehin S M Akramus
• Peters Nils Günther

Classifications

• G06F3/165 Primary

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• H04S2420/03

  Application of parametric coding in stereophonic audio systems · CPC title

• H04S2400/11

  Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) · CPC title

• H04S7/304 Primary

  For headphones · CPC title

• H04S2400/15

  Aspects of sound capture and related signal processing for recording or reproduction · CPC title