Mixed reality spatial audio

Patent metadata

Field	Value
Publication number	US-10616705-B2
Application number	US-201816163529-A
Country	US
Kind code	B2
Filing date	Oct 17, 2018
Priority date	Oct 17, 2017
Publication date	Apr 7, 2020
Grant date	Apr 7, 2020

Abstract

A method of presenting an audio signal to a user of a mixed reality environment is disclosed. According to examples of the method, an audio event associated with the mixed reality environment is detected. The audio event is associated with a first audio signal. A location of the user with respect to the mixed reality environment is determined. An acoustic region associated with the location of the user is identified. A first acoustic parameter associated with the first acoustic region is determined. A transfer function is determined using the first acoustic parameter. The transfer function is applied to the first audio signal to produce a second audio signal, which is then presented to the user.

First claim

What is claimed is:

1. A method of presenting an audio signal to a user of a mixed reality environment, the method comprising:
detecting an audio event associated with the mixed reality environment, where the audio event is associated with a first audio signal;
determining a location of the user with respect to the mixed reality environment;
detecting a spatial property of an environment of the user;
identifying, based on the detected spatial property, a predefined first acoustic region;
determining a first acoustic parameter associated with the predefined first acoustic region;
determining, using the first acoustic parameter, a transfer function;
applying the transfer function to the first audio signal to produce a second audio signal; and
presenting, to the user, the second audio signal.

2. The method of claim 1 , wherein the first audio signal comprises a waveform audio file.

3. The method of claim 1 , wherein the first audio signal comprises a live audio stream.

4. The method of claim 1 , wherein:
the user is associated with a wearable system comprising one or more sensors and a display configured to present a view of the mixed reality environment, and
identifying the predefined first acoustic region comprises detecting first sensor input from the one or more sensors and identifying the predefined first acoustic region is based on the first sensor input.

5. The method of claim 4 , wherein determining the first acoustic parameter comprises detecting second sensor input from the one or more sensors and determining the first acoustic parameter is based on the second sensor input.

6. The method of claim 5 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a geometric characteristic of the predefined first acoustic region and determining the first acoustic parameter is based on the geometric characteristic.

7. The method of claim 5 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a material associated with the predefined first acoustic region and determining the first acoustic parameter is based on the material.

8. The method of claim 4 , wherein the wearable system further comprises a microphone, the microphone is located in the predefined first acoustic region, and the first acoustic parameter is determined based on a signal detected by the microphone.

9. The method of claim 1 , wherein the first acoustic parameter corresponds to a reverberation parameter.

10. The method of claim 1 , wherein the first acoustic parameter corresponds to a filtering parameter.

11. The method of claim 1 , further comprising:
detecting a second spatial property of the environment of the user;
identifying, based on the detected second spatial property, a predefined second acoustic region, the predefined second acoustic region acoustically coupled to the predefined first acoustic region; and
determining a second acoustic parameter associated with the predefined second acoustic region,
wherein the transfer function is determined using the second acoustic parameter.

12. A system comprising:
a wearable headgear unit including:
a display configured to present a view of a mixed reality environment;
a speaker;
one or more sensors; and
circuitry configured to perform a method comprising:
detecting an audio event associated with the mixed reality environment, where the audio event is associated with a first audio signal;
determining, based on the one or more sensors, a location of the wearable headgear unit with respect to the mixed reality environment;
detecting a spatial property of an environment of the user;
identifying, based on the detected spatial property, a predefined first acoustic region;
determining a first acoustic parameter associated with the predefined first acoustic region;
determining, using the first acoustic parameter, a transfer function;
applying the transfer function to the first audio signal to produce a second audio signal; and
presenting, to the user via the speaker, the second audio signal.

13. The system of claim 12 , wherein the first audio signal comprises a waveform audio file.

14. The system of claim 12 , wherein the first audio signal comprises a live audio stream.

15. The system of claim 12 , wherein:
identifying the predefined first acoustic region comprises detecting first sensor input from the one or more sensors and identifying the first acoustic region is based on the first sensor input.

16. The system of claim 15 , wherein determining the first acoustic parameter comprises detecting second sensor input from the one or more sensors and determining the first acoustic parameter is based on the second sensor input.

17. The system of claim 16 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a geometric characteristic of the predefined first acoustic region and determining the first acoustic parameter is based on the geometric characteristic.

18. The system of claim 16 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a material associated with the predefined first acoustic region and determining the first acoustic parameter is based on the material.

19. The system of claim 15 , wherein the first acoustic parameter is determined based on a signal detected by the microphone.

20. The system of claim 12 , wherein the first acoustic parameter corresponds to a reverberation parameter.

21. The system of claim 12 , wherein the first acoustic parameter corresponds to a filtering parameter.

22. The system of claim 12 , further comprising:
detecting a second spatial property of the environment of the user;
identifying, based on the detected second spatial property, a predefined second acoustic region, the predefined second acoustic region acoustically coupled to the predefined first acoustic region; and
determining a second acoustic parameter associated with the predefined second acoustic region,
wherein the transfer function is determined using the second acoustic parameter.

23. An augmented reality system comprising:
a localization subsystem configured to detect a spatial property of an environment of the user, and determine, based on the detected spatial property, an identity of a predefined first space in which the augmented reality system is located;
a communication subsystem configured to communicate the identity of the first space in which the augmented reality system is located and further configured to receive an audio parameter associated with the first space; and
an audio output subsystem configured to process an audio segment based on the audio parameter and further configured to output the audio segment.

24. An augmented reality system comprising:
a sensor subsystem configured to detect a spatial property of an environment of the user, and determine, based on the detected spatial property, information associated with an acoustic property of a predefined first space corresponding to a location of the augmented reality system;
an audio processing subsystem configured to process an audio segment based on the information, the audio processing subsystem communicatively coupled to the sensor subsystem; and
a speaker for presenting the audio segment, the speaker coupled to the audio processing subsystem.

Assignees

• Magic Leap Inc

Inventors

• Schmidt Brian Lloyd
• Tajik Jehangir
• Jot Jean-Marc

Classifications

• H04S7/301

  Automatic calibration of stereophonic sound system, e.g. with test microphone · CPC title

• H04S2420/01

  Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• H04S2400/15

  Aspects of sound capture and related signal processing for recording or reproduction · CPC title

• H04S2400/11

  Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) · CPC title