Technique for directing audio in augmented reality system

Patent metadata

Field	Value
Publication number	US-10536783-B2
Application number	US-201715423415-A
Country	US
Kind code	B2
Filing date	Feb 2, 2017
Priority date	Feb 4, 2016
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

A hearing aid comprises a sensor configured for detecting a focus of an end user on a real sound source, a microphone assembly configured for converting sounds into electrical signals, a speaker configured for converting the electrical signals into sounds, and a control subsystem configured for modifying the direction and/or distance of a greatest sensitivity of the microphone assembly based on detected focus. A virtual image generation system comprises memory storing a three-dimensional scene, a sensor configured for detecting a focus of the end user on a sound source, a speaker configured for conveying sounds to the end user, and a control subsystem configured for causing the speaker to preferentially convey a sound originating from the sound source in response to detection of the focus, and for rendering image frames of the scene, and a display subsystem configured for sequentially displaying the image frames to the end user.

First claim

The invention claimed is:

1. A hearing aid for use by an end user, comprising:
at least one sensor configured for detecting a focus of the end user on a real sound source;
an adaptive microphone assembly configured for converting sounds into electrical signals;
at least one speaker configured for converting the electrical signals to sounds for perception by the end user; and
a control subsystem configured for modifying, based on
the detected focus of the end user,
a direction of a greatest sensitivity of the adaptive microphone assembly, and
a distance of the greatest sensitivity of the adaptive microphone assembly,

wherein the control subsystem is configured for processing the electrical signals to compare characteristics of a first sound originating from the real sound source to characteristics of a second sound originating from a different source, emphasizing sounds having the same type of characteristics as the characteristics of the first sound, and deemphasizing sounds having the same type of characteristics as the second sound.

2. The hearing aid of claim 1 , wherein the adaptive microphone assembly is configured for converting the sounds into the electrical signals in accordance with one of a dipole, cardioid, super-cardioid, or hyper-cardioid pattern.

3. The hearing aid of claim 1 , wherein the control subsystem is configured for setting the greatest sensitivity of the adaptive microphone assembly to a direction and a distance of the detected focus of the end user.

4. The hearing aid of claim 1 , wherein the at least one sensor comprises a camera.

5. The hearing aid of claim 1 , wherein the at least one sensor is configured for detecting an angular position of an eye of the end user to detect a direction of the focus of the end user.

6. The hearing aid of claim 1 , wherein the at least one sensor is configured for detecting a convergence of eyes of the end user to detect a distance of the focus of the end user.

7. The hearing aid of claim 1 , wherein the at least one sensor is configured for detecting an angular position of a head of the end user to detect the direction of the focus of the end user.

8. The hearing aid of claim 1 , wherein the adaptive microphone assembly comprises a phased array of microphone elements.

9. The hearing aid of claim 1 , further comprising a frame structure configured for being worn by the end user, the frame structure carrying the at least one sensor, the microphone assembly, and the at least one speaker.

10. The hearing aid of claim 1 , wherein the control subsystem comprises a digital signal processor (DSP) configured for modifying the direction and/or distance of the greatest sensitivity of the adaptive microphone assembly.

11. The hearing aid of claim 1 , wherein the control subsystem is configured for processing the electrical signals to localize the sounds, such that a sound originating from the real sound source is perceived by the end user as originating from the real sound source.

12. The hearing aid of claim 1 , wherein the control subsystem is configured for processing the electrical signals to selectively raising or lowering a volume of certain frequency components of the sounds, such that a sound originating from the real sound source is emphasized and/or a sound originating from a different sound source is deemphasized.

13. A virtual image generation system for use by an end user, comprising:
memory storing a three-demensional scene;
the hearing aid of claim 1 , wherein the control subsystem is further configured for rendering a plurality of synthetic image frames of the three-dimensional scene; and
a display subsystem configured for sequentially displaying the plurality of image frames to the end user.

14. The virtual image generation system of claim 13 , wherein the display subsystem is configured to being positioned in front of the eyes of the end user.

15. The vitual image generation system of claim 13 , wherein the display subsystem includes a projection subsystem and a partially transparent display surface, the projection subsystem configured for projecting the image frames on the partially transparent display surface, and the partially transparent display surface is configured for being positioned in the field of view between the eyes of the end user and an ambient environment.

16. The virtual image generation system of claim 13 , further comprising a frame structure configured for being worn by the end user, the frame structure carrying the at least one sensor, the adaptive microphone assembly, the at least one speaker, and the display subsystem.

17. The virtual image generation system of claim 13 , wherein the control subsystem comprises a graphics control subsystem unit (GPU).

18. The hearing aid of claim 1 , wherein the type of characteristics of the first sound is music.

Assignees

• Magic Leap Inc

Inventors

• Sanger George Alistair
• Miller Samuel A
• Schmidt Brian
• Tajik Anastasia Andreyevna

Classifications

• H04R25/505 Primary

  using digital signal processing · CPC title

• H04R1/028

  associated with devices performing functions other than acoustics, e.g. electric candles · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04R2499/11

  Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's · CPC title