Personalized, real-time audio processing

Patent metadata

Field	Value
Publication number	US-11715482-B2
Application number	US-201916708176-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Feb 2, 2016
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

An apparatus and method for real-time audio processing employs a gaze detection sensor to detect a direction of a user's gaze and output a gaze signal corresponding to the detected direction of the user's gaze. A digital signal processing unit responds to a plurality of signals corresponding to a plurality of sounds received at the apparatus, and the determined direction of gaze to identify a signal of interest from the plurality of signals using the gaze signal. The signal of interest is processed for output to the user. In embodiments, a microphone array provides the plurality of signals. An imaging sensor may work with either the microphone array or the gaze detection sensor to identify the signal of interest.

First claim

What is claimed is:

1. A computer-implemented method performed by a user wearable device, the method comprising:
detecting a direction of a gaze of a user based at least in part on monitoring user head motion;
identifying, in an electronic data storage, one or more available actions that correspond to an audio source in the direction of the gaze of the user;
selecting a first action from the one or more available actions based at least in part on the one or more available actions that correspond to the audio source in the direction of the gaze of the user, wherein the first action comprises emphasizing a first sound from the audio source in the direction of the gaze of the user and performing a noise cancelling algorithm on a second sound associated with a second audio source different than the audio source, wherein the audio source is in the direction of the gaze of the user and the second audio source is in a direction different than the direction of the gaze of the user; and
executing a set of computer-readable instructions that correspond to the first action.

2. The method of claim 1 , wherein detecting the direction of the gaze of the user comprises:
detecting the direction of the gaze of the user based at least in part on monitoring a position of an eye of the user.

3. The method of claim 1 , wherein detecting the direction of the gaze of the user comprises:
detecting the direction of the gaze of the user based at least in part on an image.

4. The method of claim 1 , wherein selecting the first action from the one or more available actions comprises:
selecting to adjust playback of audio, video, or both, based at least in part on the direction of the gaze of the user.

5. The method of claim 1 , wherein selecting the first action from the one or more available actions comprises:
selecting to enable or disable a feature for an application during runtime of the application based at least in part on the direction of the gaze of the user.

6. The method of claim 1 , wherein selecting the first action from the one or more available actions comprises:
selecting the first action from the one or more available actions based at least in part on a verbal command.

7. The method of claim 1 , further comprising:
tracking the direction of the gaze of the user over time to generate an updated direction of the gaze of the user.

8. The method of claim 7 , further comprising:
identifying, in the electronic data storage, a second one or more available actions that correspond to the updated direction of the gaze of the user;
selecting a second action from the second one or more available actions based at least in part on the updated direction of the gaze of the user; and
executing a second set of computer-readable instructions that correspond to the second action.

9. The method of claim 1 , wherein performing the noise cancelling algorithm on the second sound associated with the second audio source comprises:
capturing a frequency profile associated with the second sound; and
inverting the frequency profile.

10. A user wearable device, comprising:
a sensor configured to detect a direction of a gaze of a user based at least in part on monitoring user head motion; and
a digital signal processing unit configured to:
identify, in an electronic data storage, one or more available actions that correspond to an audio source in the direction of the gaze of the user;
select a first action from the one or more available actions based at least in part on the one or more available actions that correspond to the audio source in the direction of the gaze of the user, wherein the first action comprises emphasizing a first sound from the audio source in the direction of the gaze of the user and performing a noise cancelling algorithm on a second sound associated with a second audio source different than the audio source, wherein the audio source is in the direction of the gaze of the user and the second audio source is in a direction different than the direction of the gaze of the user; and
execute a set of computer-readable instructions that correspond to the first action.

11. The user wearable device of claim 10 , wherein the sensor is configured to:
detect the direction of the gaze of the user based at least in part on monitoring a position of an eye of the user.

12. The user wearable device of claim 10 , wherein the sensor is configured to:
detect the direction of the gaze of the user based at least in part on an image.

13. The user wearable device of claim 10 , wherein to select the first action from the one or more available actions the digital signal processing unit is configured to:
select to adjust playback of audio, video, or both, based at least in part on the direction of the gaze of the user.

14. The user wearable device of claim 10 , wherein to select the first action from the one or more available actions the digital signal processing unit is configured to:
select to enable or disable a feature for an application during runtime of the application based at least in part on the direction of the gaze of the user.

15. The user wearable device of claim 10 , wherein to select the first action from the one or more available actions the digital signal processing unit is configured to:
select the first action from the one or more available actions based at least in part on a verbal command.

16. The user wearable device of claim 10 , wherein the digital signal processing unit is further configured to:
track the direction of the gaze of the user over time to generate an updated direction of the gaze of the user.

17. The user wearable device of claim 16 , wherein the digital signal processing unit is further configured to:
identify, in the electronic data storage, a second one or more available actions that correspond to the updated direction of the gaze of the user;
select a second action from the second one or more available actions; and
execute a second set of computer-readable instructions that correspond to the second action.

18. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
detecting a direction of a gaze of a user based at least in part on monitoring user head motion;
identifying, in an electronic data storage, one or more available actions that correspond to an audio source in the direction of the gaze of the user;
selecting a first action from the one or more available actions based at least in part on the one or more available actions that correspond to the audio source in the direction of the gaze of the user, wherein the first action comprises emphasizing a first sound from the audio source in the direction of the gaze of the user and performing a noise cancelling algorithm on a second sound associated with a second audio source different than the audio source, wherein the audio source is in the direction of the gaze of the user and the second audio source is in a direction different than the direction of the gaze of the user; and
executing a set of computer-readable instructions that correspond to the first action.

19. The non-transitory machine-readable medium of claim 18 , wherein detecting the direction of the gaze of the user comprises instructions executable to cause the machine to perform:
detecting the direction of the gaze of the user based at least in part on monitoring a position of an eye of the user.

20. The non-transitory machine-readable medium of claim

Assignees

• Ebay Inc

Inventors

• Glasgow Dane
• Ramadge David
• Maclaurin Matthew Bret
• Van Winkle Justin
• Sherman Corinne Elizabeth

Classifications

• G10L21/0208

  Noise filtering · CPC title

• G10L21/028 Primary

  using properties of sound source · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G02B27/022

  Viewing apparatus (G02B27/04, G02B27/06, G02B27/08 take precedence) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title