Audio enhanced augmented reality

Patent metadata

Field	Value
Publication number	US-11729573-B2
Application number	US-202117323511-A
Country	US
Kind code	B2
Filing date	May 18, 2021
Priority date	May 18, 2021
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Devices, media, and methods are presented for an audio enhanced augmented reality (AR) experience using an eyewear device. The eyewear device has a microphone system, a presentation system, a support structure configured to be head-mounted on a user, and a processor. The support structure supports the microphone system and the presentation system. The eyewear device is configured to capture, with the microphone system, audio information of an environment surrounding the eyewear device, identify an audio signal within the audio information, detect a direction of the audio signal with respect to the eyewear device, classify the audio signal, and present, by the presentation system, an application associated with the classification of the audio signal.

First claim

The invention claimed is:

1. An eyewear device comprising:
a microphone system;
a presentation system;
a support structure configured to be head-mounted on a user, the support structure supporting the microphone system and the presentation system; and
a processor, a memory, and programming in the memory, wherein execution of the programming by the processor configures the eyewear device to:
capture, with the microphone system, audio information of an environment surrounding the eyewear device;
identify an audio signal within the audio information, wherein to identify the audio signal within the audio information the processor applies a signal discrimination filter to the audio information;
detect a direction of the audio signal with respect to the eyewear device, wherein to detect the direction of the audio signal with respect to the eyewear device the processor applies a beam forming algorithm;
classify the audio signal into one of a plurality of predefined classifications, each of the plurality of predefined classifications associated with a respective application for presentation by the presentation system, wherein to classify the audio signal into one of a plurality of predefined classifications the processor applies a trained convolutional neural network (CNN) to the audio signal;
monitor a direction processing timestamp corresponding to detecting the direction of the audio signal;
monitor a CNN processing timestamp corresponding to applying the trained CNN to the audio signal;
correlate the direction processing timestamp and the CNN processing timestamp; and
present, by the presentation system, the respective application associated with the one of the plurality of predefined classifications responsive to the direction of the audio signal, wherein presenting the respective application associated with the one of the plurality of predefined classifications is further responsive to the correlated CNN processing timestamp and direction processing timestamp.

2. The eyewear device of claim 1 , wherein the presentation system includes a speaker system and wherein to present the respective application the processor configures the eyewear device to:
produce a spatial filter responsive to the direction of the audio signal;
apply the spatial filter to the audio signal;
amplify the audio signal; and
present, with the speaker system, the amplified audio signal with the applied spatial filter.

3. The eyewear device of claim 1 , wherein the presentation system includes a speaker system and wherein to present the respective application the processor configures the eyewear device to:
produce a spatial filter responsive to the direction of the audio signal;
generate an audio track corresponding to the one of the plurality of predefined classifications;
apply the spatial filter to the audio track; and
present, with the speaker system, the audio track with the applied spatial filter.

4. The eyewear device of claim 1 , wherein the presentation system includes a display system and wherein to present the respective application the processor configures the eyewear device to:
produce a visual overlay including a virtual object corresponding to the one of the plurality of predefined classifications; and
present the visual overlay on the display system.

5. The eyewear device of claim 4 , wherein to present the visual overlay the processor configures the eyewear device to:
register the virtual object to the detected direction with respect to the eyewear device;
wherein to produce and present the visual overlay the processor configures the eyewear device to include the virtual object in the visual overlay in a position corresponding to the detected direction.

6. The eyewear device of claim 5 , wherein execution of the programming by the processor further configures the eyewear device to:
detect a subsequent direction of the audio signal; and
produce a subsequent visual overlay including the virtual object in another position corresponding to the subsequent detected direction to present on the display system.

7. The eyewear device of claim 1 , wherein the presentation system includes a speaker system and a display system and wherein to present the respective application the processor configures the eyewear device to:
present, with the speaker system, an audio track in the direction of the audio signal; and
present, with the display system, a virtual object in a direction corresponding to the direction of the audio signal.

8. The eyewear device of claim 1 , wherein the presentation system includes a speaker system and a display system and the eyewear device further comprises:
a camera system having a field of view and supported by the support structure, wherein execution of the programming by the processor further configures the eyewear device to capture images within a field of view;
wherein to present the respective application the processor configures the eyewear device to:
present, with the speaker system, an audio track in the direction of the audio signal;
present, with the display system, a virtual object in a direction corresponding to the direction of the audio signal when an object corresponding to the audio signal is not within the field of view; and
present, with the display system, the virtual object over the object when the object corresponding to the audio track is within the field of view.

9. The eyewear device of claim 1 , wherein execution of the programming by the processor further configures the eyewear device to:
determine an intensity of the audio signal; and
classify the audio into one of at least two intensity classification levels;
wherein the presenting of the respective application is additionally responsive to the classified intensity classification level.

10. A method for use with an eyewear device including a microphone system, a presentation system and a support structure configured to be head-mounted on a user, the method comprising:
capturing, with the microphone system, audio information of an environment surrounding an eyewear device;
identifying an audio signal within the audio information by applying a signal discrimination filter to the audio information;
detecting a direction of the audio signal with respect to the eyewear device by applying a beam forming algorithm;
classifying the audio signal into one of a plurality of predefined classifications, each of the plurality of predefined classifications associated with a respective application for presentation by the presentation system, by applying a trained convolutional neural network (CNN) to the audio signal;
monitoring a direction processing timestamp corresponding to detecting the direction of the audio signal;
monitoring a CNN processing timestamp corresponding to applying the trained CNN to the audio signal;
correlating the direction processing timestamp and the CNN processing timestamp; and
presenting, by the presentation system, the respective application associated with the one of the plurality of predefined classifications responsive to the direction of the audio signal, wherein presenting the respective application associated with the one of the plurality of predefined classifications is further responsive to the correlated CNN processing timestamp and direction processing timestamp.

11. The method of claim 10 , wherein the presentation system includes a speaker system and wherein the presenting comprises:
producing a spatial filter responsive to the direction of the audio signal;
applying the spatial filter to the audio signal;
amplifying the audio signal; and
presenting, with the speaker system, the amplified audio signal with the applied spatial fil

Assignees

• Arya Ashwani
• Snap Inc

Inventors

• Arya Ashwani

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F3/16 Primary

  Sound input; Sound output (speech processing G10L) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title