Response to sounds in an environment based on correlated audio and user events

Patent metadata

Field	Value
Publication number	US-12307012-B2
Application number	US-202218047494-A
Country	US
Kind code	B2
Filing date	Oct 18, 2022
Priority date	Oct 27, 2021
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

The disclosed systems and method correlates user behaviors with audio processing to achieve more accurate conclusions about sounds in a user's environment. These conclusions may, in turn, be used to adjust the way a device, such as AR glasses, operate or respond to the sounds. For example, audio events determined from processing speech can be correlated with behavior events determined by sensing a user to improve a speech-to-text transcript of the speech by separating, or otherwise altering, the text in the transcript by speaker.

First claim

The invention claimed is:

1. A method comprising:
capturing audio of a conversation;
analyzing the audio to detect a speaker change event with a first confidence level;
analyzing a feature of a user to detect a behavior event;
detecting a correlation between the speaker change event and the behavior event using a machine learning model;
adjusting the first confidence level of the speaker change event to a second confidence level based on the correlation; and
generating a change in a speech-to-text transcript of the conversation based on the second confidence level of the speaker change event.

2. The method according to claim 1 , wherein capturing the audio is by a microphone array of AR glasses worn by the user.

3. The method according to claim 1 , wherein the speaker change event corresponds to when a first person in the conversation has stopped speaking and a second person in the conversation has started speaking.

4. The method according to claim 1 , wherein analyzing the audio includes speaker localization, based on a sensitivity of a microphone array.

5. The method according to claim 1 , further comprising sensing the user to measure the feature of the user using an inertial measurement unit of AR glasses worn by the user.

6. The method according to claim 1 , further comprising sensing the user to measure the feature of the user using an eye tracker of AR glasses worn by the user.

7. The method according to claim 1 , further comprising sensing the user to measure the feature of the user using a galvanic skin response sensor of a device worn on a wrist of the user.

8. The method according to claim 1 , further comprising sensing the user to measure the feature of the user using a photoplethysmography sensor of a device worn on a wrist of the user.

9. The method according to claim 1 , wherein the feature of the user is included in a plurality of features of the user, the plurality of features including a position of a head of the user.

10. The method according to claim 9 , wherein the plurality of features of the user include a gaze of the user.

11. The method according to claim 10 , wherein the behavior event is a change in attention of the user.

12. The method according to claim 1 , wherein the feature of the user is included in a plurality of features of the user, the plurality of features including a pupil size of one eye, or both eyes, of the user.

13. The method according to claim 12 , wherein the behavior event is a change in cognitive load of the user.

14. The method according to claim 1 , wherein the feature of the user is included in a plurality of features of the user the plurality of features including a skin conductance of the user.

15. The method according to claim 14 , wherein the plurality of features of the user include a heart rate of the user.

16. The method according to claim 15 , wherein the behavior event a surprise of the user.

17. The method according to claim 1 , wherein the machine learning model is a neural network.

18. The method according to claim 1 , wherein the machine learning model is a state vector machine (SVM) or a random decision forest.

19. The method according to claim 3 , wherein the change includes:
inserting a line break, changing a color, or adding tag at the speaker change event in the speech-to-text transcript to separate speakers of the conversation.

20. An augmented reality device comprising:
a microphone array configured to capture audio of a conversation;
an inertial measurement unit configured to measure a position of a head of a user;
an eye tracker configured to measure a gaze of an eye of the user;
a heads-up display configured to display a speech-to-text transcript of the conversation to the user; and
a processor configured by software instructions to:
analyze the audio to detect a speaker change event with a first confidence level;
analyze the position of the head of the user and the gaze of the eye of the user to detect a behavior event;
detect a correlation between the speaker change event and the behavior event using a machine learning model;
adjust the first confidence level of the speaker change event to a second confidence level based on the correlation; and
generate a change in the speech-to-text transcript based on the second confidence level of the speaker change event.

21. The augmented reality device according to claim 20 , wherein the change includes:
inserting a line break, changing a color, or adding tag at the speaker change event in the speech-to-text transcript to separate speakers of the conversation.

22. The augmented reality device according to claim 20 , wherein the processor is further configured by the software instructions to:
adjust a beam forming of the microphone array of the augmented reality device based on the second confidence level of the speaker change event.

23. The augmented reality device according to claim 20 , wherein the processor is further configured by the software instructions to:
adjust a noise cancellation of the augmented reality device based on the second confidence level of the speaker change event.

Assignees

• Google Llc

Inventors

• Olwal Alex
• Du Ruofei

Classifications

• G06F3/165

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B27/017

  Head mounted · CPC title