Video integration with home assistant

Patent metadata

Field	Value
Publication number	US-12081830-B2
Application number	US-202318351370-A
Country	US
Kind code	B2
Filing date	Jul 12, 2023
Priority date	Jul 19, 2017
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

Various arrangements are detailed herein related to managing video recording based on spoken commands. A system receives a video stream from a video camera and analyzes a field of view in the received video stream to determine a location for one or more identified or potential users. The system can beamform audio from microphones of a home assistant device based on the location of the one or more identified or potential users. The system adjusts an audio output based on the location of the one or more identified or potential users, receives a spoken command from the one or more identified or potential users, and outputs a response to the spoken command.

First claim

What is claimed is:

1. A method for handling spoken commands based on characteristics observed by an integrated video service, the method comprising:
receiving a video stream from a streaming video camera;
analyzing a field of view in the received video stream to determine a location for one or more identified or potential users;
beamforming audio from microphones of a home assistant device based on the location of the one or more identified or potential users;
adjusting an audio output based on the location of the one or more identified or potential users;
receiving a spoken command from the one or more identified or potential users; and
outputting a response to the spoken command.

2. The method of claim 1 , wherein the adjusting the audio output comprises:
adjusting an audio output volume based on the location of the one or more identified or potential users.

3. The method of claim 1 , wherein the analyzing of the field of view in the received video stream comprises:
determining a location of the home assistant device; and
determining a distance between the home assistant device and the one or more identified or potential users.

4. The method of claim 1 , further comprising:
analyzing a gaze of the one or more identified or potential users.

5. The method of claim 4 , comprising:
substituting the analyzed gaze of one of the one or more identified or potential users for a hot word or hot phrase.

6. The method of claim 1 , further comprising:
monitoring for movement of the one or more identified or potential users.

7. The method of claim 6 , wherein the monitoring of the movement of the one or more identified or potential users comprises:
monitoring for lip movement of the one or more identified or potential users;
detecting the lip movement by one of the one or more identified or potential users; and
determining that one user of the one or more identified or potential users spoke a command received by the home assistant device.

8. The method of claim 7 , wherein the command is executed based on permissions associated with the user that spoke the command.

9. The method of claim 1 , wherein there are multiple audio output devices and wherein the adjusting the audio output comprises:
selecting an audio output device of the multiple audio output devices based on the location of the one or more identified or potential users.

10. The method of claim 9 , further comprising:
analyzing another field of view in the received video stream to determine another location for the one or more identified or potential users;
beamforming microphones of the home assistant device based on the other location of the one or more identified or potential users; and
adjusting an audio output based on the other location of the one or more identified or potential users.

11. The method of claim 10 , wherein the adjusting the audio output based on the other location comprises:
selecting an audio output device to follow movement of the one or more identified or potential users.

12. A system for handling spoken commands based on characteristics observed by the system, the system comprising:
a streaming video camera;
a cloud-based integrated video service configured to:
receive a video stream from the streaming video camera;
analyze a field of view in the received video stream to determine a location for one or more identified or potential users;
beamform audio from microphones of a home assistant device based on the location of the one or more identified or potential users;
adjust an audio output based on the location of the one or more identified or potential users;
receive a spoken command from the one or more identified or potential users; and
output a response to the spoken command.

13. The system of claim 12 , wherein the adjustment of the audio output configures the cloud-based integrated video service to:
adjust an audio output volume based on the location of the one or more identified or potential users.

14. The system of claim 12 , wherein the cloud-based integrated video service is configured to:
analyze a gaze of the one or more identified or potential users; and
substitute the analyzed gaze of one of the one or more identified or potential users as a substitute for a hot word or hot phrase.

15. The system of claim 12 ,
the cloud-based integrated video service configured to:
monitor for movement of the one or more identified or potential users.

16. The system of claim 15 , wherein the monitoring of the movement of the one or more identified or potential users configures the cloud-based integrated video service to:
monitor for lip movement of the one or more identified or potential users;
detect the lip movement by one of the one or more identified or potential users; and
determine that one user of the one or more identified or potential users spoke a command received by the home assistant device.

17. The system of claim 16 , wherein the command is executed based on permissions associated with the user that spoke the command.

18. The system of claim 12 , wherein there are multiple audio output devices and wherein the adjustment of the audio output configures the cloud-based integrated video service to:
select an audio output device of the multiple audio output devices based on the location of the one or more identified or potential users.

19. The system of claim 18 , wherein the cloud-based integrated video service is configured to:
analyze another field of view in the received video stream to determine another location for the one or more identified or potential users;
beamform microphones of the home assistant device based on the other location of the one or more identified or potential users; and
adjust an audio output based on the other location of the one or more identified or potential users.

20. The system of claim 19 , wherein the adjustment of the audio output based on the other location configures the cloud-based integrated video service to:
select an audio output device to follow movement of the one or more identified or potential users.

Assignees

• Google Llc

Inventors

• Yuan Jessica
• Stewart James
• Nongpiur Rajeev
• Lister Patrick
• Ng Chi Yeung Jonathan

Classifications

• H04L12/282

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title

• H04L12/281

  indicating a format for calling an appliance service function in a home automation network (for remote control or remote monitoring of applications H04L67/025) · CPC title

• H04L12/2803

  Home automation networks · CPC title

• G10L17/22

  Interactive procedures; Man-machine interfaces · CPC title

• H04L67/62

  Establishing a time schedule for servicing the requests · CPC title