Video integration with home assistant

Patent metadata

Field	Value
Publication number	US-10200746-B1
Application number	US-201715654491-A
Country	US
Kind code	B1
Filing date	Jul 19, 2017
Priority date	Jul 19, 2017
Publication date	Feb 5, 2019
Grant date	Feb 5, 2019

Abstract

Various arrangements are detailed herein related to managing scheduled events using a home assistant and an integrated video service. Using a network-enabled video camera, a video stream of a location may be captured. The video stream captured may be used to determine an identity of a user present at the location. In response to identifying the identity of the user, access to a cloud-based calendar account for the user may be determined to be authorized. The cloud-based calendar account linked with the user may be accessed. A next calendar event mapped to the cloud-based calendar account that is scheduled to occur within a threshold time duration may be determined. A description linked with the next calendar event may be accessed. A network-enabled smart home assistant device may be used to synthesize speech to announce details related to the next event.

First claim

What is claimed is:

1. A method for managing scheduled events using a home assistant and an integrated video service, the method comprising:
capturing, using a network-enabled video camera, a video stream of a location at which a user is present;
analyzing the video stream captured using the network-enabled video camera to determine an identity of the user;
in response to identifying the identity of the user, determining that access to a cloud-based calendar account for the user is authorized;
accessing the cloud-based calendar account linked with the user;
determining that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;
in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, accessing a description linked with the next calendar event; and
synthesizing speech, using a network-enabled smart home assistant device, to announce a scheduled start time and read a description linked with the next calendar event while the user remains present within the video stream.

2. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
determining that an event location is linked with the next calendar event;
accessing a traffic service to determine a travel time from a home location linked with the network-enabled smart home assistant device or the network-enabled video camera to the event location; and
synthesizing speech, using the network-enabled smart home assistant device, that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time.

3. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
receiving, by the network-enabled smart home assistant device, a spoken command that requests a video clip be recorded; and
storing, to a cloud-based video storage platform, the video clip from the video stream, wherein the video clip is mapped to a user account of the user.

4. The method for managing scheduled events using the home assistant of claim 3 , further comprising:
receiving, by the network-enabled smart home assistant device, a spoken command that requests the video clip be recorded at a time in the future;
scheduling, by the network-enabled smart home assistant device, a recording event based on the spoken command and the time; and
storing, to the cloud-based video storage platform, the video clip from the video stream starting at the time, wherein the video clip is mapped to a user account of the user.

5. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
receiving, by the network-enabled smart home assistant device, a spoken command that requests an image be captured;
synthesizing speech, using the network-enabled smart home assistant device, indicating an amount of time until the image is captured;
capturing the image using the network-enabled video camera; and
storing, to a cloud-based video storage platform, the image, wherein the image is mapped to a user account of the user.

6. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
receiving, by the network-enabled smart home assistant device, a spoken command that requests a time-lapse video clip be recorded, the spoken command specifying a start time and an end time;
capturing, using the network-enabled video camera, a plurality of images between the start time and the end time, wherein a frequency of images captured is dependent on a duration of time between the start time and the end time; and
creating the time-lapse video clip using the plurality of images.

7. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
receiving, by the network-enabled smart home assistant device, a spoken command that requests a history of one or more events at the location;
analyzing a stored video stream for at least a predefined time period;
identifying a plurality of events linked with an audio level above a defined threshold; and
creating a video highlight reel that includes video from the stored video stream recorded during the plurality of events.

8. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
receiving a first spoken query from an identified user;
determining the user is authorized to access the cloud-based calendar account linked with the network-enabled smart home assistant device; and
synthesizing speech, by the network-enabled smart home assistant device, to announce a response to the first spoken query that incorporates information from the cloud-based calendar account linked with the network-enabled smart home assistant device.

9. The method for managing scheduled events using the home assistant of claim 8 , further comprising:
analyzing the video stream captured using the network-enabled video camera to determine an unidentified user is present;
receiving a second spoken query from the unidentified user;
determining the unidentified user is not authorized to access any calendar linked with the network-enabled smart home assistant device; and
synthesizing speech, by the network-enabled smart home assistant device, to announce a generic response to the second spoken query that does not incorporate information from any calendar linked with the network-enabled smart home assistant device.

10. The method for managing scheduled events using the home assistant of claim 1 , further comprising:
receiving a spoken query from an identified user;
determining that the spoken query is requesting an item be added to a stored shopping list;
in response to determining the spoken query, determining a name of the item by performing object recognition on one or more frames of the video stream; and
adding the determined name of the item to the stored shopping list.

11. A system for managing scheduled events, the system comprising:
a network-connected video camera that captures a video stream of a location; and
a cloud-based home assistant integration platform, configured to:
analyze the video stream captured using the network-connected video camera to determine an identity of a user present in the video stream;
in response to identifying the identity of the user, determine that access to a cloud-based calendar account for the user is authorized;
access the cloud-based calendar account linked with the user from a plurality of cloud-based calendar accounts accessible by the cloud-based home assistant integration platform;
determine that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;
in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, access a description linked with the next calendar event; and
transmit an indication of the next calendar event and the accessed description; and

a home assistant device that is configured to:
receive the indication of the next calendar event and the accessed description; and
synthesize speech to announce the next calendar event and the accessed description.

12. The system for managing scheduled events of claim 11 , wherein the home assistant device and the network-connected video camera are incorporated as part of a single device.

13. The system for managing scheduled events of claim 11 , wherein:
the cloud-based home assistant integration

Assignees

• Google Llc

Inventors

• Yuan Jessica
• Stewart James
• Nongpiur Rajeev
• Lister Patrick
• Ng Chi Yeung Jonathan

Classifications

• H04L12/2803

  Home automation networks · CPC title

• H04N21/23418

  involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• H04L12/281

  indicating a format for calling an appliance service function in a home automation network (for remote control or remote monitoring of applications H04L67/025) · CPC title

• G10L17/22

  Interactive procedures; Man-machine interfaces · CPC title

• H04L12/282

  based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204) · CPC title