Systems and methods of presenting appropriate actions for responding to a visitor to a smart home environment

Patent metadata

Field	Value
Publication number	US-11356643-B2
Application number	US-202117400887-A
Country	US
Kind code	B2
Filing date	Aug 12, 2021
Priority date	Sep 20, 2017
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

A method of presenting appropriate actions for responding to a visitor to a smart home environment via an electronic greeting system of the smart home environment, including detecting a visitor of the smart home environment; obtaining context information from the smart home environment regarding the visitor; based on the context information, identifying a plurality of appropriate actions available to a user of a client device for interacting with the visitor via the electronic greeting system; and causing the identified actions to be presented to the user of the client device.

First claim

What is claimed is:

1. A method comprising:
receiving, by a remote server system and from a video doorbell located at an entryway, an indication that a visitor activated the video doorbell;
causing, by the remote server system, the video doorbell to output an audio greeting;
receiving, by the remote server system and from the video doorbell, audio data that includes a voice response to the greeting from the visitor;
determining, by the remote server and based on the voice response, a suggested audio response from a plurality of suggested audio responses; and
causing, by the remote server system, the video doorbell to output the suggested audio response.

2. The method of claim 1 , wherein the voice response is a first voice response, the method further comprising:
after causing the video doorbell to output the suggested audio response, receiving, by the remote server system, a second voice response from the visitor; and
storing, by the remote server system, the second voice response as an audio message.

3. The method of claim 2 , further comprising:
converting, by the remote server system, the audio message to text using speech-to-text; and
sending, by the remote server system and to a device associated with a user associated with the video doorbell, the text of the audio message.

4. The method of claim 1 ,
wherein the voice response includes an indication that the visitor is dropping off a package, and
wherein the suggested audio response includes an indication of a location to leave the package.

5. The method of claim further comprising: receiving, by the remote server system and from the video doorbell, video data captured by the video doorbell;
determining, by the remote server system and. based on the video data, that the visitor is approaching an entryway at which the video doorbell is located.

6. The method of claims, further comprising: responsive to determining that the visitor is approaching, causing the video doorbell, to initiate an observation window; during the observation window: continuously receiving, by the remote server system and from the video doorbell, additional video data; and determining, by the remote server system and using facial recognition, whether the visitor is a known visitor based On the additional video data.

7. The method of claim 6 , further comprising:
responsive to determining that the visitor is a known visitor, determining whether a pre-assigned verbal announcement is associated with the known visitor; and
responsive to determining that the pre-assigned verbal announcement is associated with the known visitor, determining that the pre-assigned verbal announcement is the audio greeting.

8. The method of claim 6 , wherein determining the suggested audio response is further based on whether the visitor is the known visitor.

9. The method of claim 5 , further comprising:
responsive to determining that the visitor is approaching the entryway, determining, by the remote server system, context information associated with the visitor, wherein the context information includes one or more of a facial recognition analysis result of the visitor, one or more behavioral characteristics of the visitor, one or more physical characteristics of the visitor, one or more clothing or accessory characteristics of the visitor, a time of day during which the visitor approaches the entryway, a day of the week during which the visitor approaches the entryway, proximity of a time at which the visitor approaches the entryway to a time of a prescheduled event, proximity of a time at which the visitor approaches the entryway to a time of a prescheduled status, a known or an unknown status of a user associated with the video doorbell, an expected or an unexpected status of the user,
wherein determining the suggested audio response from the plurality of suggested audio responses is further based on the context information, wherein the plurality of suggested audio responses includes one or more of a status of the user, a request for the visitor to perform an action, a request for the visitor to leave a message, a preprogrammed customized communication, and an audio version of a user-composed text message.

10. A computing system comprising:
one or more processors;
a memory that stores instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
receive, from a video doorbell, an indication that a visitor activated the video doorbell;
cause the video doorbell to output an audio greeting;
receive, from the video doorbell, audio data that includes a voice response to the greeting from the visitor;
determine, based on the voice response, a suggested audio response from a plurality of suggested audio responses; and
cause the video doorbell to output the suggested audio response.

11. The computing system of claim 10 , wherein the voice response is a first voice response, and wherein the instructions further cause the one or more processors to:
after causing the video doorbell to output the suggested audio response, receive a second voice response from the visitor; and
store the second voice response as an audio message.

12. The computing system of claim 11 , further comprising:
convert the audio message to text using speech-to-text; and
send, to a device associated with a user associated with the video doorbell, the text of the audio message.

13. The computing system of claim 10 ,
wherein the voice response includes an indication that the visitor is dropping off a package; and
wherein the suggested audio response includes an indication of a location to leave the package.

14. The computing system of claim wherein the instructions further cause the one or more processors to: receive, from the video doorbell, video data captured by the video doorbell; determine, based on the video data, that the visitor is approaching an entryway at which the video doorbell is located.

15. The computing system of claim 14 , wherein the instructions further cause the one or more processors to: responsive to determining that the visitor is approaching, cause the video doorbell, to initiate an observation window; during the observation window:
continuously receive, from the video doorbell, additional video data; and determine, using facial recognition, whether the visitor is a known visitor based on the additional video data.

16. The computing system of claim 15 , wherein the instructions further cause the one or more processors to:
responsive to determining that the visitor is a known visitor, determine whether a pre-assigned verbal announcement is associated with the known visitor; and
responsive to determining that the pre-assigned verbal announcement is associated with the known visitor, determine that the pre-assigned verbal announcement is the audio greeting.

17. The computing system of claim 14 , wherein the instructions further cause the one or more processors to:
responsive to determining that the visitor is approaching the entryway, determine context information associated with the visitor, wherein the context information includes one or more of a facial recognition analysis result of the visitor, one or more behavioral characteristics of the visitor, one or more physical characteristics of the visitor, one or more clothing or accessory characteristics of the visitor, a time of day during which the visitor approaches the entryway, a day of the week during which the visitor approaches the entryway, proximity of a time at which the visitor approaches the entryway to a time of a presched

Assignees

• Google Llc

Inventors

• Goulden Jason Evans
• Aravamudhan Rengarajan
• Jeong Hae Rim
• Dixon Michael
• Stewart James Edward
• Shafi Sayed Yusef
• Mysore Sahana
• Yang Seungho
• Lien Yu-An
• Burns Christopher Charles
• Nongpiur Rajeev Conrad
• Boyd Jeffrey

Classifications

• G08B3/10 Primary

  using electric transmission; using electromagnetic transmission · CPC title

• H04N7/186 Primary

  Video door telephones · CPC title

• H04M11/02

  with bell or annunciator systems · CPC title

• G05B15/02

  electric · CPC title

• G08B13/19695

  Arrangements wherein non-video detectors start video recording or forwarding but do not generate an alarm themselves · CPC title