Systems and methods of detecting and responding to a visitor to a smart home environment

Patent metadata

Field	Value
Publication number	US-10664688-B2
Application number	US-201715809924-A
Country	US
Kind code	B2
Filing date	Nov 10, 2017
Priority date	Sep 20, 2017
Publication date	May 26, 2020
Grant date	May 26, 2020

Abstract

A method of detecting and responding to a visitor to a smart home environment via an electronic greeting system of the smart home environment, including determining that a visitor is approaching an entryway of the smart home environment; initiating a facial recognition operation while the visitor is approaching the entryway; initiating an observation window in response to the determination that a visitor is approaching the entryway; obtaining context information from one or more sensors of the smart home environment during the observation window; and at the end of the time window, initiating a response to the detected approach of the visitor based on the context information and/or an outcome of the facial recognition operation.

First claim

What is claimed is:

1. A method comprising:
at an electronic greeting system of a smart home environment having one or more processors and memory:
determining that a visitor is approaching an entryway of the smart home environment;
in response to the determination that the visitor is approaching the entryway:
(a) initiating a facial recognition operation while the visitor is approaching the entryway; and
(b) initiating an observation window, including:
initiating the observation window before the visitor reaches a physical interaction area of the entryway and/or before the visitor initiates an announcement; and
ending the observation window at the earlier of (i) a predetermined time threshold and (ii) a detected visitor announcement event;

in accordance with the initiating of the observation window, obtaining context information from one or more sensors of the smart home environment during the observation window; and
in accordance with the ending of the observation window, initiating a response to the detected approach of the visitor based on the context information and an outcome of the facial recognition operation.

2. The method of claim 1 , wherein determining that a visitor is approaching the entryway comprises obtaining and analyzing presence information indicative of a visitor approaching or in proximity to the entryway.

3. The method of claim 2 , wherein the presence information is derived from video data obtained from a camera having a field of view corresponding to the entryway; and
wherein deriving the presence information from the video data comprises analyzing a plurality of image frames to determine whether a difference in position of an object in the plurality of image frames exceeds a threshold.

4. The method of claim 2 , wherein the presence information is derived from audio data obtained from an audio sensor in proximity to the entryway, and
wherein deriving the presence information from the audio data comprises identifying: a door knock, a doorbell sound, a footstep, and/or a verbal announcement.

5. The method of claim 1 , wherein detecting a visitor of the smart home environment comprises:
obtaining position data from a sensor of the smart home environment;
identifying, based on analysis of the position data, a position of the visitor with respect to the entryway; and
comparing the position of the visitor to a threshold distance from the entryway, or to a previous position of the visitor.

6. The method of claim 1 , wherein determining that a visitor is approaching the entryway comprises:
obtaining a first measurement of a dimension of a characteristic of the visitor;
obtaining a subsequent measurement of the dimension of the characteristic of the visitor; and
determining whether a difference between the first measurement and the subsequent measurement exceeds a threshold.

7. The method of claim 1 , wherein the context information is based on at least one of a time of day, a day of the week, audio data of the smart home environment, a time of a prescheduled event, a prescheduled status of the smart home environment, a status of a user of the electronic greeting system, a location of the user, and an identity of the user.

8. An electronic greeting system of a smart home environment, the system comprising one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
determining that a visitor is approaching an entryway of the smart home environment;
in response to the determination that the visitor is approaching the entryway:
(a) initiating a facial recognition operation while the visitor is approaching the entryway; and
(b) initiating an observation window, including:
initiating the observation window before the visitor reaches a physical interaction area of the entryway and/or before the visitor initiates an announcement; and
ending the observation window at the earlier of (i) a predetermined time threshold and (ii) a detected visitor announcement event;

in accordance with the initiating of the observation window, obtaining context information from one or more sensors of the smart home environment during the observation window; and
in accordance with the ending of the observation window, initiating a response to the detected approach of the visitor based on the context information and an outcome of the facial recognition operation.

9. The electronic greeting system of claim 8 , wherein the context information comprises a detected announcement event.

10. The electronic greeting system of claim 9 , wherein the detected announcement event comprises: a doorbell button press, a door knock, and/or a verbal announcement.

11. The electronic greeting system of claim 9 , wherein the detected announcement event is part of a pre-assigned doorbell button press pattern, door knock pattern, or verbal announcement associated with a known visitor.

12. The electronic greeting system of claim 8 , wherein the context information is based on:
a facial recognition analysis result;
one or more behavior characteristics of the visitor;
one or more physical characteristics of the visitor;
one or more clothing and/or accessory characteristics of the visitor;
a time of day during which the visitor approaches the entryway;
a day of the week during which the visitor approaches the entryway;
audio data from the smart home environment;
proximity in time to a prescheduled event;
proximity in time to a prescheduled status of the smart home environment;
a known or unknown status of a user of the electronic greeting system;
an expected or unexpected status of a user of the electronic greeting system;
a location of a user of the electronic greeting system;
an identity of a user of the electronic greeting system; and/or
one or more detected visitor actions.

13. The electronic greeting system of claim 8 , wherein the context information is based on one or more detected visitor actions including:
a doorbell activation;
a door knock;
an audio announcement; and/or
an interaction between the visitor and the electronic greeting system.

14. The electronic greeting system of claim 8 , wherein the context information is further based on a timing comparison between one or more detected visitor actions and the initial detection of the visitor.

15. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:
determine that a visitor is approaching an entryway of a smart home environment;
in response to the determination that the visitor is approaching the entryway:
(a) initiate a facial recognition operation while the visitor is approaching the entryway; and
(b) initiate an observation window, including:
initiating the observation window before the visitor reaches a physical interaction area of the entryway and/or before the visitor initiates an announcement; and
ending the observation window at the earlier of (i) a predetermined time threshold and (ii) a detected visitor announcement event;

in accordance with the initiating of the observation window, obtain context information from one or more sensors of the smart home environment during the observation window; and
in accordance with the ending of the observation window, initiate a response to the detected approach of the visitor based on the context information and an outcome of the facial recognition operation.

Assignees

• Google Llc

Inventors

• Goulden Jason Evans
• Aravamudhan Rengarajan
• Jeong Haerim
• Dixon Michael
• Stewart James Edward
• Shafi Sayed Yusef
• Mysore Sahana
• Yang Seungho
• Lien Yu-An
• Burns Christopher Charles
• Nongpiur Rajeev
• Boyd Jeffrey

Classifications

• H04M11/025

  Door telephones (adapted for television H04N7/186) · CPC title

• G08B13/19695

  Arrangements wherein non-video detectors start video recording or forwarding but do not generate an alarm themselves · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G08B13/19684

  Portable terminal, e.g. mobile phone, used for viewing video remotely · CPC title

• G08B13/19656

  Network used to communicate with a camera, e.g. WAN, LAN, Internet · CPC title