Systems and methods of detecting and responding to a visitor to a smart home environment

Patent metadata

Field	Value
Publication number	US-11710387-B2
Application number	US-202217651781-A
Country	US
Kind code	B2
Filing date	Feb 18, 2022
Priority date	Sep 20, 2017
Publication date	Jul 25, 2023
Grant date	Jul 25, 2023

Abstract

A method of detecting and responding to a visitor to a smart home environment via an electronic greeting system of the smart home environment, including determining that a visitor is approaching an entryway of the smart home environment; initiating a facial recognition operation while the visitor is approaching the entryway; initiating an observation window in response to the determination that a visitor is approaching the entryway; obtaining context information from one or more sensors of the smart home environment during the observation window; and at the end of the time window, initiating a response to the detected approach of the visitor based on the context information and/or an outcome of the facial recognition operation.

First claim

What is claimed is:

1. A method comprising:
at an electronic greeting system having one or more processors and memory:
determining whether a person is approaching an entryway;
in response to determining that the person is approaching the entryway, initiating an observation window during which one or more sensors associated with the electronic greeting system sense data relating to the person approaching the entryway;
obtaining identification information, the identification information usable to characterize an identity of the person approaching the entryway;
ending the observation window based on at least one factor including: (i) a predetermined time threshold, (ii) an announcement event, (iii) the identification information, or (iv) input from a user of the electronic greeting system; and
initiating a response based on the identification information.

2. The method as described in claim 1 , wherein the response includes:
one or more audio communication, the one or more audio communication including: (i) an audio communication conveying a status of the user of the electronic greeting system, (ii) an audio communication directing the person to perform an action, (iii) an audio communication directing the person to leave a message, or (iv) an audio communication of a user-composed text message;
one or more audible alarms;
a simulated dog bark;
an activation of one or more lights;
an activation of a door lock;
a notification to law enforcement personnel; or
a notification to the user of the electronic greeting system.

3. The method as described in claim 1 , wherein determining whether the person is approaching the entryway is based on at least one low-power sensor of the one or more sensors.

4. The method as described in claim 3 , wherein initiating the observation window, in response to determining that the person is approaching the entryway based on at least one low-power sensor, activates at least one high-power sensor of the one or more sensors.

5. The method as described in claim 1 , wherein determining whether the person is approaching the entryway is based on a signal transmitted from one or more electronic devices associated with a smart home environment and received at the electronic greeting system, the signal indicative of the person approaching the entryway.

6. The method as described in claim 5 , wherein a camera external to the electronic greeting system transmits the signal to the electronic greeting system indicative of the person approaching the entryway.

7. The method as described in claim 1 , further comprising, capturing, prior to determining whether the person is approaching an entryway, a plurality of images of the person, the plurality of images including at least portions of a face of the person.

8. The method as described in claim 7 , further comprising determining, based on the at least portions of the face of the person in the plurality of images, an increase in dimension in the face of the person between successive images, and wherein determining whether the person is approaching the entryway is based at least in part on determining the increase in dimension in the face of the person between successive images.

9. The method as described in claim 1 , wherein initiating the observation window is further in response to determining that the person approaching the entryway is within a threshold distance.

10. The method as described in claim 9 , wherein the threshold distance is determined based at least in part on quality information associated with sensed data or a prior configuration established by the user of the electronic greeting system.

11. The method as described in claim 1 , wherein the identification information usable to characterize the identity of the person includes one or more of data relating to facial features of the person, data relating to a voice signature of the person, or data relating to behavioral characteristics of the person.

12. The method as described in claim 11 , wherein obtaining identification information usable to characterize the identity of the person comprises performing facial recognition on the sense data to characterize the identity of the person.

13. The method as described in claim 1 , further comprising:
determining that the sensor data is insufficient to perform facial recognition; and
outputting a notification indicative of a request for the person to provide information usable to perform facial recognition.

14. The method as described in claim 13 , wherein the notification requests the person to present their face to the one or more sensors associated with the electronic greeting system.

15. The method as described in claim 1 , wherein initiating the observation window is in response to determining that a plurality of persons are determined to be approaching the entryway.

16. The method as described in claim 15 , wherein obtaining identification information, in response to initiating the observation window, includes obtaining identification information for each of the plurality of persons.

17. The method as described in claim 16 , further comprising, cropping, responsive to obtaining identification information for each of the plurality of persons, images to include at least portions of a face for each of the plurality of persons.

18. The method as described in claim 1 , wherein a predetermined time threshold ending the observation window is based at least in part on a distance of the person to the entryway.

19. An electronic greeting system comprising one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
determining whether a person is approaching an entryway;
in response to determining that the person is approaching the entryway, initiating an observation window during which one or more sensors associated with the electronic greeting system sense data relating to the person approaching the entryway;
obtaining identification information, the identification information characterizing an identity of the person approaching the entryway;
ending the observation window based on at least one factor including: (i) a predetermined time threshold, (ii) an announcement event, (iii) the identification information, or (iv) input from a user of the electronic greeting system; and
initiating a response based on the identification information.

20. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic greeting system, cause the electronic greeting system to:
determine whether a person is approaching an entryway;
in response to the determination that the person is approaching the entryway, initiate an observation window during which one or more sensors associated with the electronic greeting system sense data relating to the person approaching the entryway;
obtain identification information, the identification information characterizing an identity of the person approaching the entryway;
end the observation window based on at least one factor including: (i) a predetermined time threshold, (ii) an announcement event, (iii) the identification information, or (iv) input from a user of the electronic greeting system; and
initiate a response based on the identification information.

Assignees

• Google Llc

Inventors

• Goulden Jason Evans
• Aravamudhan Rengarajan
• Jeong Hae Rim
• Dixon Michael
• Stewart James Edward
• Shafi Sayed Yusef
• Mysore Sahana
• Yang Seungho
• Lien Yu-An
• Burns Christopher Charles
• Nongpiur Rajeev
• Boyd Jeffrey

Classifications

• G08B3/10 Primary

  using electric transmission; using electromagnetic transmission · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06V40/166

  using acquisition arrangements · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title