Methods and systems for detecting persons in a smart home environment

Patent metadata

Field	Value
Publication number	US-11011035-B2
Application number	US-201715712039-A
Country	US
Kind code	B2
Filing date	Sep 21, 2017
Priority date	Jul 7, 2014
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

The various implementations described herein include methods, devices, and systems for detecting motion and persons. In one aspect, a method is performed at a smart home system that includes a video camera, a server system, and a client device. The video camera captures video and audio, and wirelessly communicates, via the server system, the captured data to the client device. The server system: (1) receives and stores the captured data from the video camera; (2) determines whether an event has occurred, including detected motion; (3) in accordance with a determination that the event has occurred, identifies video and audio corresponding to the event; and (4) classifies the event. The client device receives information indicative of the identified events, displays a user interface for reviewing the video and audio stored by the remote server system, and displays the at least one classification for the event.

First claim

What is claimed is:

1. A smart home system, comprising:
a video camera for use in a smart home environment, the video camera including:
an image sensor having a field of view and being configured to capture video within the field of view;
a processor configured to:
detect a state change of the video camera,
wherein the state change comprises a change in one or more settings or modes that are associated with an operation state of the video camera, and
wherein the state change modifies one or more pixel values in the captured video;
in response to detecting the state change, suppress motion processing during the state change;
detect motion events from the captured video, the motion events being associated with motion of transient objects through the field of view of the image sensor;
identify false positives in the motion events based on in-camera information, the false positives comprising artifacts that occur in the field of view of the image sensor and that are not related to the motion of transient objects; and
generate motion alerts from the captured video, wherein the motion alerts exclude the false positives and the first state change; and

a wireless transceiver configured to wirelessly communicate, via a server system, the captured video to a client device;

the server system including at least one first processor and a first memory storing first programs executable by the at least one first processor, the first programs including a first application configured to:
receive and store the captured video from the video camera;
receive the motion alerts from the video camera;
determine whether an event has occurred, including determining whether motion is detected in the received video;
in response to determining that the event has occurred, identify video corresponding to the event; and
classify the event into at least one classification of a plurality of classifications, the classifications including motion detection and person detection; and

the client device including at least one second processor and a second memory storing second programs executable by the at least one second processor, the second programs including a second application configured to:
display, on a display of the client device, a user interface for reviewing the video and audio stored by the server system;
enable a user of the client device to define a plurality of zones of interest within the field of view of the displayed video, including a first zone of interest and a second zone of interest;
transmit, to the server system, the zones of interest;
receive, from the server system, information indicative of a plurality of events identified by the server system; and
display within the user interface, for each event of the plurality of events, the at least one classification for the respective event, including displaying an indication of whether the respective event involves one or more of the zones of interest,
wherein for at least one event of the plurality of events, both motion detection and person detection classifications are displayed.

2. The smart home system of claim 1 , wherein the first application is further configured to:
receive, from the client device, information regarding the zones of interest; and
determine whether the event involves one or more of the zones of interest, and
wherein the at least one classification for the event is based on whether the event involves one or more of the zones of interest.

3. The smart home system of claim 1 ,
wherein the first application is further configured to identify one or more objects within the field of view based on the captured video,
wherein the plurality of classifications includes an object detection classification, and
wherein the second application is further configured to display the object detection classification in accordance with identification of the one or more objects.

4. The smart home system of claim 1 ,
wherein the first application is further configured to determine whether the event involves one or more animals,
wherein the plurality of classifications includes an animal detection classification, and
wherein the second application is further configured to display the animal detection classification in accordance with a determination that the event involves the one or more animals.

5. The smart home system of claim 1 ,
wherein the first application is further configured to determine whether the event involves one or more machines,
wherein the plurality of classifications includes a machine detection classification, and
wherein the second application is further configured to display the machine detection classification in accordance with a determination that the event involves the one or more machines.

6. The smart home system of claim 5 , wherein the one or more machines includes one or more cars.

7. The smart home system of claim 1 ,
wherein the first application is further configured to determine whether the event involves at least one entryway of a plurality of entryways of the smart home environment,
wherein the plurality of classifications includes an entryway detection classification, and
wherein the second application is further configured to display the entryway detection classification in accordance with a determination that the event involves the at least one entryway.

8. The smart home system of claim 7 , wherein the plurality of entryways includes at least one of (i) one or more doors of the smart home environment or (ii) one or more windows of the smart home environment.

9. The smart home system of claim 1 , wherein the change in the one or more settings or modes comprises a change in a digital pan tilt zoom (DPTZ) setting of the video camera, a change in an infrared illumination setting of the video camera, or a change in an exposure setting of the video camera.

10. The smart home system of claim 1 ,
wherein the false positives in the motion events include a lighting change event,
wherein the processor is configured to identify the lighting change event, and
wherein the motion alerts exclude the lighting change event.

11. The smart home system of claim 10 , wherein the lighting change event comprises a gradual lighting change or a dramatic lighting change.

12. The smart home system of claim 1 ,
wherein the false positives in the motion events include an oscillating change event,
wherein the processor is configured to identify the oscillating change event, and
wherein the motion alerts exclude the oscillating change event.

13. The smart home system of claim 12 , wherein the oscillating change event comprises an oscillation of an object selected from a group consisting of: a tree, a bush, a fan, a curtain, a window screen, a reflection, and a shadow.

14. The smart home system of claim 1 , wherein the second application is further configured to:
display, within the user interface, a notification affordance for each of the plurality of classifications,
wherein each notification affordance is configured to selectively enable or disable notifications for events having the corresponding classification in response to a selection of the respective notification affordance by the user of the client device.

15. The smart home system of claim 14 , wherein the notifications for events having the corresponding classification includes an indication of events on an event timeline within the user interface.

16. A method comprising:
at a video camera in a smart home environment:
capturing video within a field of view of the video camera;
detecting a state change of the video

Assignees

• Google Llc

Inventors

• Laska Jason N
• Nelson Gregory R
• Duffy Greg
• Mitsuji Hiro
• Hill Cameron
• Davidsson Martin
• Montalbo Michael D
• Wan Tung Yuen

Classifications

• H04L63/0428 Primary

  wherein the data content is protected, e.g. by encrypting or encapsulating the payload · CPC title

• G08B13/19604 Primary

  involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change · CPC title

• H04N23/651

  for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera · CPC title

• H04N23/65

  Control of camera operation in relation to power supply · CPC title

• H04N23/6815

  by distinguishing pan or tilt from motion · CPC title