Camera detection of human activity with co-occurrence

Patent metadata

Field	Value
Publication number	US-11580833-B2
Application number	US-202117202528-A
Country	US
Kind code	B2
Filing date	Mar 16, 2021
Priority date	Mar 24, 2020
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

Methods, systems, and apparatus for camera detection of human activity with co-occurrence are disclosed. A method includes detecting a person in an image captured by a camera; in response to detecting the person in the image, determining optical flow in portions of a first set of images; determining that particular portions of the first set of images satisfy optical flow criteria; in response to determining that the particular portions of the first set of images satisfy optical flow criteria, classifying the particular portions of the first set of images as indicative of human activity; receiving a second set of images captured by the camera after the first set of images; and determining that the second set of images likely shows human activity based on analyzing portions of the second set of images that correspond to the particular portions of the first set of images classified as indicative of human activity.

First claim

What is claimed is:

1. A method, comprising:
detecting a person depicted in an image captured by a camera;
in response to detecting the person depicted in the image captured by the camera, determining optical flow in pixel groups of a first set of images that are captured by the camera;
determining that particular pixel groups of the first set of images satisfy optical flow criteria that i) indicate flow motion characteristics of human activity and ii) include criteria for at least one of optical flow magnitude or optical flow direction of the pixel groups of the first set of images, each of the particular pixel groups having the same location in each image of the first set of images;
in response to determining that the particular pixel groups of the first set of images satisfy the optical flow criteria, classifying the particular pixel groups of the first set of images as indicative of human activity;
receiving a second set of images captured by the camera after the first set of images;
analyzing second pixel groups of the second set of images that correspond to the locations of the particular pixel groups of the first set of images classified as indicative of human activity; and
determining that the second set of images likely shows human activity using a result of the analysis of the second pixel groups of the second set of images that correspond to the locations of the particular pixel groups of the first set of images classified as indicative of human activity.

2. The method of claim 1 , wherein determining that the second set of images likely shows human activity based on analyzing the pixel groups of the second set of images that correspond to the particular pixel groups of the first set of images classified as indicative of human activity comprises detecting optical flow in the pixel groups of the second set of images that correspond to the particular pixel groups of the first set of images that are classified as indicative of human activity.

3. The method of claim 1 , wherein classifying the particular pixel groups of the first set of images as indicative of human activity comprises determining that the particular pixel groups of the first set of images depict movement of an object that moves in co-occurrence with human motion.

4. The method of claim 3 , wherein determining that the second set of images likely shows human activity based on analyzing the pixel groups of the second set of images that correspond to the particular pixel groups of the first set of images classified as indicative of human activity comprises determining, based on detecting motion of the object that moves in co-occurrence with human motion, that the second set of images likely shows human activity.

5. The method of claim 1 , wherein classifying the particular pixel groups of the first set of images as indicative of human activity comprises determining that the particular pixel groups of the first set of images correspond to a human trajectory through a scene captured by the camera.

6. The method of claim 5 , wherein determining that the second set of images likely shows human activity based on analyzing portions of the second set of images that correspond to the particular pixel groups of the first set of images classified as indicative of human activity comprises detecting motion along the human trajectory through the scene captured by the camera.

7. The method of claim 1 , wherein determining that the second set of images likely shows human activity based on analyzing portions of the second set of images that correspond to the particular pixel groups of the first set of images classified as indicative of human activity comprises determining that a matching percentage between portions of the second set of images that exhibit optical flow and the particular pixel groups of the first set of images exceeds a threshold matching percentage.

8. The method of claim 1 , comprising generating a bounding box around the detected person, wherein classifying the particular pixel groups of the first set of images as indicative of human activity comprises:
determining that the particular pixel groups of the first set of images have less than a threshold overlap with the bounding box.

9. The method of claim 1 , wherein the particular pixel groups of the first set of images comprise segments of a grid overlaid on each image of the first set of images, the method comprising generating a gridded representation of the particular pixel groups of the first set of images that are classified as indicative of human activity.

10. The method of claim 9 , wherein the gridded representation includes binary representations indicating whether each portion of the first set of images is indicative of human activity.

11. The method of claim 9 , wherein the gridded representation includes gradient representations indicating a degree to which each portion of the first set of images is indicative of human activity.

12. The method of claim 1 , wherein the first set of images includes consecutive images captured by the camera, the consecutive images including the image in which the person was detected.

13. The method of claim 12 , wherein a first image of the consecutive images is the image in which the person was detected.

14. The method of claim 12 , wherein a final image of the consecutive images is the image in which the person was detected.

15. The method of claim 1 , wherein determining optical flow in portions of the first set of images comprises comparing pixel values in corresponding portions of consecutive images.

16. The method of claim 1 , comprising:
in response to determining that the second set of images likely shows human activity based on analyzing portions of the second set of images that correspond to the particular pixel groups of the first set of images classified as indicative of human activity, generating a notification that indicates that human activity was likely detected.

17. The method of claim 1 , comprising:
detecting a non-human object in a second image captured by the camera;
in response to detecting the non-human object in the second image captured by the camera, determining optical flow in portions of a third set of images that are captured by the camera, wherein the third set of images includes the second image;
determining that particular pixel groups of the third set of images satisfy optical flow criteria;
in response to determining that the particular pixel groups of the third set of images satisfy optical flow criteria, classifying the particular pixel groups of the third set of images as indicative of non-human object motion;
receiving a fourth set of images captured by the camera after the third set of images; and
determining that the fourth set of images likely shows non-human object motion based on analyzing portions of the fourth set of images that correspond to the particular pixel groups of the third set of images classified as indicative of non-human object motion.

18. A monitoring system for monitoring a property, the monitoring system comprising one or more computers configured to perform operations comprising:
detecting a person depicted in an image captured by a camera;
in response to detecting the person depicted in the image captured by the camera, determining optical flow in pixel groups of a first set of images that are captured by the camera;
determining that particular pixel groups of the first set of images satisfy optical flow criteria that i) indicate flow motion characteristics of human activity and ii) include criteria for at least one of

Assignees

• Objectvideo Labs Llc
• Object Video Labs Llc

Inventors

• Ramanathan Narayanan
• Beach Allison
• Qian Gang
• Madden Donald Gerard

Classifications

• H04N23/611

  where the recognised objects include parts of the human body · CPC title

• G08B13/19626

  optical details, e.g. lenses, mirrors or multiple lenses · CPC title

• H04N7/183 Primary

  for receiving images from a single remote source · CPC title

• G08B13/19608

  Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position · CPC title

• H04N23/71

  Circuitry for evaluating the brightness variation · CPC title