System and method for monitoring a retail environment using video content analysis with depth sensing

Patent metadata

Field	Value
Publication number	US-9740937-B2
Application number	US-201313744251-A
Country	US
Kind code	B2
Filing date	Jan 17, 2013
Priority date	Jan 17, 2012
Publication date	Aug 22, 2017
Grant date	Aug 22, 2017

Abstract

A method and system for monitoring a retail environment by performing video content analysis based on two-dimensional image data and depth data are disclosed. Accuracy in customer actions to provide assistance, change marketing behavior, safety and theft, for example, is increase by analyzing video containing two-dimensional image data and associated depth data. Height data may be obtained from depth data to assist in object detection, object classification (e.g., detection a customer or inventory) and/or event detection.

First claim

What is claimed is:

1. A method of monitoring a retail environment comprising:
taking a video at a store with a video sensor, the video comprising a plurality of frames, each frame including two-dimensional (2D) image data;
for each frame, receiving depth data associated with the 2D image data, the depth data corresponding to one or more distances from the video sensor to features represented by the 2D image data;
analyzing the 2D image data without analyzing the depth data to detect one or more objects in the video;
using the depth data to classify the one or more detected objects in the store depicted in the video, classification of the one or more detected objects comprising at least one of person classification and inventory classification and being based on a volume of the one or more detected objects; and
detecting an event of the classified one or more objects,
wherein the volume is determined by using the depth data along with the 2D image data to determine a plurality of convex hull slices on different Z-planes, and by summing areas of the plurality of convex hull slices.

2. The method of claim 1 ,
wherein the classification of the one or more objects comprises person classification, and
wherein the detecting an event of the classified one or more objects comprises using the depth data to detect a detected person is lying down in the store.

3. The method of claim 2 , wherein the detection of a person lying down comprises identifying one or more parts of the detected person and comparing height information of the one or more parts of the detected person with a height of the floor of the store.

4. The method of claim 1 , wherein the detecting an event of the classified one or more objects comprises using the depth data associated with the 2D image data to detect a spill of a substance on a floor of the store.

5. The method of claim 4 , wherein the detection of a spill comprises obtaining height information of a first object and determining that the height information indicates the first object is within a first distance of the height of the floor.

6. The method of claim 1 ,
wherein analyzing the 2D image data and using the depth data to classify the one or more objects detects and classifies plural objects, each as a respective person, and
wherein the detecting of the event comprises counting a number of people.

7. The method of claim 6 , further comprising monitoring at least one of a height and a distance from a depth sensor providing the depth data of an upper portion of each object classified person.

8. The method of claim 7 , further comprising:
for each of the objects classified as a person, associating a depth value with a portion of the person most distant from the floor of the store.

9. The method of claim 7 , further comprising determining a height value of each object classified as a person, and using the height value to track each object classified as a person.

10. The method of claim 9 , wherein the detecting of the event comprises counting a number of people entering the store.

11. The method of claim 9 , wherein the detecting of the event comprises counting a number of people in a line of the store.

12. The method of claim 1 , wherein using the depth data to classify the one or more objects comprises obtaining height data of at least some of the detected one or more objects from corresponding depth data of the at least some of the detected one or more objects.

13. The method of claim 12 , wherein the height data is used to classify a person as an adult.

14. The method of claim 13 , wherein the height data is used to classify a person as a child.

15. The method of claim 12 , wherein the height data is used to classify an object as an object located in a shopping cart.

16. The method of claim 15 , further comprising classifying a type of the object located in the shopping cart.

17. The method of claim 12 , wherein the height data is used to classify an object as a shopping cart.

18. The method of claim 17 , further comprising tracking shopping cart movement within the store.

19. The method of claim 1 , wherein a first object is classified as a person, and detecting an event of the person comprises detecting a reaching of the person.

20. The method of claim 19 , wherein detecting the reaching of the person comprises detecting an arm height of the person from the depth data.

21. The method of claim 20 , wherein detecting the reaching of the person comprises comparing the detected arm height of the person with a known height of a background feature represented by the image data.

22. The method of claim 1 , wherein a first object is classified as a person, and detecting an event of the person comprises detecting a time the person dwells within a certain location within the store.

23. The method of claim 22 , further comprising generating an alert in response to detecting that the person has dwelled within a certain location greater than a predetermined time.

24. The method of claim 22 , further comprising analyzing dwell times of multiple customers over plural weeks to determine a pattern of dwell times of customers of the store.

25. The method of claim 24 , further comprising using the pattern of dwell times to determine employee allocation within the store.

26. The method of claim 24 , further comprising using the pattern of dwell times to adjust merchandising within the store.

27. The method of claim 1 , wherein a first object is classified as inventory, and detecting an event of the inventory comprises detecting that the inventory is low or absent.

28. The method of claim 27 , wherein the depth data is used to determine a depth of the inventory on a shelf, within a refrigerator, or within a freezer in the store, and wherein the depth data is used to detect that the inventory is low or absent.

29. The method of claim 28 , further comprising generating an alert in response to determining that the inventory is low or absent to notify at least one of store management or inventory suppliers of the need to supplement the inventory.

30. The method of claim 1 , wherein a first object is classified as inventory, and detecting an event of the inventory comprises detecting that the inventory is delivered at a delivery location of the store.

31. The method of claim 30 , wherein the depth data is used to detect appropriate delivery of inventory to the store.

32. The method of claim 1 , wherein a first object is classified as inventory, and detecting an event of the inventory comprises tracking the inventory to detect if the inventory is removed without delivery.

Assignees

• Avigilon Fortress Corp

Inventors

• Zhang Zhong
• Myers Gary W
• Venetianer Peter L

Classifications

• G06T7/73

  using feature-based methods · CPC title

• G06K9/00087

  Physics · mapped topic

• G08B13/19615

  wherein said pattern is defined by the user · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• H04N7/18

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title