Action detection during image tracking

Patent metadata

Field	Value
Publication number	US-10621444-B1
Application number	US-201916663500-A
Country	US
Kind code	B1
Filing date	Oct 25, 2019
Priority date	Oct 25, 2019
Publication date	Apr 14, 2020
Grant date	Apr 14, 2020

Abstract

A system includes a sensor, a weight sensor, and a tracking subsystem. The tracking subsystem receives an image feed of top-view images generated by the sensor and weight measurements from the weight sensor. The tracking subsystem detects an event associated with an item being removed from a rack in which the weight sensor is installed. The tracking subsystem determines that a first person and a second person may be associated with the event. The tracking subsystem then determines, using a first approach, whether an action associated with the event was performed by the first person or the second person. If results of the first approach do not satisfy criteria, a second approach is used to assign the action to the first or second person.

First claim

The invention claimed is:

1. A system, comprising:
a sensor positioned above a rack in a space, the sensor configured to generate top-view images of at least a portion of a space comprising the rack;
a plurality of weight sensors, each weight sensor associated with a corresponding item stored on a shelf of the rack; and
a tracking subsystem coupled to the image sensor and the weight sensors, the tracking subsystem configured to:
receive an image feed comprising frames of the top-view images generated by the sensor;
receive weight measurements from the weight sensors;
detect an event associated with one or both of a portion of a person entering a zone adjacent to the rack and a change of weight associated with a first item being removed from a first shelf associated with a first weight sensor;
in response to detecting the event, determine that a first person and a second person may be associated with the detected event, based on one or more of a first distance between the first person and the rack, a second distance between the second person and the rack, and an inter-person distance between the first person and the second person;
in response to determining that the first and second person may be associated with the detected event, store buffer frames of top-view images generated by the sensor following the detected event;
determine, using at least one of the stored buffer frames and a first action-detection algorithm, whether an action associated with the detected event was performed by the first person or the second person, wherein the first action-detection algorithm is configured to detect the action based on characteristics of one or more contours in the at least one stored buffer frame(s);
determine whether results of the first action-detection algorithm satisfy criteria based at least in part on a number of iterations required to implement the first action-detection algorithm;
in response to determining the results of the first action-detection algorithm do not satisfy the criteria, determine, by applying a second action-detection algorithm to at least one of the buffer frames, whether the action associated with the detected event was performed by the first person or the second person, wherein the second action-detection algorithm is configured to detect the action using an artificial neural network;
in response to determining the action was performed by the first person, assign the action to the first person; and
in response to determining the action was performed by the second person, assign the action to the second person.

2. The system of claim 1 , wherein the tracking subsystem is further configured to:
following storing the buffer frames, determine a region-of-interest of the top-view images of the stored frames; and
determine, using the region-of-interest of at least one of the stored buffer frames and the first action-detection algorithm, whether the action associated with the detected event was performed by the first person or the second person.

3. The system of claim 1 , wherein the stored buffer frames comprise three or fewer frames of top-view images following one or both of: the portion of the person entering the zone adjacent to the rack and the portion of the person exiting the zone adjacent to the rack.

4. The system of claim 3 , wherein the tracking subsystem is further configured to determine a subset of the buffer frames to use with the first action-detection algorithm and a second subset of the buffer frames to use with the second action detection algorithm.

5. The system of claim 1 , wherein the tracking subsystem is further configured to determine that the first person and the second person may be associated with the detected event based on a first relative orientation between the first person and the rack and a second relative orientation between the second person and the rack.

6. The system of claim 1 , wherein:
the detected action is associated with a person picking up the first item stored on the first shelf of the rack; and
the tracking subsystem is further configured to:
in response to determining the action was performed by the first person, assign the first item to the first person; and
in response to determining the action was performed by the second person, assign the first item to the second person.

7. The system of claim 1 , wherein:
the first action-detection algorithm involves iterative dilation of a first contour associated with the first person and a second contour associated with the second contour; and
the criteria comprise a requirement that the portion of the person entering the zone adjacent to the rack is associated with either the first person or the second person within a maximum number of iterative dilations of the first and second contours.

8. The system of claim 7 , wherein the tracking subsystem is further configured to:
in response to determining the first person is associated with the portion of the person entering the zone adjacent to the rack within the maximum number of dilations, assign the action to the first person.

9. A method, comprising:
receiving an image feed comprising frames of top-view images generated by a sensor, the sensor positioned above a rack in a space and configured to generate top-view images of at least a portion of a space comprising the rack;
receiving weight measurements from a weight sensor associated with a corresponding item stored on a shelf of the rack;
detecting an event associated with one or both of a portion of a person entering a zone adjacent to the rack and a change of weight associated with a first item being removed from a first shelf associated with the weight sensor;
in response to detecting the event, determining that a first person and a second person may be associated with the detected event, based on one or more of a first distance between the first person and the rack, a second distance between the second person and the rack, and an inter-person distance between the first person and the second person;
in response to determining that the first and second person may be associated with the detected event, storing buffer frames of top-view images generated by the sensor following the detected event;
determining, using at least one of the stored buffer frames and a first action-detection algorithm, whether an action associated with the detected event was performed by the first person or the second person, wherein the first action-detection algorithm is configured to detect the action based on characteristics of one or more contours in the at least one stored buffer frame(s);
determining whether results of the first action-detection algorithm satisfy criteria based at least in part on a number of iterations required to implement the first action-detection algorithm;
in response to determining the results of the first action-detection algorithm do not satisfy the criteria, determining, by applying a second action-detection algorithm to at least one of the buffer frames, whether the action associated with the detected event was performed by the first person or the second person, wherein the second action-detection algorithm is configured to detect the action using an artificial neural network;
in response to determining the action was performed by the first person, assigning the action to the first person; and
in response to determining the action was performed by the second person, assigning the action to the second person.

10. The method of claim 9 , further comprising:
following storing the buffer frames, determining a region-of-interest of the top-view images of the stored frames; and
determining, using the region-of-interest of at least one of the stored buffer frames and th

Assignees

• 7 Eleven Inc

Inventors

• Mirza Shahmeer Ali
• Krishnamurthy Sailesh Bharathwaaj
• Vakacharla Sarath
• Paul Deepanjan

Classifications

• H04N7/188 Primary

  Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253) · CPC title

• H04N7/183

  for receiving images from a single remote source · CPC title

• G08B13/19669

  Event triggers storage or change of storage policy · CPC title

• G08B13/19613

  Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion · CPC title

• G08B13/19645

  Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over · CPC title