Object detection based on wrist-area region-of-interest

Patent metadata

Field	Value
Publication number	US-11403852-B2
Application number	US-202017104364-A
Country	US
Kind code	B2
Filing date	Nov 25, 2020
Priority date	Oct 25, 2019
Publication date	Aug 2, 2022
Grant date	Aug 2, 2022

Abstract

An image sensor is positioned such that a field-of-view of the image sensor encompasses at least a portion of a rack storing items. The image sensor generates angled-view images of the items stored on the rack. A tracking subsystem receives image frames of the angled-view images. The tracking subsystem detects that a trigger event has occurred. A set of one or more image frames from the image feed are determined that are associated with the detected trigger event. A region-of-interest of the image frame is determined based on the pixel position of the wrist of the person. The region-of-interest includes a subset of the pixels of the image frame. A first item in the determined region-of-interest using an object detection algorithm. The identified first item is assigned to the person.

First claim

The invention claimed is:

1. A system, comprising:
a rack comprising shelves configured to store items;
an image sensor positioned such that a field-of-view of the image sensor encompasses at least a portion of the rack, wherein the image sensor is configured to generate angled-view images of the items stored on the shelves of the rack; and
a tracking subsystem coupled to the image sensor, the tracking subsystem comprising at least one processor configured to:
receive an image feed comprising image frames of the angled-view images generated by the image sensor;
detect that a trigger event has occurred associated with a person interacting with the items stored on the shelves of the rack;
following detecting the trigger event:
determine a set of one or more image frames from the image feed associated with the detected trigger event;
for each image frame of the determined set of one or more image frames:
determine a pixel position of a wrist of the person in the image frame; and
determine, in the image frame, a region-of-interest based on the pixel position of the wrist of the person, wherein the region-of-interest includes a subset of the pixels of the image frame comprising at least a first item, wherein a size of the region-of-interest is determined based at least in part on features of the person;

over a period of time, track the pixel position of the wrist of the person; and
determine, based on the pixel positions tracked over the period of time, a maximum depth into the rack to which the person reached over the period of time;
determine candidate items based on the maximum depth into the rack to which the person reached over the period of time, wherein the candidate items include the first item and one or more items at adjacent positions in the rack to the first item;
for each candidate item, determine, based on a comparison of a predefined position associated with the candidate items to the maximum depth into the rack to which the person reached over the period of time, a probability value that the candidate item was interacted with by the person;
if a highest probability value determined for the candidate items is greater than a threshold value, determine that reliability criteria are satisfied and identify the first item based at least in part on the determined probability values;
if the highest probability value determined for the candidate items is less than the threshold value, determine that the reliability criteria are not satisfied and identify the first item in the determined region-of-interest using an object detection algorithm; and
assign the identified first item to the person.

2. The system of claim 1 , wherein the processor is further configured to:
determine, based on properties of the person in at least one of the determined image frames, a first identifier of the person; and
assign the identified first item to the person by adding a description of the first item to a digital shopping cart associated with the first identifier of the person.

3. The system of claim 1 , wherein the processor is further configured to detect that the trigger event has occurred by determining that the person is within a threshold distance of the rack.

4. The system of claim 1 , wherein the processor is further configured to detect that the trigger event has occurred by determining that a portion of the person has entered a predefined zone adjacent to a front of the rack.

5. The system of claim 1 , wherein:
the system further comprises a weight sensor disposed on a shelf of the rack, the weight sensor configured to measure a weight of items on the weight sensor; and
the processor is communicatively coupled to the weight sensor and is further configured to detect that the trigger event has occurred by detecting a decrease in weight on the weight sensor.

6. The system of claim 1 , wherein the processor is further configured to detect the trigger event based at least in part on the maximum depth into the rack to which the person reached over the period of time.

7. The system of claim 1 , wherein the processor is further configured to, prior to identifying the first item, if probabilities determined for two or more of the candidate items are greater than the threshold value, determine that the reliability criteria are not satisfied and identify the first item in the determined region-of-interest using the object detection algorithm.

8. The system of claim 1 , wherein the features of the person on which the size of the region-of-interest is based at least in part comprise one or more of an arm length of the person, a height of the person, and a ratio of a shoulder width of the person to the arm length of the person.

9. A method, comprising:
receiving an image feed comprising angled-view images generated by an image sensor, the image sensor positioned such that a field-of-view of the image sensor encompasses at least a portion of a rack comprising shelves configured to store items, wherein the image sensor is configured to generate the angled-view images of the items stored on the shelves of the rack;
detecting that a trigger event has occurred associated with a person interacting with the items stored on the shelves of the rack;
following detecting the trigger event:
determining a set of one or more image frames from the image feed associated with the detected trigger event;
for each image frame of the determined set of one or more image frames:
determining a pixel position of a wrist of the person in the image frame; and
determining, in the image frame, a region-of-interest based on the pixel position of the wrist of the person, wherein the region-of-interest includes a subset of the pixels of the image frame comprising at least a first item, wherein a size of the region-of-interest is determined based at least in part on features of the person;

over a period of time, tracking the pixel position of the wrist of the person; and
determining, based on the pixel positions tracked over the period of time, a maximum depth into the rack to which the person reached over the period of time;
determining candidate items based on the maximum depth into the rack to which the person reached over the period of time, wherein the candidate items include the first item and one or more items at adjacent positions in the rack to the first item;
for each candidate item, determining, based on a comparison of a predefined position associated with the candidate items to the maximum depth into the rack to which the person reached over the period of time, a probability value that the candidate item was interacted with by the person;
if a highest probability value determined for the candidate items is greater than a threshold value, determining that reliability criteria are satisfied and identifying the first item based at least in part on the determined probability values;
if the highest probability value determined for the candidate items is less than the threshold value, determining that the reliability criteria are not satisfied and identifying the first item in the determined region-of-interest using an object detection algorithm; and
assigning the identified first item to the person.

10. The method of claim 9 , further comprising:
determining, based on properties of the person in at least one of the determined image frames, a first identifier of the person; and
assigning the identified first item to the person by adding a description of the first item to a digital shopping cart associated with the first identifier of the person.

11. The method of claim 9 , further comprising detecting that the trigger event has occurred by determining that the person is within a threshold d

Assignees

• 7 Eleven Inc

Inventors

• Datar Sumedh Vilas
• Krishnamurthy Sailesh Bharathwaaj
• Mirza Shahmeer Ali

Classifications

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title