Method and apparatus for shelf feature and object placement detection from shelf images

Patent metadata

Field	Value
Publication number	US-10726273-B2
Application number	US-201715583740-A
Country	US
Kind code	B2
Filing date	May 1, 2017
Priority date	May 1, 2017
Publication date	Jul 28, 2020
Grant date	Jul 28, 2020

Abstract

A method of detecting a back of a shelf for supporting objects includes: obtaining an image depicting a shelf having a shelf edge and a support surface extending from the shelf edge to a shelf back; decomposing the image into a plurality of patches; for each patch: generating a feature descriptor; based on the feature descriptor, assigning one of a shelf back classification and a non-shelf back classification to the patch; generating a mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches; and presenting the mask.

First claim

The invention claimed is:

1. A method of detecting a back of a shelf for supporting objects, comprising:
obtaining, by an imaging controller, an image depicting a shelf having a shelf edge and a support surface extending from the shelf edge to a shelf back;
decomposing, by the imaging controller, the image into a plurality of patches;
for each patch:
generating a feature descriptor;
based on the feature descriptor, assigning one of a shelf back classification and a non-shelf back classification to the patch;

generating a mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches, wherein the indication of the classification assigned to each of the patches comprises one of a plurality of intensity values; and
presenting the mask to a gap detector for detection of gaps from the mask.

2. The method of claim 1 , wherein assigning the classification to each patch includes determining a confidence value corresponding to the assigned classification.

3. The method of claim 2 , wherein generating the mask comprises, for each patch:
assigning one of two intensity values to each patch based on the confidence value and a confidence threshold.

4. The method of claim 1 , further comprising:
prior to decomposing the image, converting the image to greyscale.

5. The method of claim 1 , further comprising:
prior to decomposing the image, performing a contrast normalization operation on the image.

6. The method of claim 1 , further comprising:
generating a plurality of scaled versions of the image;
repeating the decomposing, generating a feature descriptor, and assigning a classification for each scaled version; and
wherein generating the mask includes generating a sub-mask corresponding to each scaled version, and combining the sub-masks.

7. The method of claim 1 , wherein the feature descriptor is a histogram of oriented gradients (HOG) descriptor.

8. The method of claim 1 , further comprising:
obtaining, by the imaging controller, a location of the shelf edge within the mask;
generating, by the imaging controller, a gap mask identifying a subset of the areas having shelf back classifications as gaps on the shelf, by:
selecting an area of the mask within a predetermined distance of the shelf edge location and having the shelf back classification; and
generating a joining area between the selected area and the shelf edge location; and
assigning a gap classification to the selected area and the joining area; and

presenting the gap mask.

9. A method of detecting, by an imaging controller, gaps in an image of a shelf having a shelf edge and a support surface for supporting objects extending from the shelf edge to a shelf back, the method comprising:
obtaining, by the imaging controller, a mask generated from the image and containing indications, for each of a plurality of areas of the mask, of a classification assigned to the area and selected from a shelf back classification and a non-shelf back classification;
obtaining, by the imaging controller, a location of the shelf edge within the mask;
generating, by the imaging controller, a gap mask identifying a subset of the areas having shelf back classifications as gaps on the shelf, by:
selecting an area of the mask within a predetermined distance of the shelf edge location and having the shelf back classification; and
generating a joining area between the selected area and the shelf edge location; and
assigning a gap classification to the selected area and the joining area; and

presenting the gap mask.

10. The method of claim 9 , wherein generating the gap mask further comprises:
discarding a portion of the mask having a preconfigured position relative to the shelf edge location.

11. The method of claim 10 , wherein the preconfigured position includes below the shelf edge location.

12. The method of claim 10 , wherein the preconfigured position includes horizontally spaced beyond an end of the shelf edge location.

13. The method of claim 9 , wherein generating the gap mask further comprises:
comparing each area having a shelf back classification to a preconfigured area threshold, and discarding any areas that do not meet the threshold.

14. The method of claim 9 , wherein generating the gap mask further comprises:
performing a region growth operation on the selected area with the joining area; and
assigning the gap classification to a grown region resulting from the region growth operation.

15. The method of claim 9 , further comprising:
obtaining depth data captured by a depth sensor and indicating a point depth for each of a plurality of points in the gap mask;
dividing the gap mask into a plurality of vertical slices;
for each vertical slice of the gap mask:
determining a proportion of the point depths that exceed a median point depth for the vertical slice; and
assigning a non-gap classification to the vertical slice when the proportion does not meet a preconfigured depth distribution threshold.

16. The method of claim 9 , wherein obtaining the mask comprises:
obtaining, by the imaging controller, the image;
decomposing, by the imaging controller, the image into a plurality of patches;
for each patch:
generating a feature descriptor;
based on the feature descriptor, assigning one of the shelf back classification and the non-shelf back classification to the patch; and
generating the mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches.

17. A server for detecting a back of a shelf for supporting objects, comprising:
a memory storing an image depicting a shelf having a shelf edge and a support surface extending from the shelf edge to a shelf back; and
an imaging controller comprising:
a pre-processor configured to decompose the image into a plurality of patches;
a back of shelf detector configured to:
(a) for each patch:
generate a feature descriptor;
based on the feature descriptor, assign one of a shelf back classification and a non-shelf back classification to the patch;

(b) generate a mask corresponding to the image, the mask containing an indication of the classification assigned to each of the patches, wherein the indication of the classification assigned to each of the patches comprises one of a plurality of intensity values;

the back of shelf detector further configured to present the mask.

18. A server for detecting gaps in an image of a shelf having a shelf edge and a support surface for supporting objects extending from the shelf edge to a shelf back, the system comprising:
a memory storing a mask generated from the image, the mask containing indications, for each of a plurality of areas of the mask, of a classification assigned to the area and selected from a shelf back classification and a non-shelf back classification; and
an imaging controller comprising:
a shelf edge detector configured to obtain a location of the shelf edge within the mask;
a gap generator configured to generate a gap mask identifying a subset of the areas having shelf back classifications as gaps on the shelf, by:
selecting an area of the mask within a predetermined distance of the shelf edge location and having the shelf back classification; and
generating a joining area between the selected area and the shelf edge location; and
assigning a gap classification to the selected area and the joining area; and

the gap generator further configured to present the gap mask.

19. The method of cla

Assignees

• Symbol Technologies Llc

Inventors

• Phan Raymond
• Zhang Yan
• Rzeszutek Richard Jeffrey
• Fu Bo

Classifications

• G06V10/50

  by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis · CPC title

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06K9/4642

  Physics · mapped topic

• G06K9/00771 Primary

  Physics · mapped topic