Method, system and apparatus for detecting product facings

Patent metadata

Field	Value
Publication number	US-12443652-B2
Application number	US-201916429820-A
Country	US
Kind code	B2
Filing date	Jun 3, 2019
Priority date	Jun 3, 2019
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

A method of detecting product facings from captured depth and image data includes: obtaining, at an imaging controller, (i) depth measurements representing a support structure supporting a plurality of product facings, (ii) image data representing the support structure, and (iii) a set of region of interest (ROI) indicators each indicating a position of a plurality of the product facings; generating a first set of candidate facing edges from the depth measurements; generating a second set of candidate facing edges from the image data; generating a third set of candidate facing edges by combining the first and second sets; generating, for each adjacent pair of the third set of candidate facing edges, a candidate facing boundary; selecting a subset of output facing boundaries from the candidate facing boundaries, based on the ROI indicators; and storing the output facing boundaries in a memory coupled to the imaging controller.

First claim

The invention claimed is:

1. A method by an imaging controller of detecting product facings from captured depth and image data, the method comprising:
obtaining at the imaging controller: (i) depth measurements from at least one depth sensor, the depth measurements representing a support structure supporting a plurality of product facings, (ii) image data from at least one image sensor, the image data representing the support structure, and (iii) a set of region of interest (ROI) indicators, each ROI indicator indicating a position of a respective subset of the plurality of product facings on the support structure;
generating, by a depth detector of the imaging controller, from the depth measurements of the at least one depth sensor, a first set of candidate facing edges;
generating, by an image detector of the imaging controller, from the image data of the at least one image sensor, a second set of candidate facing edges from the image data;
generating by a boundary generator of the imaging controller:
(i) a third set of candidate facing edges by combining the first and second sets, and
(ii) a candidate facing boundary for each adjacent pair of candidate facing edges in the third set of candidate facing edges;

verifying, by the imaging controller, that the candidate facing boundaries correspond to positions of products on the support structure by selecting, by the boundary generator of the imaging controller, a subset of output facing boundaries from the candidate facing boundaries that are disposed within the ROI indicators;
detecting, by the boundary generator of the imaging controller, product facings including the selected subset of output facing boundaries.

2. The method of claim 1 , wherein generating the first set of candidate facing edges includes:
generating a two-dimensional depth map from the depth measurements; and
detecting edges in the depth map.

3. The method of claim 2 , wherein detecting edges in the depth map includes:
applying an edge detection operation to the depth map to generate an edge-weighted depth map; and
applying a line detection operation to the edge-weighted depth map.

4. The method of claim 3 , further comprising discarding candidate edges of the first set that are not within an ROI indicator.

5. The method of claim 1 , wherein generating the second set of candidate facing edges includes:
selecting a plurality of windows from the image data; and
classifying each window as one of containing an edge and not containing an edge.

6. The method of claim 5 , further comprising:
determining a position of each candidate facing edge of the second set based on an intensity profile of a corresponding one of the windows.

7. The method of claim 1 , wherein combining the first and second sets of candidate facing edges includes:
for each pair of adjacent candidate facing edges, determining whether a distance separating the pair is below a threshold;
when the distance separating the pair is below the threshold, replacing the pair with a single candidate facing edge.

8. The method of claim 1 , wherein generating the candidate facing boundaries includes:
joining upper ends of the adjacent pair of the third set of candidate facing edges with an upper boundary segment; and
joining lower ends of the adjacent pair of the third set of candidate facing edges with a lower boundary segment.

9. The method of claim 1 , wherein selecting the subset of output facing boundaries includes discarding candidate facing boundaries outside the ROI indicators.

10. A computing device, comprising:
a memory storing: (i) depth measurements from at least one depth sensor, the depth measurements representing a support structure supporting a plurality of product facings, (ii) image data from at least one image sensor, the image data representing the support structure, and (iii) a set of region of interest (ROI) indicators, each ROI indicator indicating a position of a respective subset of the plurality of product facings on the support structure;
an imaging controller having:
a depth detector configured to:
obtain the depth measurements and the ROI indicators; and
generate, from the depth measurements of the at least one depth sensor, a first set of candidate facing edges;

an image detector configured to:
obtain the image data and the ROI indicators; and
generate, from the image data of the at least one image sensor, a second set of candidate facing edges; and

a boundary generator configured to generate:
a third set of candidate facing edges by combining the first and second sets;
a candidate facing boundary for each adjacent pair of candidate facing edges in the third set of candidate facing edges;
verify that the candidate facing boundaries correspond to positions of products on the support structure by selecting a subset of output facing boundaries from the candidate facing boundaries that are disposed within the ROI indicators; and
detect product facings including the selected subset of output facing boundaries.

11. The computing device of claim 10 , wherein the depth detector is configured, in order to generate the first set of candidate facing edges, to:
generate a two-dimensional depth map from the depth measurements; and
detect edges in the depth map.

12. The computing device of claim 11 , wherein the depth detector is configured, in order to detect edges in the depth map, to:
apply an edge detection operation to the depth map to generate an edge-weighted depth map; and
apply a line detection operation to the edge-weighted depth map.

13. The computing device of claim 12 , wherein the depth detector is further configured to discard candidate edges of the first set that are not within an ROI indicator.

14. The computing device of claim 10 , wherein the image detector is configured, in order to generate the second set of candidate facing edges, to:
select a plurality of windows from the image data; and
classify each window as one of containing an edge and not containing an edge.

15. The computing device of claim 14 , wherein the image detector is further configured to determine a position of each candidate facing edge of the second set based on an intensity profile of a corresponding one of the windows.

16. The computing device of claim 10 , wherein the boundary generator is configured, in order to combine the first and second sets of candidate facing edges, to:
for each pair of adjacent candidate facing edges, determine whether a distance separating the pair is below a threshold;
when the distance separating the pair is below the threshold, replace the pair with a single candidate facing edge.

17. The computing device of claim 10 , wherein the boundary generator is configured, in order to generate the candidate facing boundaries, to:
join upper ends of the adjacent pair of the third set of candidate facing edges with an upper boundary segment; and
join lower ends of the adjacent pair of the third set of candidate facing edges with a lower boundary segment.

18. The computing device of claim 10 , wherein the boundary generator is configured, in order to select the subset of output facing boundaries, to discard candidate facing boundaries outside the ROI indicators.

19. A non-transitory computer-readable medium storing instructions executable by an imaging controller to configure the imaging controller to:
obtain (i) depth measurements from at least one depth sensor, the depth measurements representing a support structure supporting a plurality of product facings, (ii) image

Assignees

• Zebra Tech Corp

Inventors

• Lam Joseph
• Yu Yuanhao
• Rawat Abhishek

Classifications

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/24

  Aligning, centring, orientation detection or correction of the image · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title