Image processing of a retail shelf area

Patent metadata

Field	Value
Publication number	US-9710926-B2
Application number	US-201514609741-A
Country	US
Kind code	B2
Filing date	Jan 30, 2015
Priority date	Jan 30, 2015
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

A system and method for processing an image of a retail shelf area. According to one aspect, a portion of an image that depicts the retail shelf area at an angled and depthwise perspective is determined to depict shelves. The image can be modified so that the portion of the image determined to depict shelves is substantially parallel to a predefined horizontal reference frame. Products can then be detected in the remainder portion of the image, separate from the portion of the image determined to depict shelves. An applicable planogram can be used to generate compliance information based on the detected products and the determined portion of the image depicting shelves.

First claim

What is claimed is:

1. A method for processing an image of a retail shelf area, the method being implemented by one or more processors and comprising:
receiving an image of the retail shelf area, the image depicting the retail shelf area at an angled and depthwise perspective in relation to a point of image capture;
determining a portion of the image that depicts one or more shelves in the retail shelf area in an angled alignment with respect to a predefined horizontal reference frame, to reflect the angled and depthwise perspective;
based on the determination of the portion of the image depicting the one or more shelves, modifying the image to depict the one or more shelves as being parallel to the predefined horizontal reference frame;
based on the modified image, detecting one or more products in a remainder portion of the image, separate from the portion of the image determined to depict the one or more shelves;
determining a discrepancy of at least one product depicted in the remainder portion of the image with respect to a planogram of the retail shelf area; and
generating an output that identifies the determined discrepancy.

2. The method of claim 1 , wherein detecting the one or more products in the remainder portion of the modified image includes:
identifying, in the remainder portion, a plurality of candidate image portions for depicting the one or more products; and
analyzing each of the plurality of candidate image portions to determine the one or more products based on a plurality of stored product images.

3. The method of claim 2 , wherein the analyzing each of the plurality of candidate image portions is based on a dominant color in each of the plurality of candidate image portions.

4. The method of claim 2 , wherein the analyzing each of the plurality of candidate image portions is based on a number of key points identifiable in each of the plurality of candidate image portions.

5. The method of claim 2 , wherein the analyzing each of the plurality of candidate image portions includes:
identifying a dominant color in each of the plurality of candidate image portions; and
determining a shelved product in each of the plurality of candidate image portions from the plurality of the stored product images after excluding the stored product images that do not contain the dominant color.

6. The method of claim 5 , wherein the determining a shelved product in each of the plurality of candidate image portions from the plurality of the stored product images after excluding the stored product images that do not contain the dominant color includes:
identifying a number of key points in each of the candidate image portions; and
based on the identified key points, determining the shelved product in each of the plurality of candidate image portions from the plurality of stored product images after excluding the stored product images that do not contain the dominant color.

7. The method of claim 2 , wherein analyzing each of the plurality of candidate image portions includes:
identifying a number of key points in each of the candidate image portions; and
determining a shelved product in each of the plurality of candidate image portions from the plurality of stored product images based on the identified key points.

8. The method of claim 7 , wherein the number of key points is a predetermined threshold number.

9. The method of claim 1 , further comprising:
identifying a planogram applicable for the retail shelf area in the image.

10. The method of claim 1 , wherein the detecting one or more products in a remainder portion of the modified image includes:
identifying a planogram applicable for the retail shelf area in the image; and
determining one or more products in the remainder portion of the modified image based on the applicable planogram.

11. The method of claim 10 , wherein the determining one or more products in the remainder portion of the modified image based on the planogram includes:
identifying one or more target products in the planogram;
identifying, in the remainder portion, a plurality of candidate image portions for depicting the one or more products; and
analyzing each of the plurality of candidate image portions to determine the one or more products, from a stored product images in a plurality of stored product images that correspond to the one or more target products.

12. The method of claim 11 , wherein the analyzing each of the plurality of candidate image portions includes:
identifying a dominant color in each of the plurality of candidate image portions;
identifying a number of key points in each of the candidate image portions; and
based on the identified key points, determining a shelved product in each of the plurality of candidate image portions, from the stored product images in the plurality of stored product images that correspond to the one or more target products, after excluding the stored product images that do not contain the dominant color.

13. A system for processing an image of a retail shelf area, comprising:
a memory resource to store instructions; and
one or more processors using the instructions stored in the memory resource to:
receive an image of the retail shelf area, the image depicting the retail shelf area at an angled and depthwise perspective in relation to a point of image capture;
determine a portion of the image that depicts one or more shelves in the retail shelf area in an angled alignment with respect to a predefined horizontal reference frame, to reflect the angled and depthwise perspective;
based on the determination of the portion of the image depicting the one or more shelves, modify the image to depict the one or more shelves as being parallel to the predefined horizontal reference frame;
detect one or more products in a remainder portion of the modified image, separate from the portion of the image determined to depict the one or more shelves;
determine a discrepancy of at least one product depicted in the remainder portion of the image with respect to a planogram of the retail shelf area; and
generate an output based on the determination of the discrepancy.

14. A non-transitory computer-readable medium for processing an image of a retail shelf area, the computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include:
capturing an image of the retail shelf area, the image depicting the retail shelf area at an angled and depthwise perspective in relation to a point of image capture;
determining a portion of the image that depicts one or more shelves in the retail shelf area in an angled alignment with respect to a predefined horizontal reference frame, to reflect the angled and depthwise perspective;
based on the determination of the portion of the image depicting the one or more shelves, modifying the image to depict the one or more shelves as being parallel to the predefined horizontal reference frame;
detecting one or more products in a remainder portion of the modified image, separate from the portion of the image determined to depict the one or more shelves;
determine a discrepancy of at least one product depicted in the remainder portion of the image with respect to a planogram of the retail shelf area; and
generating an output based on the determination of the discrepancy.

Assignees

• Hewlett Packard Development Co Lp

Inventors

• Zhang Tong
• Fan Jian

Classifications

• G06V10/806

  of extracted features · CPC title

• G06F18/253

  of extracted features · CPC title

• G06V10/56

  relating to colour · CPC title

• G06V10/467

  Encoded features or binary features, e.g. local binary patterns [LBP] · CPC title

• G06V10/457

  by analysing connectivity, e.g. edge linking, connected component analysis or slices · CPC title