System and method for live product report generation without annotated training data

Patent metadata

Field	Value
Publication number	US-11037099-B2
Application number	US-201916436518-A
Country	US
Kind code	B2
Filing date	Jun 10, 2019
Priority date	Jun 10, 2019
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

Embodiments described herein provide a method for obtaining information on product inventory and placement in a retail setting. An image including unannotated image data indicative of the retail setting is received. One or more shelves in the retail setting are determined from the unannotated image data, and the image is segmented into one or more sub-images corresponding to the one or more detected shelves. For each sub-image corresponding to a respective detected shelf, a product name is then and a number of appearances of the product name are detected using text recognition on the respective sub-image. Product inventory information and first placement information are derived based at least in part on the detected number of appearances and a shelf level corresponding to the sub-image.

First claim

What is claimed is:

1. A system for obtaining information on product inventory and placement in a retail setting, the system comprising:
a communication interface that receives a first image including first unannotated image data indicative of at least one shelf and one object in the retail setting;
a memory containing machine readable medium storing machine executable code; and
one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
detect one or more shelves in the retail setting from the first unannotated image data;
segment the image into one or more sub-images corresponding to the one or more detected shelves;
for each sub-image corresponding to a respective detected shelf, detect a product name and a number of appearances of the product name using text recognition on the respective sub-image;
derive first product inventory information and first placement information relating to a product having the detected product name based at least in part on the detected number of appearances and a shelf level corresponding to the sub-image from which the product name is detected.

2. The system of claim 1 , wherein the machine executable code causes the one or more processors further to generate a report based on the derived product inventory and placement information in response to the receipt of the first image.

3. The system of claim 1 , wherein the communication interface receives metadata associated with the first image captured by an image capture device, and wherein the metadata includes identifying information of a rack that the first image is taken at.

4. The system of claim 1 , wherein the machine executable code causes the one or more processors further to allocate a shelf identifier to each sub-image while segmenting the first image into one or more sub-images corresponding to the one or more detected shelves.

5. The system of claim 1 , wherein the machine executable code causes the one or more processors further to:
for each sub-image:
detect a first textual area on the respective sub-image;
identify a first text within the detected textual area via text recognition;
query a product database based on the identified first text; and
identify the product name based on a first query result.

6. The system of claim 5 , wherein the machine executable code causes the one or more processors further to:
for the respective sub-image:
detect a second textual area within a pre-defined distance to the first textual area;
identify a second text within the detected textual area via text recognition;
combine the first text and the second text to form a third text;
query the product database based on the third text;
identify the product name based on the third text when the third text at least partially matches a second query result.

7. The system of claim 1 , wherein the machine executable code causes the one or more processors further to:
derive the first product inventory information for the detected product name by:
determining a number of appearances of the detected product name within each sub-image; and
aggregating the number of appearances of the detected product name across the one or more sub-images.

8. The system of claim 1 , wherein the machine executable code causes the one or more processors further to:
derive the first product placement information for the detected product name by:
for each sub-image corresponding to a respective shelf:
determining a start coordinate and an end coordinate on the respective shelf corresponding to the product having the detected product name; and
associating a rack number, a shelf number corresponding to the respective shelf, the start and the end coordinates with the product name.

9. The system of claim 1 , wherein the communication interface receives a second image including second unannotated image data indicative of the at least one shelf and the one object and the machine executable code causes the one or more processors further to:
derive second product inventory information and second placement information relating to the product having the detected product name.

10. The system of claim 9 , wherein the machine executable code causes the one or more processors further to:
determine a first confidence indicator associated with the first product inventory information and the first placement information based on a first quality of the first image;
determine a second confidence indicator associated with the second product inventory information and the second placement information based on a second quality of the second image; and
adopt one of the first product inventory information and the first placement information, and the second product inventory information and the second placement information based on a comparison of the first confidence indicator and the second confidence indicator.

11. A method for obtaining information on product inventory and placement in a retail setting, the method comprising:
receiving, via a communication interface, a first image including first unannotated image data indicative of at least one shelf and one object in the retail setting;
detecting, via a processor, one or more shelves in the retail setting from the first unannotated image data;
segment, via the processor, the image into one or more sub-images corresponding to the one or more detected shelves;
for each sub-image corresponding to a respective detected shelf, detecting a product name and a number of appearances of the product name using text recognition on the respective sub-image;
deriving first product inventory information and first placement information relating to a product having the detected product name based at least in part on the detected number of appearances and a shelf level corresponding to the sub-image from which the product name is detected.

12. The method of claim 11 , further comprising generating a report based on the derived product inventory and placement information in response to the receipt of the first image.

13. The method of claim 11 , further comprising receiving metadata associated with the first image captured by an image capture device, and wherein the metadata includes identifying information of a rack that the first image is taken at.

14. The method of claim 11 , further comprising allocating a shelf identifier to each sub-image while segmenting the first image into one or more sub-images corresponding to the one or more detected shelves.

15. The method of claim 11 , further comprising:
for each sub-image:
detecting a first textual area on the respective sub-image;
identifying a first text within the detected textual area via text recognition;
querying a product database based on the identified first text; and
identifying the product name based on a first query result.

16. The method of claim 15 , further comprising:
for the respective sub-image:
detecting a second textual area within a pre-defined distance to the first textual area;
identifying a second text within the detected textual area via text recognition;
combining the first text and the second text to form a third text;
querying the product database based on the third text;
identifying the product name based on the third text when the third text at least partially matches a second query result.

17. The method of claim 11 , further comprising:
deriving the first product inventory information for the detected product name by:
determining a number of appearances of the detected product name wi

Assignees

• Salesforce Com Inc

Inventors

• Mustafi Joy
• Kumar Lakshya
• Dua Rajdeep Singh

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title