Methods, systems, articles of manufacture, and apparatus for decoding purchase data using an image

Patent metadata

Field	Value
Publication number	US-12327425-B2
Application number	US-202217710649-A
Country	US
Kind code	B2
Filing date	Mar 31, 2022
Priority date	Jun 24, 2021
Publication date	Jun 10, 2025
Grant date	Jun 10, 2025

Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed that decode purchase data using an image. An example apparatus includes processor circuitry to execute machine readable instructions to at least crop an image of a receipt based on detected regions of interest, apply a first mask to a first cropped image to generate first bounding boxes corresponding to rows of the receipt, apply a second mask to a second cropped image to generate second bounding boxes corresponding to columns of the receipt, generate a structure of the receipt by mapping words detected by an optical character recognition engine to corresponding first bounding boxes and second bounding boxes based on a mapping criterion, classify the second bounding boxes by identifying an expression of interest in ones of the second bounding boxes, and generate purchase information by extracting text of interest from the structured receipt based on the classifications.

First claim

What is claimed is:

1. An apparatus comprising:
interface circuitry to obtain an image of a receipt;
machine-readable instructions; and
at least one processor circuit to be programmed by the machine-readable instructions to at least:
cause generation of a first cropped image and a second cropped image based on the image of the receipt, the first and second cropped images based on regions of the receipt detected in the image;
apply a first mask to the first cropped image to generate first bounding boxes, the first bounding boxes corresponding to rows of the receipt;
apply a second mask to the second cropped image to generate second bounding boxes, the second bounding boxes corresponding to columns of the receipt;
generate a structure of the receipt by mapping (a) words detected by an optical character recognition (OCR) engine to corresponding first bounding boxes based on a mapping criterion, and (b) ones of the words to respective ones of the second bounding boxes based on the mapping criterion;
cause classification of the columns based on the ones of the words mapped to the respective ones of the second bounding boxes;
extract purchase information from the structured receipt based on the classifications, the purchase information corresponding to products listed in the receipt, the products including a first product; and
generate promotion information based on the first product being associated with a promotion by:
identifying a first price associated with the first product, the first price corresponding to a first column, the first column classified as a price column;
identifying a second price associated with the first product, the second price corresponding to the promotion; and
determining a third price associated with the first product based on the first price and the second price.

2. The apparatus of claim 1 , wherein the regions include (1) a receipt region, and (2) a purchase region, the purchase region corresponding to an area of the receipt that includes information about the products listed in the receipt.

3. The apparatus of claim 2 , wherein the first cropped image is based on the receipt region and the second cropped image is based on the purchase region.

4. The apparatus of claim 1 , wherein the first mask causes identification of first pixels of the first cropped image classified as belonging to a first class and second pixels of the first cropped image classified as belonging to a second class.

5. The apparatus of claim 4 , wherein the first class is corresponds to a text line and the second class is corresponds to a background.

6. The apparatus of claim 1 , wherein the second mask identifies of first pixels of the second cropped image classified as belonging to a first class and second pixels of the second cropped image classified as belonging to a second class.

7. The apparatus of claim 6 , wherein the first class is corresponds to a column and the second class corresponds to another class.

8. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to merge a first one of the first bounding boxes and a second one of the first bounding boxes based on a row connection criterion, the first one and the second one of the first bounding boxes to satisfy a length criterion.

9. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to merge a first one of the second bounding boxes and a second one of the second bound boxes based on a column connection criterion.

10. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to identify first words of the ones of the words assigned to the respective ones of the second bounding boxes that include a regular expression, the regular expression to correspond to a targeted fact.

11. The apparatus of claim 10 , wherein the targeted fact includes at least one of a product description, a quantity, or a price.

12. The apparatus of claim 1 , wherein the purchase information includes purchase details and promotion information, and wherein one or more of the at least one processor circuit is to;
identify first ones of the words assigned to the first bounding boxes that do not correspond to a respective classification of a respective bounding box; and
remove the first ones of the words from the first bounding boxes prior to extracting the purchase details.

13. The apparatus of claim 12 , wherein the purchase details include, for respective ones of the products in the receipt, a respective product description, a respective price, and a respective quantity.

14. The apparatus of claim 13 , wherein one or more of the at least one processor circuitry circuit is to cause decoding of the purchase details to decoding circuitry.

15. The apparatus of claim 1 , wherein the first price is an original price of the first product, the second price is a discount amount, and the third price is a purchase price for the first product.

16. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to apply a trained object detection model to the image to detect the regions.

17. At least one non-transitory computer readable storage medium comprising instructions that, when executed, to cause at least one processor circuit to at least:
cause generation of a first receipt region and a second receipt region based on an image of a receipt, the first and second receipt regions based on regions detected in the image;
apply a first mask to the first receipt region to generate first bounding boxes, the first bounding boxes corresponding to rows of the receipt;
apply a second mask to the second of the receipt region to generate second bounding boxes, the second bounding boxes corresponding to columns of the receipt;
form a structure of the receipt by (a) assigning words detected by an optical character recognition (OCR) engine to corresponding first bounding boxes based on a mapping criterion and (b) assigning ones of the words to respective ones of the second bounding boxes based on the mapping criterion;
cause classification of the the columns based on ones of the words assigned to the respective ones of the second bounding boxes;
extract purchase information from the structured receipt based on the classifications, the purchase information to include items listed in the receipt, the items to include a first item associated with a promotion; and
generate promotion information for the first item by:
identifying a first price associated with the first item, the first price corresponding identified from a first column classified as a price column;
identifying a second price associated with the first item, the second price corresponding to the promotion; and
determining a third price associated with the first item based on the first price and the second price.

18. The at least one non-transitory computer readable storage medium as defined in claim 17 , wherein the instructions, are to cause one or more of the at least one processor circuit to merge a first one of the first bounding boxes and a second one of the first bounding boxes based on a row connection criterion, the first one and the second one of the first bounding boxes to satisfy a length criterion.

19. The at least one non-transitory computer readable storage medium as defined in claim 17 , wherein the instructions cause one or more of the at least one processor circuit to apply a trained object detection model to the image to detect the regions.

20. An apparatus comprising:
means for generatin

Assignees

• Nielsen Consumer Llc

Inventors

• Yebes Torres Jose Javier
• Sinha Aditi
• Lebrun Christine
• Oppini Fabio
• Bansal Atul
• Kumar Mukul
• Chandramouli Vignesh
• Sousa Filipa
• Mercaldi Gisella

Classifications

• G06V30/15

  Cutting or merging image elements, e.g. region growing, watershed or clustering-based techniques · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V30/414

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06Q30/0283 Primary

  Price estimation or determination · CPC title

• G06V30/147

  Determination of region of interest · CPC title