Optical scanning using receipt imagery for automated tax reconciliation

Patent metadata

Field	Value
Publication number	US-12260438-B2
Application number	US-202217853619-A
Country	US
Kind code	B2
Filing date	Jun 29, 2022
Priority date	Mar 31, 2022
Publication date	Mar 25, 2025
Grant date	Mar 25, 2025

Abstract

An online concierge system requests an image of a receipt of an order from a picker after the picker fulfills the order at a store. The online concierge system performs image processing on the image of the receipt and uses machine learning and optical character recognition to determine a tax amount paid for the order and a confidence score associated with the tax amount. The online concierge system may use the machine learning model for segmenting extracted text in the image of the receipt into tokens. The online concierge system may then determine at least one token associated with a tax item and the tax amount associated with the tax item. The online concierge system communicates the tax amount to the store for reimbursement based on the tax amount and the confidence score.

First claim

What is claimed is:

1. A method comprising:
at a computer system comprising at least one processor and memory:
receiving, from a computing device associated with a shopper, an image of a receipt for an order that has been fulfilled;
applying an image processing algorithm to identify a tax item in the image of the receipt, wherein applying the image processing algorithm comprises:
determining a location of an instance of text that is related to the tax item in the image;
applying a bounding box around the instance of text in the image at the determined location; and
applying an optical character recognition (OCR) to the image within the bounding box to extract text that is related to the tax item from the image;

identifying, using the extracted text, a tax amount associated with the tax item in the image, the tax amount representing an amount of tax paid at a point-of-sale (POS) system, wherein identifying the tax amount comprises:
generating, using the extracted text, an input to a machine learning model by segmenting the extracted text into one or more tokens, wherein the machine learning model is trained with training data comprising tokens segmented from extracted text that are associated with actual tax paid,
applying the machine learning model to the one or more tokens, and
receiving, from the machine learning model, the tax amount associated with the tax item in the image;

determining a confidence score of the identified tax amount, the confidence score indicating a level of accuracy of the identified tax amount; and
performing an automated tax reconciliation process based on the tax amount and the confidence score.

2. The method of claim 1 , further comprising:
applying a confidence score thresholding logic to assign the confidence score in three categories including: high, intermediate, and low.

3. The method of claim 2 , further comprising:
sending, to an auditor, the image of the receipt and the tax amount having a low confidence score to confirm the tax amount is correct.

4. The method of claim 2 , further comprising:
sending, to an auditor, the image of the receipt and the tax amount having an intermediate confidence score and a large monetary value to confirm the tax amount is correct.

5. The method of claim 1 , further comprising:
displaying the image of the receipt to an auditor based on the tax amount and the confidence score; and
displaying, to the auditor, a notification comprising a selectable option to confirm whether the image is readable.

6. The method of claim 1 , further comprising:
displaying, to an auditor, a notification comprising the tax amount, an option to confirm the tax amount is correct, and an alternate option to decline the tax amount and input a correct amount of taxes paid.

7. The method of claim 1 , further comprising:
determining whether the image of the receipt is of sufficient quality to resolve text of the receipt; and
responsive to determining the image of the receipt is not of sufficient quality, prompting the shopper to take another image of the receipt.

8. The method of claim 1 , wherein performing the automated tax reconciliation process comprises:
retrieving tax data generated in a predetermined period of time, the tax data comprising one or more identified tax amounts;
aggregating the one or more identified tax amounts to generate a tax reconciliation data; and
sending the tax reconciliation data to a system to cause the system to reimburse the one or more identified tax amounts.

9. The method of claim 1 , wherein identifying the tax amount further comprises:
determining whether the image of the receipt is of sufficient quality to resolve text of the receipt;
responsive to determining the image of the receipt is of sufficient quality, after the text being extracted, using the machine learning model to identify the tax amount from the extracted text.

10. A non-transitory computer-readable storage medium storing instructions, the instructions when executed cause a processor to:
receive, from a computing device associated with a shopper, an image of a receipt for an order that has been fulfilled;
apply an image processing algorithm to identify a tax item in the image of the receipt, wherein applying the image processing algorithm comprises:
determining a location of an instance of text that is related to the tax item in the image;
applying a bounding box around the instance of text in the image at the determined location; and
applying an optical character recognition (OCR) to the image within the bounding box to extract text that is related to the tax item from the image;
identify, using the extracted text, a tax amount associated with the tax item in the image, the tax amount representing an amount of tax paid at a point-of-sale (POS) system, wherein identifying the tax amount comprises:
generating, using the extracted text, an input to a machine learning model by segmenting the extracted text into one or more tokens, wherein the machine learning model is trained with training data comprising tokens segmented from extracted text that are associated with actual tax paid,
applying the machine learning model to the one or more tokens, and
receiving, from the machine learning model, the tax amount associated with the tax item in the image;

determine a confidence score of the identified tax amount, the confidence score indicating a level of accuracy of the identified tax amount; and
perform an automated tax reconciliation process based on the tax amount and the confidence score.

11. The computer-readable storage medium of claim 10 , further comprising instructions that when executed cause the processor to:
apply a confidence score thresholding logic to assign the confidence score in three categories including: high, intermediate, and low.

12. The computer-readable storage medium of claim 11 , further comprising instructions that when executed cause the processor to:
send, to an auditor, the image of the receipt and the tax amount having a low confidence score to confirm the tax amount is correct.

13. The computer-readable storage medium of claim 11 , further comprising instructions that when executed cause the processor to:
send, to an auditor, the image of the receipt and the tax amount having an intermediate confidence score and a large monetary value to confirm the tax amount is correct.

14. The computer-readable storage medium of claim 10 , further comprising instructions that when executed cause the processor to:
display the image of the receipt to an auditor based on the tax amount and the confidence score; and
display, to the auditor, a notification comprising a selectable option to confirm whether the image is readable.

15. The computer-readable storage medium of claim 10 , wherein the instruction to perform the automated tax reconciliation process comprises:
retrieving tax data generated in a predetermined period of time, the tax data comprising one or more identified tax amounts;
aggregating the one or more identified tax amounts to generate a tax reconciliation data; and
sending the tax reconciliation data to a system to cause the system to reimburse the one or more identified tax amounts.

16. The computer-readable storage medium of claim 10 , further comprising instructions that when executed cause the processor to:
determine whether the image of the receipt is of sufficient quality to resolve text of the receipt; and
responsive to determining the image of the receipt is not of sufficient quality, prompt the shopper to take another image of the receipt.

Assignees

• Maplebear Inc

Inventors

• Knight Benjamin
• Peyrot Benjamin
• Gluhovic Djordje
• Turumella Rohit
• Han Alice

Classifications

• G06Q40/123

  Tax preparation or submission · CPC title

• G06Q40/12

  Accounting · CPC title

• G06Q20/207

  Tax processing · CPC title

• G06Q40/10 Primary

  Tax strategies · CPC title

• G06Q30/06

  Buying, selling or leasing transactions · CPC title