Object detection and image classification based optical character recognition

Patent metadata

Field	Value
Publication number	US-11100319-B2
Application number	US-202016773813-A
Country	US
Kind code	B2
Filing date	Jan 27, 2020
Priority date	Jan 27, 2020
Publication date	Aug 24, 2021
Grant date	Aug 24, 2021

Abstract

A system performs optical character recognition (OCR) on an image displaying a portion of an object. An image classification system identifies the object in the image, based on which one or more object detection models identify labels associated with the object within the image. The system determines text of the identified labels using OCR, and analyzes the OCR resultant text for discrepancies and/or inaccuracies. In response to identifying a discrepancy, the system provides a recommendation for improving the accuracy of the OCR resultant text.

First claim

We claim:

1. A computer implemented method for performing optical character recognition comprising:
receiving an image displaying a portion of an object, the image displaying one or more labels of the object;
accessing an image classification model configured to receive an input image displaying a portion of an object and determine an object type of the object, the object type associated with one or more label types;
determining an object type of the object of the received image using the image classification model;
accessing one or more object detection models for the determined object type, each object detection model configured to receive an image as input and identify a label of a label type associated with the object type;
for each accessed object detection model, providing the received image as input to the object detection model to detect a label of a label type in the received image;
for each detected label of a label type in the received image:
determining a text of the detected label using optical character recognition;
accessing expected phrase types for phrases of the label type; and
analyzing the text based on the expected phrase type to identify discrepancies in the text; and

responsive to identifying a discrepancy in a text of a label, sending a recommendation for improving an accuracy of the text of the label determined using optical character recognition.

2. The computer implemented method of claim 1 , further comprising:
identifying a plurality of zones of the received image, wherein at least two of the zones have an overlapping area comprising a text;
for each of the plurality of zones, predicting a value of the text; and
responsive to identifying differences in the predicted values of the text, determining that the value of the text is the predicted value of the text having the highest number of occurrences.

3. The computer implemented method of claim 2 , the method further comprising:
responsive to determining that two predicted values of the text have the highest number of occurrences, combining at least two of the plurality of zones to obtain a tiebreaker zone;
predicting a value of the text for the tiebreaker zone; and
determining that the value of the text is the predicted value of the text for the tiebreaker zone.

4. The computer implemented method of claim 3 , the method further comprising:
determining a size of each zone of the plurality of zones; and
creating multiple copies of the image;
cropping each copy of the image to determine a zone such that each phrase being recognized is included in a set of zones having the determined size.

5. The computer implemented method of claim 3 , the method further comprising:
responsive to determining that the value of the text is not the predicted value of the text for the tiebreaker zone, sending the recommendation for improving the accuracy of the text of the label determined using optical character recognition.

6. The computer implemented method of claim 5 , wherein the recommendation for improving the accuracy of the text of the label includes capturing a new image displaying the portion of the object.

7. The computer implemented method of claim 1 , wherein identifying a discrepancy in a text comprises determining that an expected data type of a value in the expected phrase type fails to match an actual data type of the value in the text.

8. The computer implemented method of claim 1 , wherein identifying a discrepancy in a text comprises comparing a value in the text against a predetermined list of expected values for the text.

9. A computer readable non-transitory storage medium storing instructions for:
receiving an image displaying a portion of an object, the image displaying one or more labels of the object;
accessing an image classification model configured to receive an input image displaying a portion of an object and determine an object type of the object, the object type associated with one or more label types;
determining an object type of the object of the received image using the image classification model;
accessing one or more object detection models for the determined object type, each object detection model configured to receive an image as input and identify a label of a label type associated with the object type;
for each accessed object detection model, providing the received image as input to the object detection model to detect a label of a label type in the received image;
for each detected label of a label type in the received image:
determining a text of the detected label using optical character recognition;
accessing expected phrase types for phrases of the label type; and
analyzing the text based on the expected phrase type to identify discrepancies in the text; and

responsive to identifying a discrepancy in a text of a label, sending a recommendation for improving an accuracy of the text of the label determined using optical character recognition.

10. The computer readable non-transitory storage medium of claim 9 , further storing instructions for:
identifying a plurality of zones of the received image, wherein at least two of the zones have an overlapping area comprising a text;
for each of the plurality of zones, predicting a value of the text; and
responsive to identifying differences in the predicted values of the text, determining that the value of the text is the predicted value of the text having the highest number of occurrences.

11. The computer readable non-transitory storage medium of claim 10 , further storing instructions for:
responsive to determining that two predicted values of the text have the highest number of occurrences, combining at least two of the plurality of zones to obtain a tiebreaker zone;
predicting a value of the text for the tiebreaker zone; and
determining that the value of the text is the predicted value of the text for the tiebreaker zone.

12. The computer readable non-transitory storage medium of claim 11 , further storing instructions for:
determining a size of each zone of the plurality of zones;
creating multiple copies of the image; and
cropping each copy of the image to determine a zone such that each phrase being recognized is included in a set of zones having the determined size.

13. The computer readable non-transitory storage medium of claim 11 , further storing instructions for:
responsive to determining that the value of the text is not the predicted value of the text for the tiebreaker zone, sending the recommendation for improving the accuracy of the text of the label determined using optical character recognition.

14. The computer readable non-transitory storage medium of claim 13 , wherein the recommendation for improving the accuracy of the text of the label includes capturing a new image displaying the portion of the object.

15. The computer readable non-transitory storage medium of claim 9 , wherein identifying a discrepancy in a text comprises determining that an expected data type of a value in the expected phrase type fails to match an actual data type of the value in the text.

16. The computer readable non-transitory storage medium of claim 9 , wherein identifying a discrepancy in a text comprises comparing a value in the text against a predetermined list of expected values for the text.

17. A computer-implemented system comprising:
a computer processor; and
a computer readable non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to perform the steps of:
receiving an image displaying a

Assignees

• Salesforce Com Inc

Inventors

• Schultz Dennis
• Harrison Daniel Thomas
• Kemp Christopher Anthony
• Salem Michael A

Classifications

• G06V20/63 Primary

  Scene text, e.g. street names · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V30/10

  Character recognition · CPC title

• G06K9/2063

  Physics · mapped topic