Using a front-facing camera to improve OCR with a rear-facing camera

Patent metadata

Field	Value
Publication number	US-9269009-B1
Application number	US-201414283115-A
Country	US
Kind code	B1
Filing date	May 20, 2014
Priority date	May 20, 2014
Publication date	Feb 23, 2016
Grant date	Feb 23, 2016

Abstract

Various embodiments enable a computing device to incorporate frame selection or preprocessing techniques into a text recognition pipeline in an attempt to improve text recognition accuracy in various environments and situations. For example, a mobile computing device can capture images of text using a first camera, such as a rear-facing camera, while capturing images of the environment or a user with a second camera, such as a front-facing camera. Based on the images captured of the environment or user, one or more image preprocessing parameters can be determined and applied to the captured images in an attempt to improve text recognition accuracy.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause a computing device to:
acquire, using a rear-facing camera of the computing device, a plurality of image frames;
acquire, using a front-facing camera of the computing device, information corresponding to an environment of the computing device;
determine one or more conditions of the environment of the computing device using the information acquired by the front-facing camera of the computing device;
select, using at least one frame selection parameter associated with the one or more conditions, a first frame of the plurality of image frames for processing by an optical character recognition (OCR) engine in electronic communication with the computing device;
determine at least one threshold for performing binarization of the first frame, wherein the at least one threshold is based on the one or more conditions;
binarize, using the at least one threshold, at least a portion of the first frame; and
cause the at least a portion of the binarized first frame to be processed using the OCR engine.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the computing device to binarize the at least a portion of the first frame by:
determining a first portion of the first frame having pixel values above a character threshold value to be a character portion; and
determining a second portion of the first frame having pixel values below a background threshold value to be a background portion.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the computing device to select the first frame by:
determining a focus measure value or a contrast measure value.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the computing device to acquire information corresponding to an environment of the computing device by:
acquiring an image with the front-facing camera;
identifying an object in the image acquired by the front-facing camera; and
comparing, using an object matching algorithm, the object to objects stored in a database.

5. A computer-implemented method, comprising:
under the control of one or more computer systems configured with executable instructions,
acquiring, using a first camera of a computing device, at least one first image;
acquiring, using a second camera of the computing device, information corresponding to an environment of the computing device, wherein the second camera faces a different direction than the first camera;
determine one or more conditions of the environment using the information acquired by the second camera of the computing device;
determining at least one parameter associated with the one or more conditions;
performing at least one preprocessing operation associated with the at least one first image, wherein the at least one preprocessing operation includes binarizing at least a portion of each of the at least one first image based upon the one or more conditions; and
causing the at least one first image to be processed using an optical character recognition (OCR) engine in electronic communication with at least one of the one or more computer systems, wherein (i) the at least one parameter is used when performing the preprocessing operation or (ii) the at least one parameter is used by the OCR engine.

6. The computer-implemented method of claim 5 , wherein the at least one preprocessing operation further includes:
determining, based at least in part on the at least one parameter, a background threshold value or a character threshold value for text of the at least one first image, and
wherein the binarizing uses the background threshold and the character threshold.

7. The computer-implemented method of claim 6 , further comprising:
determining that the environment is associated with at least one of high lighting conditions or a high contrast measure based at least in part on the at least one first image;
causing, based on a determination that the environment is associated with at least one of high lighting conditions or a high contrast measure based at least in part on the at least one first image, the at least one parameter to include a first focus measure, wherein the first focus measure is lower than a second focus measure, the second focus measure associated with low lighting conditions or image frames having a low contrast measure; and
causing, based on the determination, the background threshold to be defined at a first intensity value and the character threshold to be defined at a second intensity value, wherein the first intensity value and second intensity value are separated by a first difference, wherein the first difference is greater than a second difference, the second difference associated with low lighting conditions or image frames having low contrast measure.

8. The computer-implemented method of claim 5 , further comprising:
determining, based on the at least one first image, that the environment is associated with at least one of a lowlighting condition or a low contrast measure; and
causing, based on a determination that the environment is associated with at least one of a lowlighting condition or a low contrast measure based at least in part on the at least one first image, the at least one parameter to include a first focus measure, wherein the first focus measure is higher than a second focus measure, the second focus measure associated with a high lighting condition or high contrast environment.

9. The computer-implemented method of claim 8 , further comprising:
causing, based on a determination that the environment is associated with at least one of a lowlighting condition or a low contrast measure based at least in part on the at least one first image, a set of additional images to be processed relative to the low lighting condition or the low contrast environment.

10. The computer-implemented method of claim 5 , further comprising:
analyzing the information corresponding to the environment to identify objects captured by the second camera; and
comparing, using an object matching algorithm, the objects captured by the second camera to objects stored in a database, wherein at least a portion of the objects stored in the database are associated with one of a plurality of environments.

11. The computer-implemented method of claim 5 , wherein the information corresponding to the environment captured by the second camera is a representation of a face and the at least one parameter indicates a facial expression.

12. The computer-implemented method of claim 11 , further comprising:
displaying a first recognition result from the OCR engine for the at least one first image;
determining, based at least in part on a first facial expression, dissatisfaction with the first recognition result;
acquiring, using the first camera, at least one second image; and
causing the at least one second image to be processed using the OCR engine.

13. The computer-implemented method of claim 12 , further comprising:
displaying a second recognition result for the at least one second image;
acquiring, using the second camera, information corresponding to a second facial expression; and
determining, based at least in part on the second facial expression, satisfaction with the second recognition result.

14. The computer-

Assignees

• Amazon Tech Inc

Inventors

• Liu Yue
• Jahagirdar Sonjeev
• Cole Matthew Joseph
• Prateek Utkarsh
• Mcconville Emilie Noelle
• Wilenson Daniel Makoto
• Sikka Avnish

Classifications

• G06V30/127 Primary

  with the intervention of an operator · CPC title

• G06V30/224

  of printed characters having additional code marks or containing code marks · CPC title

• G06V30/10

  Character recognition · CPC title

• G06K9/18 Primary

  Physics · mapped topic

• G06K2209/01

  Physics · mapped topic