Detecting aspect ratios of document pages on smartphone photographs by learning camera view angles

Patent metadata

Field	Value
Publication number	US-10032073-B1
Application number	US-201615191800-A
Country	US
Kind code	B1
Filing date	Jun 24, 2016
Priority date	Jun 25, 2015
Publication date	Jul 24, 2018
Grant date	Jul 24, 2018

Abstract

Detecting an aspect ratio of an image captured with a smartphone includes detecting at least one convex quadrangle of arbitrary shape on the image and generating a plurality of additional convex quadrangles having vertices in a pre-determined vicinity of vertices of the quadrangle on the image. A linear projective mapping matrix is generated for mapping each of the quadrangle and the plurality of additional quadrangles onto a unit square. A plurality of estimated focal lengths of the camera of the smartphone is determined according to matrixes corresponding to the linear projective mappings onto a unit square of the quadrangle and each of the plurality of additional quadrangles. The quadrangle is used to determine the aspect ratio of the image in response to a range of the plurality of estimated focal lengths including a true value of the focal length of the camera of the smartphone.

First claim

What is claimed is:

1. A method of correcting image distortion, the method comprising:
storing an image of a physical object captured by a device that includes a camera;
analyzing the image to detect at least one convex quadrangle of arbitrary shape within the image;
generating a plurality of additional convex quadrangles, wherein a respective vertex of a respective additional convex quadrangle is in a pre-determined vicinity of a corresponding vertex of the at least one convex quadrangle within the image;
generating a plurality of linear projective mapping matrixes, including, for each of the at least one convex quadrangle and the plurality of additional convex quadrangles, generating a respective linear projective mapping matrix;
using the plurality of linear projective mapping matrixes to map each of the at least one convex quadrangle and the plurality of additional convex quadrangles onto a unit square;
determining a plurality of estimated focal lengths of the camera using the mappings of the plurality of linear projective mapping matrixes onto the unit square;
using the at least one convex quadrangle to determine the aspect ratio of the image in response to a range of the plurality of estimated focal lengths including a true value of the focal length of the camera; and
correcting distortion of the image using the determined aspect ratio.

2. A method, according to claim 1 , wherein the image is one of: a still photo captured with the camera of the device or a frame in a video flow captured with the camera of the device.

3. A method, according to claim 1 , wherein each of the plurality of the estimated focal lengths of the camera is determined using the formula:

C
⁡

(
Q
)

=

H
11

·

H
12

+

H
21

·

H
22

H
31

·

H
32

where H 11 , H 12 , H 21 , H 22 , H 31 , H 32 , are all components of the linear projective mapping matrix.

4. A method, according to claim 1 , wherein two opposite vertices of the at least one convex quadrangle are mapped onto two opposite corners of a new quadrangle having a predefined distance between the two opposite vertices and a predefined orientation of an interval connecting the two opposite vertices and wherein four coordinates of two unmapped vertices of the new quadrangle are used to determine the aspect ratio.

5. A method, according to claim 4 , wherein the four coordinates of two unmapped vertices of the new quadrangle are entered into a pre-built Support Vector Regression model which derives the aspect ratio.

6. A method, according to claim 5 , wherein the pre-built Support Vector Regression model is built by:
artificially generating a plurality of document pages that represent a range of aspect ratios; and
capturing images of each document page of the plurality of document pages from a plurality of view angles.

7. A method, according to claim 1 , further comprising:
lowering resolution of the image prior to detecting the at least one convex quadrangle.

8. A method, according to claim 7 , wherein the pre-determined vicinity depends upon the lowered resolution of the image.

9. A method, according to claim 1 , wherein the pre-determined vicinity varies according to a tolerance level of acceptance of the plurality of estimated focal lengths of the camera.

10. A method, according to claim 9 , wherein the tolerance level is set by a user.

11. A non-transitory computer-readable medium containing software that corrects image distortion, the software comprising:
executable code that stores an image of a physical object captured by a device that includes a camera;
executable code that analyzes the image to detect at least one convex quadrangle of arbitrary shape within the image;
executable code that generates a plurality of additional convex quadrangles, wherein a respective vertex of a respective additional convex quadrangle is in a pre-determined vicinity of a corresponding vertex of the at least one convex quadrangle within the image;
executable code that generates a plurality of linear projective mapping matrixes, including, for each of the at least one convex quadrangle and the plurality of additional convex quadrangles, generating a respective linear projective mapping matrix;
executable code that uses the plurality of linear projective mapping matrixes to map each of the at least one convex quadrangle and the plurality of additional convex quadrangles onto a unit square;
executable code that determines a plurality of estimated focal lengths of the camera using the mappings of the plurality of linear projective mapping matrixes onto the unit square;
executable code that uses the at least one convex quadrangle to determine the aspect ratio of the image in response to a range of the plurality of estimated focal lengths including a true value of the focal length of the camera; and
correcting distortion of the image using the determined aspect ratio.

12. A non-transitory computer-readable medium, according to claim 11 , wherein the image is one of: a still photo captured with the camera of the device or a frame in a video flow captured with the camera of the device.

13. A non-transitory computer-readable medium, according to claim 11 , wherein each of the plurality of the estimated focal lengths of the camera is determined using the formula:

C
⁡

(
Q
)

=

H
11

·

H
12

+

H
21

·

H
22

H
31

·

H
32

where

Assignees

• Evernote Corp

Inventors

• Buryak Ilya
• Livshitz Eugene
• Pashintsev Alexander
• Gorbatov Boris

Classifications

• H04N7/185 Primary

  from a mobile camera, e.g. for remote control · CPC title

• G06V30/414

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06V10/147

  Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) · CPC title

• G06K9/00463 Primary

  Physics · mapped topic

• G06V30/142

  using hand-held instruments; Constructional details of the instruments · CPC title