Layout reconstruction using spatial and grammatical constraints

Patent metadata

Field	Value
Publication number	US-10565289-B2
Application number	US-201816138839-A
Country	US
Kind code	B2
Filing date	Sep 21, 2018
Priority date	Dec 5, 2013
Publication date	Feb 18, 2020
Grant date	Feb 18, 2020

Abstract

During an image-analysis technique, the system calculates features by performing image analysis (such as optical character recognition) on a received image of a document. Using these features, as well as spatial and grammatical constraints, the system determines a layout of the document. For example, the layout may be determined using constraint-based optimization based on the spatial and the grammatical constraints. Note that the layout specifies locations of content in the document, and may be used to subsequently extract the content from the image and/or to allow a user to provide feedback on the extracted content by presenting the extracted content to the user in a context (i.e., the determined layout) that is familiar to the user.

First claim

What is claimed is:

1. A method for determining a layout of information in a document, comprising:
receiving an image of a document;
determining a first, a second, and a third feature in the document based on the image of the document;
determining a layout of the document, wherein the layout specifies a first and a second location of content in the document, by:
determining the first location of content based on at least the first feature;
associating the first location of content with a first content type based on whether a grammatical constraint is satisfied by a second feature; and
associating the second location of content with a second content type based on the association between the first location of content and the first content type and whether a spatial constraint is satisfied by where the third feature is positioned with respect to the first feature;

determining a plurality of contents in the document based on the determined layout;
populating a plurality of fields in a form with the plurality of contents based on the determined layout; and
causing the form to be displayed with the plurality of fields populated with the plurality of contents.

2. The method of claim 1 , further comprising:
prior to determining the plurality of contents in the document:
requesting feedback regarding the determined layout from a user; and
receiving feedback from the user confirming that the layout matches the document.

3. The method of claim 1 , further comprising:
determining the spatial constraint based on evaluating the first and the third features with respect to a set of known spatial constraints stored in a data repository; and
determining the grammatical constraint based on evaluating the second feature with respect to a set of known grammatical constraints stored in the data repository.

4. The method of claim 3 , further comprising: associating the grammatical constraint with a known title associated with the determined layout using an n-gram model.

5. The method of claim 1 , wherein causing the form to be displayed with the plurality of fields populated with the plurality of contents comprises: providing the form with the plurality of fields populated with the plurality of contents to a remote electronic device for display thereon.

6. The method of claim 1 , wherein the spatial constraint is calculated based on a scale-invariant distance metric between the first and the third features.

7. The method of claim 1 , wherein the second feature corresponds to a set of alphanumeric characters proximate to the first feature.

8. The method of claim 1 , wherein the first, the second, and the third features comprise edges associated with objects in the document.

9. The method of claim 1 , wherein the first, the second, and the third features comprise corners associated with objects in document.

10. An apparatus, comprising:
a memory comprising computer-executable instructions;
a processor configured to execute the computer-executable instructions and cause the apparatus to perform a method for determining a layout of information in a document, the method comprising:
receiving an image of a document;
determining a first, a second, and a third feature in the document based on the image of the document;
determining a layout of the document, wherein the layout specifies a first and a second location of content in the document, by:
determining the first location of content based on at least the first feature;
associating the first location of content with a first content type based on whether a grammatical constraint is satisfied by a second feature; and
associating the second location of content with a second content type based on the association between the first location of content and the first content type and whether a spatial constraint is satisfied by where the third feature is positioned with respect to the first feature;

determining a plurality of contents in the document based on the determined layout;
populating a plurality of fields in a form with the plurality of contents based on the determined layout; and
causing the form to be displayed with the plurality of fields populated with the plurality of contents.

11. The apparatus of claim 10 , wherein the method further comprises:
prior to determining the plurality of contents in the document:
requesting feedback regarding the determined layout from a user; and
receiving feedback from the user confirming that the layout matches the document.

12. The apparatus of claim 10 , wherein the method further comprises:
determining the spatial constraint based on evaluating the first and the third features with respect to a set of known spatial constraints stored in a data repository; and
determining the grammatical constraint based on evaluating the second feature with respect to a set of known grammatical constraints stored in the data repository.

13. The apparatus of claim 12 , wherein the method further comprises: associating the grammatical constraint with a known title associated with the determined layout using an n-gram model.

14. The apparatus of claim 10 , wherein causing the form to be displayed with the plurality of fields populated with the plurality of contents comprises: providing the form with the plurality of fields populated with the plurality of contents to a remote electronic device for display thereon.

15. The apparatus of claim 10 , wherein the spatial constraint is calculated based on a scale-invariant distance metric between the first and the third features.

16. The apparatus of claim 10 , wherein the second feature corresponds to a set of alphanumeric characters proximate to the first feature.

17. The apparatus of claim 10 , wherein the first, the second, and the third features comprise edges associated with objects in the document.

18. The apparatus of claim 10 , wherein the plurality of features comprise corners associated with objects in document.

19. A method for determining a layout of information in a document, comprising:
determining a set of spatial constraints and a set of grammatical constraints by applying a machine-learning technique to a set of documents, the set of documents comprising a first and second plurality of documents, the set of grammatical constraints comprising a grammatical constraint that is associated with the first plurality of documents and the set of spatial constraints comprising a spatial constraint that is associated with the second plurality of documents;
storing the set of spatial constraints and the set of grammatical constraints in a data repository;
receiving an image of a document;
determining a first, a second, and a third feature in the document based on the image of the document, wherein the second feature corresponds to a set of alphanumeric characters;
determining a layout of the document, wherein the layout specifies a first and a second location of content in the document, by:
determining the first location of content based on a comparison between the first feature and at least one feature associated with the set of documents;
associating the first location of content with a first content type based on where the set of alphanumeric characters is positioned with respect to the first feature and whether the grammatical constraint is satisfied by the set of alphanumeric characters; and
associating the second location of content with a second content type based on the association between the first location of content and the first content type and whether the spatial constraint i

Assignees

• Intuit Inc

Inventors

• Huang Nankun
• Howe Carol A
• Dye Christopher M
• Bamford Robert

Classifications

• G06V30/18171

  Syntactic representation, e.g. using a grammatical approach · CPC title

• G06V30/18143

  Extracting features based on salient regional features, e.g. scale invariant feature transform [SIFT] keypoints · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06F40/174

  Form filling; Merging · CPC title

• G06F40/106 Primary

  Display of layout of documents; Previewing · CPC title