Extracting structured data from handwritten and audio notes

Patent metadata

Field	Value
Publication number	US-11550995-B2
Application number	US-202016908590-A
Country	US
Kind code	B2
Filing date	Jun 22, 2020
Priority date	Mar 30, 2016
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

This application is directed to recognizing unstructured information based on hints provided by structured information. A computer system obtains unstructured information collected from a handwritten or audio source, and identifies one or more terms from the unstructured information. The one or more terms includes a first term that is ambiguous. The computer system performs a recognition operation on the first term to derive a first plurality of candidate terms for the first term, and obtains first contextual information from an information template associated with the unstructured information. In accordance with the first contextual information, the computer system selects a first answer term from the first plurality of candidate terms, such that the first term is recognized as the first answer term.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
at a computer system having one or more processors and memory storing one or more programs executed by the one or more processors:
obtaining unstructured information collected from a handwritten or audio source;
extracting one or more terms from the unstructured information, the one or more terms including at least one ambiguous term;
determining (i) a first ambiguous term from the unstructured information and (ii) a first plurality of candidate terms for the first ambiguous term, wherein each candidate of the plurality of candidate terms is a potential interpretation of the first ambiguous term;
obtaining contextual information from an information template associated with the unstructured information and stored in the computer system, wherein the contextual information includes at least a form field name;

selecting a first answer term from the first plurality of candidate terms based on a form field name from the contextual information and the first ambiguous term; and
presenting the first answer term in association with the corresponding form field name of the information template.

2. The method of claim 1 , further comprising:
determining a second term to derive a second plurality of candidate terms, wherein the second term is adjacent to the first answer term;
obtaining contextual information from an information template associated with the unstructured information;
selecting a second answer term from the second plurality of candidate terms based on the contextual information and at least one other adjacent term; and
presenting the second answer term in association with a corresponding portion of the information template.

3. The method of claim 2 , further comprising:
updating the first answer term based on the second answer term.

4. The method of claim 3 , wherein updating the first answer term based on the second answer term includes expanding the first answer term to include the second answer term.

5. The method of claim 2 , further comprising:
removing the first answer term based on the second answer term.

6. The method of claim 2 , further comprising:
selecting a third answer term from the first plurality of candidate terms based on the form field name of the contextual information; and
presenting the third answer term in association with the corresponding form field name of the information template;
determining whether the first answer term and the third answer term are conflicting; and
in accordance with a determination that the first answer term and the third answer term are conflicting, resolving the conflict.

7. The method of claim 6 , wherein resolving the conflict includes:
validating the first answer term and the third answer term with the contextual information from the information template and/or an answer term of a term adjacent to the first answer term.

8. The method of claim 6 , wherein resolving the conflict includes:
obtaining second contextual information from a parallel source that is independent from the handwritten or audio source and the information template; and
validating the first answer term and the third answer term based on the second contextual information.

9. The method of claim 6 , further comprising:
determining whether the conflict was resolved;
in accordance with the conflict not being resolved, displaying a user interface to present the first and third answer terms; and
receiving a user selection of the first answer term, such that the first answer term is associated with the corresponding form field name of the information template based on the user selection.

10. The method of claim 1 , wherein the first plurality of candidate terms corresponds to different segmentations or recognitions of the first answer term.

11. The method of claim 1 , wherein the contextual information includes a plurality of contextual options, wherein the first answer term at least partially matches one of the plurality of contextual options, and other candidate terms of the first plurality of candidate terms do not match any of the plurality of contextual options.

12. The method of claim 1 , wherein:
the information template further comprises a plurality of data units, each data unit having a field name and a form field associated with the field name, the plurality of data units further including a first data unit;
the contextual information relates to the form field of the first data unit; and
the first answer term is associated with the form field of the first data unit.

13. The method of claim 12 , wherein the form field corresponds to a plurality of predefined menu entries, and the first answer term at least partially matches one of the plurality of predefined menu entries.

14. The method of claim 1 , further comprising:
building one or more attachment tokens and hints based on at least the first answer term; and
associating the one or more attachment tokens and hints with the information template.

15. A computer system, comprising:
one or more processors; and
memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for:
obtaining unstructured information collected from a handwritten or audio source;
extracting one or more terms from the unstructured information, the one or more terms including at least one ambiguous term;
determining (i) a first ambiguous term from the unstructured information and (ii) a first plurality of candidate terms for the first ambiguous term, wherein each candidate of the plurality of candidate terms is a potential interpretation of the first ambiguous term;
obtaining contextual information from an information template associated with the unstructured information and stored in the computer system, wherein the contextual information includes at least a form field name;
selecting a first answer term from the first plurality of candidate terms based on a form field name from the contextual information and the first ambiguous term; and
presenting the first answer term in association with the corresponding form field name of the information template.

16. The computer system of claim 15 , wherein the one or more programs further comprise instructions for:
determining a second term to derive a second plurality of candidate terms, wherein the second term is adjacent to the first answer term;
obtaining contextual information from an information template associated with the unstructured information;
selecting a second answer term from the second plurality of candidate terms based on the contextual information and at least one other adjacent term; and
presenting the second answer term in association with a corresponding portion of the information template.

17. The computer system of claim 16 , wherein the one or more programs further comprise instructions for:
updating the first answer term based on the second answer term.

18. A non-transitory, computer readable storage medium storing one or more programs configured for execution by a computer system, the one or more programs comprising instructions for:
obtaining unstructured information collected from a handwritten or audio source;
extracting one or more terms from the unstructured information, the one or more terms including at least one ambiguous term;
determining (i) a first ambiguous term from the unstructured information and (ii) a first plurality of candidate terms for the first ambiguous term, wherein each candidate of the plurality of candidate terms i

Assignees

• Evernote Corp

Inventors

• O'Neill Chris
• Malcolm Andrew
• Pashintsev Alexander
• Livshitz Eugene
• Gorbatov Boris
• Galaktionova Natalia
• Buryak Ilya

Classifications

• G06F40/279 Primary

  Recognition of textual entities · CPC title

• G06F40/186 Primary

  Templates · CPC title