Triggering execution of machine learning based prediction of document metadata

Patent metadata

Field	Value
Publication number	US-12222975-B2
Application number	US-202318096994-A
Country	US
Kind code	B2
Filing date	Jan 13, 2023
Priority date	Jan 13, 2023
Publication date	Feb 11, 2025
Grant date	Feb 11, 2025

Abstract

A system predicts metadata attributes associated with documents using machine learning models. The document may represent an interaction between entities. The system trains machine learning models to predict scores indicating whether a token or a sequence of token of a document represents a metadata attribute. The metadata prediction is used to annotate the document and display to users. The system receives user feedback via the user interface and uses the user feedback to evaluate or retrain the model. The system generates training data by receiving a set of annotated documents and comparing the annotated documents against other documents to identify matching documents. The system determines when to execute the machine learning based metadata prediction based on steps of document workflow executed by the system.

First claim

What is claimed is:

1. A computer-implemented method for machine learning based prediction of document metadata, the computer-implemented method comprising:
execute, by a document management system, document workflows for one or more documents, wherein each of the one or more documents represents an interaction between a plurality of entities, each document workflow comprising a plurality of steps;
receive one or more trigger criteria, each trigger criterion specifying conditions for triggering execution of machine learning based prediction of metadata attributes for a document;
execute, for each document of the one or more documents:
receiving status of execution of a particular step of document workflow for the document;
evaluating a trigger criterion associated with the document based on the received status of execution of the particular step;
responsive to the evaluation of the trigger criterion indicating that metadata prediction should be triggered, send request for prediction of metadata attributes based on the document by executing one or more machine learning models trained to predict a likelihood that a portion of an input document represents a metadata attribute describing an interaction between a plurality of entities, wherein the portion of the input document comprises a token or sequence of tokens from the document associated with the metadata attribute;
annotate the document with one or more metadata attributes predicted using the one or more machine learning models; and
cause for display, the annotated document via a user interface.

2. The computer-implemented method of claim 1 , wherein a trigger criterion specifies a type of action associated with the document performed in a particular step of the document workflow.

3. The computer-implemented method of claim 1 , wherein the steps of the document workflow include one or more of: a document upload step, a document update step, a document signing step, an identity verification step, or a step for configuring and presenting a form for receiving information.

4. The computer-implemented method of claim 1 , wherein a document workflow for a document is performed by execution of a document workflow specification comprising a sequence of steps associated with the document, the document workflow executed on a cloud platform.

5. The computer-implemented method of claim 1 , further comprising:
receive user feedback via the user interface, the user feedback comprising one or more of:
(1) a correction of a predicted metadata attribute or
(2) an approval of a predicted metadata attribute.

6. The computer-implemented method of claim 5 , further comprising:
use the user feedback for evaluation of the one or more machine learning models.

7. The computer-implemented method of claim 5 , further comprising:
use the user feedback for generating training data for retraining the one or more machine learning models.

8. The computer-implemented method of claim 1 , wherein a machine learning model predicts a score indicating that the portion of the input document represents a date associated with the interaction between the plurality of entities.

9. The computer-implemented method of claim 1 , wherein a machine learning model predicts a role of an entity from the plurality of entities.

10. A non-transitory computer-readable storage medium storing executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations comprising:
execute, by a document management system, document workflows for one or more documents, wherein each of the one or more documents represents an interaction between a plurality of entities, each document workflow comprising a plurality of steps;
receive one or more trigger criteria, each trigger criterion specifying conditions for triggering execution of machine learning based prediction of metadata attributes for a document;
execute, for each document of the one or more documents:
receive status of execution of a particular step of document workflow for the document;
evaluate a trigger criterion associated with the document based on the received status of execution of the particular step;
responsive to the evaluation of the trigger criterion indicating that metadata prediction should be triggered, send request for prediction of metadata attributes based on the document by executing one or more machine learning models trained to predict a likelihood that a portion of an input document represents a metadata attribute describing an interaction between a plurality of entities, wherein the portion of the input document comprises a token or sequence of tokens from the document associated with the metadata attribute;
annotate the document with one or more metadata attributes predicted using the one or more machine learning models; and
cause for display, the annotated document via a user interface.

11. The non-transitory computer-readable storage medium of claim 10 , wherein a trigger criterion specifies a type of action associated with the document performed in a particular step of the document workflow.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the steps of the document workflow include one or more of: a document upload step, a document update step, a document signing step, an identity verification step, or a step for configuring and presenting a form for receiving information.

13. The non-transitory computer-readable storage medium of claim 10 , wherein a document workflow for a document is performed by execution of a document workflow specification comprising a sequence of steps associated with the document, the document workflow executed on a cloud platform.

14. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions further cause the one or more computer processors to perform steps comprising:
receive user feedback via the user interface, the user feedback comprising one or more of:
(1) a correction of a predicted metadata attribute or
(2) an approval of a predicted metadata attribute.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the one or more computer processors to perform steps comprising:
use the user feedback for evaluation of the one or more machine learning models.

16. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the one or more computer processors to perform steps comprising:
use the user feedback for generating training data for retraining the one or more machine learning models.

17. The non-transitory computer-readable storage medium of claim 10 , wherein a machine learning model predicts a score indicating that the portion of the input document represents a date associated with the interaction between the plurality of entities.

18. The non-transitory computer-readable storage medium of claim 10 , wherein a machine learning model predicts a role of an entity from the plurality of entities.

19. A computer system comprising:
one or more computer processors; and
a non-transitory computer-readable storage medium storing executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
execute, by a document management system, document workflows for one or more documents, wherein each of the one or more documents represents an interaction between a plurality of entities, eac

Assignees

• Docusign Inc

Inventors

• Narayanan Kaushik
• Wong David Matthew
• Lange David
• Jethava Vinay
• Zheng Qing
• Ghanimifard Mohammad Mehdi
• Lindstrom Pontus
• Raman Gowtham Rangarajan
• Debarr David
• He Yan

Classifications

• G06F16/34

  Browsing; Visualisation therefor (browsing or visualisation for clustering or classification G06F16/358) · CPC title

• G06F16/383 Primary

  using metadata automatically derived from the content · CPC title