Automatically generating labeled synthetic documents

Patent metadata

Field	Value
Publication number	US-11238312-B2
Application number	US-201916690695-A
Country	US
Kind code	B2
Filing date	Nov 21, 2019
Priority date	Nov 21, 2019
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating diverse and realistic synthetic documents using deep learning. In particular, the disclosed systems can utilize a trained neural network to generate realistic image layouts comprising page elements that comply with layout parameters. The disclosed systems can also generate synthetic content corresponding to the page elements within the image layouts. The disclosed systems insert the synthetic content into the corresponding page elements of documents based on the image layouts to generate synthetic documents.

First claim

What is claimed is:

1. A non-transitory computer readable medium for generating synthetic documents, the non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:
generate, utilizing a neural network, a new image layout comprising an image of a document indicating structure of page elements within the document;
generate a digital document corresponding to the new image layout, the digital document comprising page elements corresponding to the structure of page elements in the new image layout;
generate synthetic content corresponding to the page elements; and
generate a synthetic document by inserting the synthetic content into the corresponding page elements in the digital document.

2. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the new image layout utilizing a generative adversarial network.

3. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the new image layout by:
accessing a real document;
identifying a page element of the real document; and
replacing the page element of the real document with a new page element.

4. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the synthetic content corresponding to the page elements by generating, by utilizing a machine learning model, synthetic content corresponding to the page elements.

5. The non-transitory computer readable medium as recited in claim 1 , further comprising generating the synthetic content corresponding to the page elements by determining page elements of real content that correspond to the page elements of the synthetic document.

6. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the synthetic document by:
determining styling parameters for the synthetic document; and
applying the styling parameters to the synthetic content.

7. The non-transitory computer readable medium as recited in claim 6 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the styling parameters based on style characteristics of the page elements.

8. The non-transitory computer readable medium as recited in claim 6 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the styling parameters based on style characteristics of a distribution of real documents.

9. A system comprising:
one or more memory devices storing a neural network; and
at least one server configured to cause the system to:
determine layout parameters;
generate, utilizing the neural network, a plurality of new image layouts comprising images of documents indicating structures of page elements within the documents conforming to the layout parameters;
generate a plurality of digital documents corresponding to the plurality of new image layouts, each digital document of the plurality of digital documents comprising page elements corresponding to the structures of page elements in a corresponding new image layout;
determine content parameters for synthetic content corresponding to the page elements;
generate the synthetic content corresponding to the page elements, wherein the synthetic content conforms to the content parameters; and
generate a plurality of synthetic documents by inserting the synthetic content into the corresponding page elements of the plurality of digital documents.

10. The system as recited in claim 9 , wherein the at least one server is further configured to cause the system to generate element labels for the plurality of synthetic documents, wherein the element labels indicate the page elements within the plurality of synthetic documents.

11. The system as recited in claim 9 , wherein the at least one server is further configured to cause the system to train a machine learning model using the plurality of synthetic documents.

12. The system as recited in claim 9 , wherein the at least one server is further configured to cause the system to determine the layout parameters by analyzing layout characteristics of a plurality of real documents.

13. The system as recited in claim 9 , wherein the at least one server is further configured to cause the system to generate the plurality of new image layouts by:
generating, by an image layout prediction neural network, predicted image layouts;
feeding the predicted image layouts to an adversarial discrimination neural network to determine if the predicted image layouts resemble realistic image layouts; and
training the image layout prediction neural network based on the determination of the adversarial discrimination neural network.

14. The system as recited in claim 9 , wherein the at least one server is further configured to cause the system to generate the synthetic content corresponding to the page elements by:
training a machine learning model using real page content corresponding to real page elements; and
generating, by the trained machine learning model, synthetic content corresponding to the page elements.

15. The system as recited in claim 9 , wherein the at least one server is further configured to cause the system to generate the plurality of synthetic documents by:
determining styling parameters for the plurality of synthetic documents; and
applying the styling parameters to the synthetic content within the plurality of synthetic documents.

16. The system as recited in claim 15 , wherein the at least one server is further configured to cause the system to generate a synthetic document based on style characteristics of the page elements.

17. The system as recited in claim 15 , wherein the at least one server is further configured to cause the system to determine the styling parameters based on style characteristics of a distribution of real documents.

18. In a digital medium environment for training machine learning models using training documents, a computer-implemented method for generating synthetic documents with element labels comprising:
performing a step for generating a plurality of new image layouts conforming to layout parameters;
performing a step for generating synthetic content conforming to content parameters; and
generating a plurality of synthetic documents comprising the plurality of new image layouts comprising the synthetic content.

19. The computer-implemented method as recited in claim 18 , further comprising generating element labels for the plurality of synthetic documents, wherein the element labels indicate page elements within the plurality of synthetic documents.

20. The computer-implemented method as recited in claim 18 , wherein the layout parameters comprise user input.

Assignees

• Adobe Inc

Inventors

• Kaynig-Fittkau Verena
• Ravi Sruthi Madapoosi
• Cohn Richard
• Barmpalios Nikolaos
• Kraley Michael
• Sethu Kanchana

Classifications

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06V30/414 Primary

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06N3/045

  Combinations of networks · CPC title