Classifying Structural Features of a Digital Document by Feature Type using Machine Learning

Patent metadata

Field	Value
Publication number	US-2020302016-A1
Application number	US-201916359402-A
Country	US
Kind code	A1
Filing date	Mar 20, 2019
Priority date	Mar 20, 2019
Publication date	Sep 24, 2020
Grant date	—

Abstract

Classifying structural features of a digital document by feature type using machine learning is leveraged in a digital medium environment. A document analysis system is leveraged to extract structural features from digital documents, and to classifying the structural features by respective feature types. To do this, the document analysis system employs a character analysis model and a classification model. The character analysis model takes text content from a digital document and generates text vectors that represent the text content. A vector sequence is generated based on the text vectors and position information for structural features of the digital document, and the classification model processes the vector sequence to classify the structural features into different feature types. The document analysis system can generate a modifiable version of the digital document that enables its structural features to be modified based on their respective feature types.

First claim

What is claimed is:

1 . In a digital medium environment to extract structural features from a digital document and generate an editable version of the digital document, a method implemented by at least one computing device, the method comprising:
extracting, by the at least one computing device, structural features from a digital document, including position information for each of the structural features and text content from one or more of the structural features; generating, by the at least one computing device, text vectors for the one or more of the structural features by processing the text content and converting the text content into text vectors; generating, by the at least one computing device, a vector sequence that includes the text vectors and the position information for each of the structural features; classifying, by the at least one computing device, each of the structural features by feature type by processing the vector sequence to determine a document context for each of the structural features relative to the digital document, and classifying each of the structural features into a respective feature type based on the document context for each structural feature; and generating, by the at least one computing device, a modifiable version of the digital document that enables the structural features to be reformatted based on the feature type for each of the structural features.

2 . A method as described in claim 1 , wherein the position information for one or more of the structural features comprises coordinates for a corner of a bounding box of the one or more structural features, a width of the bounding box, and a height of the bounding box.

3 . A method as described in claim 1 , wherein said generating the text vectors comprises inputting the text content for a particular structural feature into a machine learning model, and receiving a text vector representation of the text content as output from the machine learning model.

4 . A method as described in claim 1 , wherein said generating the text vectors comprises, for a first portion of a text sequence of the text content:
inputting the first portion of the text sequence into a long short-term memory (LSTM) neural network that is trained to predict a subsequent character in a second portion of the text sequence; and receiving a text vector for the text sequence including a portion of the text vector that is based on the predicted subsequent character in the second portion of the text sequence.

5 . A method as described in claim 1 , wherein said generating the vector sequence comprises:
generating feature vectors for each of the structural features based on a text vector and position information for each of the structural features; and concatenating the feature vectors for the structural features to generate the vector sequence.

6 . A method as described in claim 1 , wherein said generating the vector sequence comprises:
generating feature vectors for each of the structural features based on a text vector and position information for each of the structural features; sorting the structural features into a sorted order based on their position in the digital document; and concatenating the feature vectors based on the sorted order to generate the vector sequence.

7 . A method as described in claim 6 , wherein the sorted order is based on a reading order for the digital document.

8 . A method as described in claim 1 , wherein said classifying each of the structural features by feature type comprises:
processing the vector sequence by inputting the vector sequence into a context determination machine learning model that considers structural features represented by the vector sequence in a forward and backward direction relative to the vector sequence, and receiving a context aware representation of the structural features from the context determination machine learning model; and inputting the context aware representation into a decoder machine learning model that is configured to categorize structural features into a defined set of categories of feature types, and receiving a feature type for each of the structural features as output from the decoder machine learning model.

9 . A method as described in claim 1 , wherein the modifiable version of the digital document enables a particular structural feature of the digital document to be modified while maintaining a semantic context of the particular structural feature relative to the digital document.

10 . In a digital medium environment to classify structural features of a digital document by feature type and to generate an editable version of the digital document, a method implemented by at least one computing device, the method comprising:
generating, by the at least one computing device, a vector sequence for the digital document by inputting text content and position information for structural features of the digital document to a trained machine learning system, and receiving the vector sequence as output from the machine learning system; classifying, by the at least one computing device, each of the structural features of the digital document by feature type by inputting the vector sequence to the machine learning system, and receiving the feature type for each of the structural features as output from the machine learning system based on the vector sequence; and generating, by the at least one computing device, a modifiable version of the digital document that enables the structural features to be reformatted based on the feature type for each of the structural features.

11 . A method as described in claim 10 , wherein the machine learning system includes a character analysis machine learning model and a classification machine learning model, and wherein the method further comprises training the machine learning system by:
training the character analysis machine learning model to predict text characters in text strings of the text content, and to generate text vectors that represent the text content; and training the classification machine learning model to receive the vector sequence including the text vectors as input, and to output the feature type for each of the structural features based on the vector sequence.

12 . A method as described in claim 10 , wherein said generating the vector sequence for the digital document further comprises:
receiving text vectors that represent the text content, and concatenating the text vectors and the position information to generate feature vectors that each represent a respective structural feature of the digital document; and concatenating the feature vectors to generate the vector sequence.

13 . A method as described in claim 10 , wherein said generating the vector sequence for the digital document comprises:
receiving text vectors that represent the text content, and concatenating the text vectors and the position information to generate feature vectors that each represent a respective structural feature of the digital document; sorting the structural features into a sorted order based on their relative position in the digital document; and concatenating the feature vectors based on the sorted order to generate the vector sequence.

14 . A method as described in claim 13 , wherein said sorting comprises a vertical sort starting from a top of the digital document to vertically locate structural features, and a horizontal sort to horizontally locate structural features relative to the vertical sort.

15 . A method as described in claim 10 , wherein the machine learning system comprises a character anal

Assignees

• Adobe Inc

Inventors

• Aggarwal Milan
• Krishnamurthy Balaji

Classifications

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V30/10

  Character recognition · CPC title

• G06V30/414

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title