Document analysis using machine learning and neural networks

Patent metadata

Field	Value
Publication number	US-11144880-B2
Application number	US-201816211631-A
Country	US
Kind code	B2
Filing date	Dec 6, 2018
Priority date	Dec 6, 2018
Publication date	Oct 12, 2021
Grant date	Oct 12, 2021

Abstract

A method for providing training data for a machine learning model includes: monitoring a specific user as the specific user reads electronic documents on a display to determine indications of pauses in reading for greater than a specified period of time; correlating objects on each of the displayed plurality of electronic documents to the pauses in reading; identifying features for the machine learning model based on the objects and textual analysis of each of the plurality of electronic documents; presenting information related to each identified feature to the specific user; obtaining from the specific user a descriptor defining each of the identified features and a value for each of the identified features indicating a relative importance or applicability of each of the identified features; and associating obtained descriptors and values with each of the identified features.

First claim

What is claimed is:

1. A method for providing training data for a machine learning model, the method comprising:
obtaining a selection from equipment of a specific user responsive presentation of proposed words according to an A/B test;
identifying, by application of the selection to a dedicated neural network, a pre-trained, baseline machine learning model;
displaying a plurality of electronic documents for evaluation on a display of a computing device to obtain a displayed plurality of electronic documents;
tracking, by an eye movement tracking device, a gaze of the specific user as the specific user reads each of the displayed plurality of electronic documents on the display to determine indications of pauses in reading each of the displayed plurality of electronic documents for greater than a specified period of time by the specific user, the pauses indicative of the specific user's gaze;
correlating objects on each of the displayed plurality of electronic documents to the pauses in reading by the specific user;
identifying features for the machine learning model to obtain identified features based on the objects and textual analysis of each of the plurality of electronic documents;
presenting information related to each identified feature to the specific user;
obtaining from the specific user a descriptor defining each of the identified features and a value for each of the identified features indicating a relative importance or applicability of each of the identified features;
associating obtained descriptors and values with each of the identified features;
obtaining from the specific user an overall value for each of the plurality of electronic documents indicating an overall applicability of each of the plurality of electronic documents with respect to specific requirements of the specific user;
associating obtained overall values with each of the plurality of electronic documents; and
combining the pre-trained, baseline machine learning model, identified features, associated descriptors, associated values, and associated overall values as training data for the machine learning model associated with the specific user.

2. The method of claim 1 , wherein a plurality of same features and a plurality of different features are identified among the plurality of electronic documents.

3. The method of claim 1 , further comprising:
determining that a value associated with an identified feature is an indication of bias; and
identifying the indication of bias in the training data for mitigation during subsequent electronic document evaluation.

4. The method of claim 1 , wherein the plurality of electronic documents comprise job applications or resumes; and
the features comprise one or more of logical skill development information, career progression information, misleading phrases related to employment experience, and gaps in employment history.

5. The method of claim 1 , wherein the objects are one of a word, a phrase, and a picture.

6. The method of claim 1 , wherein the determining indications of pauses in reading comprises:
tracking, by the eye movement tracking device, eye positions of the specific user's eyes on each of the displayed plurality of electronic documents as the specific user reads each of the displayed plurality of electronic documents; and
detecting pauses in movement of the specific user's eyes for greater than the specified period of time on a displayed electronic document or detecting repeated eye movements between different portions of the displayed electronic document.

7. The method of claim 6 , wherein the correlating of the objects on each of the displayed plurality of electronic documents to the pauses in reading comprises:
determining, based on data from the eye movement tracking device, the eye positions of the specific user's eyes on the display;
correlating portions of each of the displayed plurality of electronic documents with the eye positions of the specific user's eyes; and
identifying the objects corresponding to the portions of each of the displayed plurality of electronic documents.

8. The method of claim 1 , wherein the display is a touch-sensitive display,
wherein the eye movement tracking device comprises the touch-sensitive display,
wherein the specific user follows text of each of the displayed plurality of electronic documents across the touch-sensitive display with a finger or a stylus in contact with the touch-sensitive display as the specific user reads each of the displayed plurality of electronic documents, and
wherein a determining of the indications of pauses in reading comprises determining, based on data from the touch-sensitive display, pauses in movement of the finger or the stylus in contact with the touch-sensitive display for the specified period of time.

9. The method of claim 8 , wherein the correlating an object on each of the displayed plurality of electronic documents to the pauses in the specific user's reading comprises:
determining, based on the data from the touch-sensitive display, positions on the touch-sensitive display of the finger or the stylus in contact with the touch-sensitive display;
correlating portions of each of the displayed plurality of electronic documents with the positions of the finger or the stylus; and
identifying the objects corresponding to the portions of each of the displayed plurality of electronic documents.

10. The method of claim 1 , wherein the display is a touch-sensitive display configured to provide haptic feedback in response to a specified amount of pressure exerted on the display by a finger or a stylus,
wherein the specific user follows text of each of the displayed plurality of electronic documents with the finger or the stylus as the specific user reads each of the displayed plurality of electronic documents, and
wherein a determining of the indications of pause in reading comprises exerting the specified amount of pressure on the touch-sensitive display with the finger or the stylus to generate the haptic feedback for the specified period of time.

11. The method of claim 10 , wherein the correlating of the objects on each of the displayed plurality of electronic documents to the pauses in reading comprises:
determining, based on data from the touch-sensitive display, positions of the finger or the stylus exerting the specified amount of pressure on the touch-sensitive display to generate the haptic feedback for the specified period of time;
correlating portions of each of the displayed plurality of electronic documents with the positions of the finger or the stylus; and
identifying the objects corresponding to the portions of each of the displayed plurality of electronic documents.

12. The method of claim 1 , further comprising identifying training data for a plurality of different models, each of the plurality of different models trained based on training data identified for a different user.

13. A system, comprising:
a plurality of machine learning models, each of the plurality of machine learning models trained with initial training data generated by a training method including, for each machine learning model:
obtaining a selection from equipment of a specific user responsive presentation of proposed words according to an A/B test;
identifying, by application of the selection to a dedicated neural network, a pre-trained, baseline machine learning model;
displaying a plurality of electronic documents for evaluation on a display of a computing device to obtain a displayed plurality of electronic documents;
tracking, by an eye movement tracking device, a gaze of the specific user as the specific user read

Assignees

• At & T Ip I Lp

Inventors

• Mcenroe Martin Patrick
• Vishnyakov Andrew
• Flynn Daniel
• Rice Anna Kay

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title