Systems and methods for automated machine learning model training for a custom authored prompt

Patent metadata

Field	Value
Publication number	US-11443140-B2
Application number	US-201916280984-A
Country	US
Kind code	B2
Filing date	Feb 20, 2019
Priority date	Feb 20, 2018
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

First claim

What is claimed is:

1. A system for training a model for a custom authored prompt, the system comprising:
a memory comprising:
a content library database comprising a plurality of prompts, and
a model database comprising at least one prompt evaluation model trained to evaluate prompts; and

at least one processor configured to:
receive a prompt;
parse the prompt to identify a plurality of prompt evaluation portions;
identify pre-existing data relevant to at least one prompt evaluation portion of the prompt evaluation portions;
train a response evaluation model for evaluating responses to the prompt at least in part based on the pre-existing data;
provide the prompt to a user;
receive a response to the provided prompt;
generate an evaluation of the received response;
determine sufficiency of the training of the response evaluation model;
update the training based on the received response and on the evaluation of the received response to determine new training data when the training of the response evaluation model is insufficient;
determine a plurality of responses and a plurality of evaluations of the plurality of received responses;
determine an ordering to the received responses;
train based on the ordering of the received responses; and
generate a training indicator, wherein the training indicator is configured to provide a graphical depiction of a degree to which the response evaluation model is trained based upon the updating of the training of the response evaluation model.

2. The system of claim 1 , wherein parsing the prompt comprises identifying a complexity of each of the plurality of prompt evaluation portions associated with the received prompt.

3. The system of claim 2 ,
wherein the pre-existing data comprises at least one of:
a pre-existing model trained to evaluate responses to another prompt, and
pre-existing response data generated from responses to other prompts.

4. The system of claim 3 , wherein the at least one processor is further configured to identify a creator of the received prompt, and wherein the at least one of:
the pre-existing model, and the pre-existing response data, are identified based on the creator of the received prompt.

5. The system of claim 4 , wherein identifying pre-existing response data comprises identifying response data corresponding to at least one of the plurality of prompt evaluation portions via a similarity score.

6. The system of claim 1 , wherein the at least one processor is further configured to control the training indicator to reflect the degree to which the response evaluation model is trained subsequent to the updating of the training of the response evaluation model.

7. A method of training a model for a custom authored prompt, the method comprising:
receiving a prompt;
parsing the prompt to identify a plurality of prompt evaluation portions associated with the received prompt;
identifying pre-existing data relevant to at least one of the prompt evaluation portions;
training a response evaluation model for evaluating responses to the prompt at least in part based on the pre-existing data;
receiving a response to the prompt;
generating an evaluation of the received response;
determine sufficiency of the training of the response evaluation model;
update the training based on the received response and on the evaluation of the received response to determine new training data when the training of the response evaluation model is insufficient;
determining a plurality of responses and a plurality of evaluations of the plurality of received responses;
training based on an ordering of the received responses; and
generating a training indicator, wherein the training indicator is configured to provide a graphical depiction of a degree to which the response evaluation model is trained based upon the updating of the training of the response evaluation model.

8. The method of claim 7 , wherein parsing the prompt comprises identifying a complexity of each of the plurality of prompt evaluation portions associated with the received prompt.

9. The method of claim 7 , wherein the pre-existing data comprises at least one of: a pre-existing model trained to evaluate responses to another prompt, and pre-existing response data generated from responses to other prompts.

10. The method of claim 9 , further comprising identifying a creator of the received prompt, and wherein the at least one of: the pre-existing model, and the pre-existing response data, are identified based on the creator of the received prompt.

11. The method of claim 9 , wherein identifying pre-existing response data comprises identifying response data corresponding to at least one of the plurality of prompt evaluation portions via a similarity score.

12. The method of claim 7 , further comprising:
providing the prompt to a user.

13. The method of claim 7 , wherein updating the training comprises determining the ordering to the received responses.

14. The method of claim 13 , further comprising controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Assignees

• Pearson Education Inc

Inventors

• Hellman Scott
• Becker Lee
• Downs Samuel
• Baikadi Alok
• Murray William
• Habermehl Kyle
• Foltz Peter
• Rosenstein Mark

Classifications

• G09B7/02 Primary

  of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title