Systems and methods for automated evaluation model customization

Patent metadata

Field	Value
Publication number	US-11475245-B2
Application number	US-201916280639-A
Country	US
Kind code	B2
Filing date	Feb 20, 2019
Priority date	Feb 20, 2018
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

First claim

What is claimed is:

1. A system for customizing a evaluation model to an evaluation style, the system comprising:
a memory comprising: a content library database containing a plurality of prompts and evaluation data associated with each of the plurality of prompts; and a model database comprising a plurality of evaluation models for automated evaluation of received user responses, wherein the evaluation data of each of the plurality of prompts comprises a pointer linking to the associated evaluation model; and
at least one processor configured to:
receive a plurality of responses received from a plurality of users in response to providing of a prompt;
identify an evaluation model relevant to the provided prompt, wherein the evaluation model comprises a machine learning model trained to output a score relevant to at least portions of a response;
generate a training indicator, wherein the training indicator provides a graphical depiction of the degree to which the identified evaluation model is trained;
determine a training status of the identified model;
when the model is identified as insufficiently trained, receive at least one evaluation input;
update training of the evaluation model based on the at least one received evaluation input; and
control the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

2. The system of claim 1 , wherein the evaluation model comprises a plurality of evaluation models.

3. The system of claim 2 , wherein each of the plurality of evaluation models is associated with an evaluation portion of the provided prompt.

4. The system of claim 3 , wherein the at least one processor is further configured to determine a first response ordering, wherein the first response ordering identifies an order of providing responses to the user for evaluation.

5. The system of claim 4 , wherein the response ordering is determined based on the estimated contribution of each response towards completion of training of the evaluation model.

6. The system of claim 5 , wherein the at least one processor is further configured to determine a second training status of the identified model subsequent to the updating of the training of the evaluation model based on the at least one received evaluation input.

7. The system of claim 6 , wherein the at least one processor is further configured to: auto-evaluate the response when the second training status of the identified model is identified as sufficiently trained; determine an acceptability of the auto-evaluating of the response; and determine a second response ordering when the auto-evaluating of the response is determined as unacceptable.

8. The system of claim 7 , wherein identifying an evaluation model relevant to the provided prompt comprises: identifying prompt evaluation portions; and retrieving a sub-model associated with each of the identified prompt evaluation portions.

9. The system of claim 8 , wherein the at least one server is further configured to determine a training level of the identified model.

10. The system of claim 9 , wherein determining a training level of the identified model comprises: retrieving sub-model data for each of the retrieved sub-models; determining a sub-model confidence level for each of the retrieved sub-models; and determining an aggregate confidence level based on a combination of the determined sub-model confidence levels.

11. A method of customizing an evaluation model to an evaluation style, the method comprising:
receiving a plurality of responses received from a plurality of users in response to providing of a prompt;
identifying an evaluation model relevant to the provided prompt, wherein the evaluation model comprises a machine learning model trained to output a score relevant to at least portions of a response;
generating a training indicator, wherein the training indicator provides a graphical depiction of the degree to which the identified evaluation model is trained;
determining a training status of the identified model;
when the model is identified as insufficiently trained, receiving at least one evaluation input;
updating training of the evaluation model based on the at least one received evaluation input; and
controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

12. The method of claim 11 , wherein the evaluation model comprises a plurality of evaluation models.

13. The method of claim 12 , wherein each of the plurality of evaluation models is associated with an evaluation portion of the provided prompt.

14. The method of claim 13 , further comprising determining a first response ordering, wherein the first response ordering identifies an order of providing responses for evaluation by the user.

15. The method of claim 14 , wherein the response ordering is determined based on the estimated contribution of each response towards completion of training of the evaluation model.

16. The method of claim 15 , further comprising: determining a second training status of the identified model subsequent to the updating of the training of the evaluation model based on the at least one received evaluation input.

17. The method of claim 16 , further comprising: auto-evaluating the response when the second training status of the identified model is identified as sufficiently trained; determining an acceptability of the auto-evaluating of the response; and determining a second response ordering when the auto-evaluating of the response is determined as unacceptable.

18. The method of claim 17 , wherein identifying an evaluation model relevant to the provided prompt comprises: identifying prompt evaluation portions; and retrieving a sub-model associated with each of the identified prompt evaluation portions.

19. The method of claim 18 , further comprising determining a training level of the identified model.

20. The method of claim 19 , wherein determining a training level of the identified model comprises: retrieving sub-model data for each of the retrieved sub-models; determining a sub-model confidence level for each of the retrieved sub-models; and determining an aggregate confidence level based on a combination of the determined sub-model confidence levels.

Assignees

• Pearson Education Inc

Inventors

• Foltz Peter
• Rosenstein Mark
• Baikadi Alok
• Becker Lee
• Hopkins Stephen
• Budden Jill
• Oros Luis M
• Habermehl Kyle
• Hellman Scott
• Murray William
• Gorman Andrew

Classifications

• G09B7/02 Primary

  of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title