Real time development of auto scoring essay models for custom created prompts

Patent metadata

Field	Value
Publication number	US-11449762-B2
Application number	US-201916544745-A
Country	US
Kind code	B2
Filing date	Aug 19, 2019
Priority date	Feb 20, 2018
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

First claim

What is claimed is:

1. A system for dynamic composite machine learning model selection, the system comprising:
a database coupled to a network of computing devices and storing:
a plurality of prompts;
a prompt metadata comprising a type or classification of, and a subject matter for, each of the plurality of prompts;
an instructor metadata associated with, and identifying an instructor that created, each of the plurality of prompts;
scoring data associated with each of the plurality of prompts; and
a plurality of machine learning models for automated scoring of received user responses; and

a server comprising a computing device coupled to the network and comprising at least one processor executing instructions within a memory which, when executed, cause the system to:
receive a first plurality of responses received from a plurality of users in response to providing of a prompt in the plurality of prompts;
extract, from each of the plurality of responses, at least one feature;
identify rules relevant to the received first plurality of responses based on the at least one feature, and sharing a common prompt metadata and instructor metadata;
select a model in the plurality of machine learning models for scoring of the first plurality of responses according to a link to the common prompt metadata and instructor metadata within the rules identified as relevant to the received plurality of responses;
input, into the model, the at least one feature;
auto-generate scores for the first plurality of responses as output from the model;
receive, as a user input on a client device from the instructor in the instructor metadata, evaluation of the auto-generated scores;
identify a set of machine learning models, of varying accuracy or performance complexity, via the rules relevant to the received first plurality of responses associated in the database with the common prompt metadata and instructor metadata;
evaluate the efficacy of each of the set of machine learning models based on the received evaluation of the auto-generated scores;
identify the best performing machine learning model as having the smallest error value based on a least number of mistakes; and
update the rules relevant to the received prompt to select the identified best performing machine learning models in response to receipt of a next plurality of responses.

2. The system of claim 1 , wherein the at least one processor is further configured to update the training of the set of machine learning models based on the received evaluation of the auto-generated scores.

3. The system of claim 2 , wherein the at least one server is further configured to identify response characteristics in the first plurality of responses.

4. The system of claim 3 , wherein the at least one server is further configured to identify prompt characteristics of the prompt associated with the first plurality of responses.

5. The system of claim 4 , wherein the model is selected for scoring based on application of the identified rules to the characteristics of the prompt and the response characteristics of the first plurality of responses.

6. The system of claim 5 , wherein the set of machine learning models includes the model.

7. The system of claim 6 , wherein the at least one server is further configured to receive metadata of an evaluator associated with the prompt.

8. The system of claim 7 , wherein the plurality of machine learning models are linked with the evaluator.

9. The system of claim 7 , wherein the plurality of machine learning models are identified based on the metadata of the evaluator.

10. The system of claim 9 , wherein the at least one server is further configured to:
subsequent to updating of the rules, receive a second plurality of responses in response to the prompt;
identify the rules as relevant to the received second plurality of responses;
select a second one of the plurality of machine learning models for scoring of the second plurality of responses according to the rules;
auto-generate scores for the second plurality of responses with the model.

11. A method for dynamic composite machine learning model selection, the method comprising:
storing, by a server comprising at least one computing device coupled to a network and comprising at least one processor executing instructions within a memory, within a database coupled to the network:
a plurality of prompts;
a prompt metadata comprising a type or classification of, and a subject matter for, each of the plurality of prompts;
an instructor metadata associated with, and identifying an instructor that created, each of the plurality of prompts;
scoring data associated with each of the plurality of prompts; and
a plurality of machine learning models for automated scoring of received user responses;

receiving, by the server, a first plurality of responses received from a plurality of users in response to providing of a prompt in a plurality of prompts;
extracting, by the server from each of the plurality of responses, at least one feature;
identifying, by the server, rules relevant to the received first plurality of responses based on the at least one feature, and sharing a common prompt metadata and instructor metadata;
selecting, by the server, a model in the plurality of machine learning models for scoring of the first plurality of responses according to a link to the common prompt metadata and instructor metadata within the rules identified as relevant to the received plurality of responses;
inputting, by the server into the model, the at least one feature;
auto-generating, by the server, scores for the first plurality of responses as output from the model;
receiving, by the server as a user input on a client device from the instructor in the instructor metadata, evaluation of the auto-generated scores;
identifying, by the server, a set of machine learning models, of varying accuracy or performance complexity, via the rules relevant to the received first plurality of responses associated in the database with the common prompt metadata and instructor metadata;
evaluating, by the server, the efficacy of each of the set of machine learning models based on the received evaluation of the auto-generated scores;
identifying, by the server, the best performing machine learning model as having the smallest error value based on a least number of mistakes; and
updating the rules relevant to the received prompt to select the identified best performing machine learning models in response to receipt of a next plurality of responses.

12. The method of claim 11 , further comprising updating the training of the set of machine learning models based on the received evaluation of the auto-generated scores.

13. The method of claim 12 , further comprising identifying response characteristics in the first plurality of responses.

14. The method of claim 13 , further comprising identifying prompt characteristics of the prompt associated with the first plurality of responses.

15. The method of claim 14 , wherein the model is selected for scoring based on application of the identified rules to the characteristics of the prompt and the response characteristics of the first plurality of responses.

16. The method of claim 15 , wherein the set of machine learning models includes the model.

17. The method of claim 16 , further comprising receiving metadata of an evaluator associated with the prompt.

18. The method of claim 17 , wherein the plurality of machine learning models are linked with the evaluator.

19. The m

Assignees

• Pearson Education Inc

Inventors

• Rosenstein Mark
• Habermehl Kyle
• Hellman Scott
• Baikadi Alok
• Foltz Peter
• Becker Lee
• Oros Luis M
• Budden Jill
• Derr Marcia

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• H04L67/01

  Protocols · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/046

  Forward inferencing; Production systems · CPC title

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title