Systems and methods for interface-based automated custom authored prompt evaluation

Patent metadata

Field	Value
Publication number	US-11817014-B2
Application number	US-201916281033-A
Country	US
Kind code	B2
Filing date	Feb 20, 2019
Priority date	Feb 20, 2018
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

First claim

What is claimed is:

1. A system for training a model, the system comprising:
a memory comprising:
a content library database comprising a plurality of prompts, and
a model database comprising at least one prompt evaluation model trained to evaluate prompts; and

at least one processor configured to:
receive a prompt via a prompt creation window within a user interface;
identify a creator of the prompt;
provide, based at least in part on the at least one prompt evaluation model, feedback via the prompt creation window, wherein the feedback includes a recommended modification to the prompt;
parse the prompt to identify a plurality of prompt evaluation portions;
identify, based on the creator of the prompt, pre-existing data relevant to at least one prompt evaluation portion of the plurality of prompt evaluation portions;
train a response evaluation model for evaluating responses to the prompt at least in part based on the pre-existing data;
determine training information corresponding to the response evaluation model, wherein the training information relates to progress towards training of the response evaluation model; and
provide the training information via a training level indicator in the user interface, wherein the training level indicator indicates a training level of the response evaluation model.

2. The system of claim 1 , wherein the at least one processor is configured to:
gather new training data;
update training of the response evaluation model with the new training data; and
evaluate the training of the response evaluation model.

3. The system of claim 2 , wherein the at least one processor is configured to:
control an evaluation interface to generate a launch window upon determining that the response evaluation model is sufficiently trained.

4. The system of claim 3 ,
wherein the launch window is configured to provide at least one of:
an indicator of sufficiency of training of the response evaluation model, and
a feature manipulable to initiate auto evaluation with the response evaluation model of responses to the prompt.

5. The system of claim 4 , wherein the at least one processor is configured to:
provide the prompt to a plurality of users;
receive a response from each of the plurality of users;
determine that the response evaluation model is sufficiently trained; and
auto-evaluate the responses with the response evaluation model.

6. The system of claim 5 , wherein the at least one processor is configured to:
control the evaluation interface to display at least one auto-evaluated response of the auto-evaluated responses, and
receive a modification to the at least one auto-evaluated response via an input feature of the evaluation interface.

7. The system of claim 6 , wherein the at least one processor is configured to:
control the evaluation interface to modify an appearance of the input feature in response to the received modification.

8. The system of claim 5 , wherein the at least one processor is configured to:
control generation of an output data interface, wherein the output data interface comprises a scoring summary window identifying a scoring status of the received responses, and
wherein the output data interface comprises a distribution window comprising a graphical display of a distribution of the auto-evaluations of the responses.

9. The system of claim 1 , wherein the pre-existing data includes a pre-existing model trained to evaluate a set of responses associated with a pre-existing prompt, wherein the pre-existing prompt is relevant to the received prompt.

10. The system of claim 1 , wherein the pre-exiting data includes (a) a pre-existing model trained to evaluate responses to another prompt and (b) pre-existing response data generated from the responses to the another prompt.

11. The system of claim 10 , wherein the at least one processor is configured to identify the pre-existing response data by identifying response data corresponding to at least one prompt evaluation portion included in the plurality of prompt evaluation portions via a similarity score.

12. The system of claim 1 , wherein the at least one processor is configured to identify the pre-existing data by performing a comparison of at least one prompt evaluation portion included in the plurality of prompt evaluation portions to at least one evaluation portion of a pre-existing prompt associated with the creator of the prompt.

13. The system of claim 1 , wherein the pre-existing data includes user data specific to the creator of the prompt, the user data including information relating to at least one of a set of previously-trained models for the creator or a set of previously generated training data for the creator.

14. A method for training a model, the method comprising:
receiving a prompt via a prompt creation window within a user interface;
identifying a creator of the prompt;
providing, based at least in part on a prompt evaluation model, feedback via the prompt creation window, wherein the feedback includes a recommended modification to the prompt;
parsing the prompt to identify a plurality of prompt evaluation portions;
identifying, based on the creator of the prompt, pre-existing data relevant to at least one evaluation portion of the plurality of prompt evaluation portions;
training a response evaluation model for evaluating responses to the prompt at least in part based on the pre-existing data;
determining training information corresponding to the response evaluation model, wherein the training information relates to progress towards training of the response evaluation model; and
providing the training information via a training level indicator in the user interface, wherein the training level indicator indicates a training level of the response evaluation model.

15. The method of claim 14 , further comprising:
gathering new training data;
updating training of the response evaluation model with the new training data; and
evaluating the training of the response evaluation model.

16. The method of claim 14 ,
determining that the response evaluation model is sufficiently trained; and
controlling an evaluation interface to generate a launch window, wherein the launch window is configured to provide at least one of:
an indicator of sufficiency of training of the response evaluation model, and
a feature manipulable to initiate auto evaluation with the response evaluation model of responses to the prompt.

17. The method of claim 16 , further comprising:
providing the prompt to a plurality of users;
receiving a response from each of the plurality of users;
determining that the response evaluation model is sufficiently trained; and
auto-evaluating the responses with the response evaluation model.

18. The method of claim 17 , further comprising:
controlling the evaluation interface to display at least one auto-evaluated response of the auto-evaluated responses.

19. The method of claim 14 , further comprising:
training the prompt evaluation model based at least in part on the pre-existing data, the prompt evaluation model trained to evaluate the prompt.

20. A method for training an evaluation model, the method comprising:
identifying a prompt, the prompt configured for being evaluated via a prompt evaluation model, and wherein the prompt is configured to elicit at least one response, the at least one response configured to be evaluated via a response evaluation model;
determining, based at least in part on a creator of the prompt, pre-

Assignees

• Pearson Education Inc

Inventors

• Becker Lee
• Murray William
• Foltz Peter
• Rosenstein Mark
• Baikadi Alok
• Hellman Scott
• Habermehl Kyle
• Budden Jill
• Hopkins Stephen
• Gorman Andrew

Classifications

• G09B7/02 Primary

  of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student · CPC title

• G06F3/0481

  based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F16/40

  of multimedia data, e.g. slideshows comprising image and additional audio data (retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70) · CPC title

• G06F16/904

  Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957) · CPC title