Systems and methods for generating automated evaluation models

Patent metadata

Field	Value
Publication number	US-9443193-B2
Application number	US-201414257380-A
Country	US
Kind code	B2
Filing date	Apr 21, 2014
Priority date	Apr 19, 2013
Publication date	Sep 13, 2016
Grant date	Sep 13, 2016

Abstract

Systems and methods are described for generating a scoring model for responses. A computer-implemented method of calibrating a scoring model using a processing system for scoring examinee responses includes accessing a plurality of training responses for training the scoring model. The plurality of training responses are analyzed to derive values of multiple features (variables) of the training responses. The scoring model is trained based on the values of the multiple features of the training responses and one or more external measures of proficiency for each individual associated with a training response utilized in the training. The one or more external measures are not derived from the training responses. Based on the training, a weight for each of the multiple features is determined. The scoring model is calibrated to include the weights for at least some of the features for scoring examinee responses.

First claim

What is claimed is:

1. A computer-implemented method of calibrating a scoring model for scoring examinee responses, comprising:
accessing a plurality of training responses with a processing system for training a scoring model for scoring examinee responses, the training responses and examinee responses being constructed responses;
analyzing the plurality of training responses with the processing system to derive values of multiple features of the training responses, the multiple features corresponding to variables of the scoring model;
training the scoring model with the processing system based on the values of the multiple features of the training responses and one or more external measures of proficiency for each individual associated with a training response utilized in the training, the one or more external measures not being derived from the training responses;
determining, based on said training, a weight for each of the multiple features; and
calibrating the scoring model to include the weights for at least some of the features such that the scoring model is configured to generate scores for examinee responses.

2. The method of claim 1 , further comprising:
using the scoring model to automatically generate a score for an examinee response constructed by an examinee; and
validating the score using the one or more external measures of proficiency for the examinee.

3. The method of claim 1 , wherein the examinee responses are written responses or spoken responses.

4. The method of claim 1 , wherein the multiple features are treated as independent variables in the calibrated scoring model and the one or more external measures are treated as temporary dependent variables for training the scoring model.

5. The method of claim 4 , wherein the training further includes treatment of predetermined scores for the training responses as temporary independent variables that are not included in the calibrated scoring model.

6. The method of claim 4 , wherein the training further includes treatment of predetermined scores for the training responses as additional temporary dependent variables in combination with the one or more external measures.

7. The method of claim 1 , wherein at least one of the one or more external measures of proficiency for an individual associated with a training response is selected from the group consisting of:
a score associated with a different portion of a same test from which the training response is derived;
a class grade received by the individual; and
a competency measure of the individual.

8. The method of claim 1 , wherein the training comprises canonical correlation analysis to analyze relationships between the multiple features and the one or more external measures.

9. A non-transitory computer-readable medium encoded with instructions for causing a processing system to execute steps for calibrating a scoring model for scoring examinee responses, comprising:
accessing a plurality of training responses with a processing system for training a scoring model for scoring examinee responses, the training responses and examinee responses being constructed responses;
analyzing the plurality of training responses with the processing system to derive values of multiple features of the training responses, the multiple features corresponding to variables of the scoring model;
training the scoring model with the processing system based on the values of the multiple features of the training responses and one or more external measures of proficiency for each individual associated with a training response utilized in the training, the one or more external measures not being derived from the training responses;
determining, based on said training, a weight for each of the multiple features; and
calibrating the scoring model to include the weights for at least some of the features such that the scoring model is configured to generate scores for examinee responses.

10. The non-transitory computer-readable medium of claim 9 , further comprising instructions for causing the processing system to execute steps, including:
using the scoring model to automatically generate a score for an examinee response constructed by an examinee; and
validating the score using the one or more external measures of proficiency for the examinee.

11. The non-transitory computer-readable medium of claim 9 , wherein the examinee responses are written responses or spoken responses.

12. The non-transitory computer-readable medium of claim 9 , wherein the multiple features are treated as independent variables in the calibrated scoring model and the one or more external measures are treated as temporary dependent variables for training the scoring model.

13. The non-transitory computer-readable medium of claim 12 , wherein the training further includes treatment of predetermined scores for the training responses as temporary independent variables that are not included in the calibrated scoring model.

14. The non-transitory computer-readable medium of claim 12 , wherein the training further includes treatment of predetermined scores for the training responses as additional temporary dependent variables in combination with the one or more external measures.

15. The non-transitory computer-readable medium of claim 9 , wherein at least one of the one or more external measures of proficiency for an individual associated with a training response is selected from the group consisting of:
a score associated with a different portion of a same test from which the training response is derived;
a class grade received by the individual; and
a competency measure of the individual.

16. The non-transitory computer-readable medium of claim 9 , wherein the training comprises canonical correlation analysis to analyze relationships between the multiple features and the one or more external measures.

17. A system for calibrating a scoring model for scoring examinee responses, comprising:
a processing system; and
a memory coupled to the processing system, wherein the processing system is configured to execute steps, comprising:
accessing a plurality of training responses with the processing system for training a scoring model for scoring examinee responses, the training responses and examinee responses being constructed responses;
analyzing the plurality of training responses with the processing system to derive values of multiple features of the training responses, the multiple features corresponding to variables of the scoring model;
training the scoring model with the processing system based on the values of the multiple features of the training responses and one or more external measures of proficiency for each individual associated with a training response utilized in the training, the one or more external measures not being derived from the training responses;
determining, based on said training, a weight for each of the multiple features; and
calibrating the scoring model to include the weights for at least some of the features such that the scoring model is configured to generate scores for examinee responses.

18. The system of claim 17 , wherein the processing system is further configured to execute steps, including:
using the scoring model to automatically generate a score for an examinee response constructed by an examinee; and
validating the score using the one or more external measures of proficiency for the examinee.

19. The system of claim 17 , wherein the examinee responses are written responses or spoken responses.

Assignees

• Educational Testing Service

Inventors

• Haberman Shelby J
• Zhang Mo
• Bridgeman Brent

Classifications

• G09B7/00

  Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F40/253

  Grammatical analysis; Style critique · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F17/274

  Physics · mapped topic