Systems and methods for evaluating multilingual text sequences

Patent metadata

Field	Value
Publication number	US-9471667-B2
Application number	US-201313848837-A
Country	US
Kind code	B2
Filing date	Mar 22, 2013
Priority date	Mar 26, 2012
Publication date	Oct 18, 2016
Grant date	Oct 18, 2016

Abstract

Systems and methods are provided for scoring a response to a character-by-character highlighting task. A similarity value for the response is calculated by comparing the response to one or more correct responses to the task to determine the similarity or dissimilarity of the response to the one or more correct responses to the task. A threshold similarity value is calculated for the task, where the threshold similarity value is indicative of an amount of similarity or dissimilarity to the one or more correct responses required for the response to be scored at a certain level. The similarity value for the response is compared to the threshold similarity value. A score is assigned at, above, or below the certain level based on the comparison.

First claim

It is claimed:

1. A computer-implemented method of scoring a response to a character-by-character highlighting task, comprising:
comparing, using a processing system, the response to the character-by-character highlighting task to a correct response to the task, the comparison generating a dissimilarity value that indicates an extent of a dissimilarity of the response and the correct response;
selecting, using the processing system, a threshold dissimilarity value for the task, the threshold dissimilarity value being based on (i) a first set of responses to the task, each response of the first set differing from the correct response and being scored by a human grader as being correct or incorrect, and (ii) a second set of responses to the task, each response of the second set being scored by a human grader as being correct, the threshold dissimilarity value being further based on
filtering the first set to remove null responses from the first set;
comparing each response of the first set to each response of the second set, each comparison generating a dissimilarity value that is associated with the response of the first set, and populating a data structure comprising M×N dissimilarity values, where M is a number of non-null responses in the first set and N is a number of responses in the second set, M and N each being an integer that is greater than 1,
for each response of the first set, determining a minimum dissimilarity value associated with the response from the data structure, and
determining the threshold dissimilarity value further based on (i) the minimum dissimilarity values associated with the responses of the first set, and (ii) the scoring of the responses of the first set as being correct or incorrect by the human grader;

comparing, using the processing system, the dissimilarity value for the response to the threshold dissimilarity value; and
scoring the response based on the comparing of the dissimilarity value for the response to the threshold dissimilarity value, wherein the response is scored as being correct if the dissimilarity value is less than or equal to the threshold dissimilarity value, and wherein the response is scored as being incorrect if the dissimilarity value is greater than the threshold dissimilarity value.

2. The computer-implemented method of claim 1 , wherein the comparing of the response to the correct response includes
comparing a sequence of characters in the response to a sequence of characters in the correct response.

3. The computer-implemented method of claim 1 , wherein the dissimilarity value is based on
a difference between a number of characters in the response and a number of characters in the correct response.

4. The computer-implemented method of claim 3 , wherein the dissimilarity value is based on
a normalized difference between the number of characters in the response and the number of characters in the correct response, the normalized difference being calculated based on

Normalized
⁢

⁢
Difference

=

abs
⁡

(

[

F
⁢

⁢
R

]

-

[

T
⁢

⁢
R

]

)

[

F
⁢

⁢
R

]

+

[

T
⁢

⁢
R

]

,

where [FR] is the number of characters in the response, [TR] is the number of characters in the correct response, and abs([FR]−[TR]) is an absolute value of the difference between the number of characters in the response and the number of characters in the correct response.

5. The computer-implemented method of claim 1 , wherein the dissimilarity value is based on
a longest common subsequence, the longest common subsequence being a longest set of characters that appears in sequential order in both the response and the correct response.

6. The computer-implemented method of claim 5 ,
wherein the longest common subsequence is calculated using a recursive algorithm.

7. The computer-implemented method of claim 1 , wherein the dissimilarity value is determined based on
([ T 1 ]−[Similarity( T 1 ,T 2 )])+([ T 2 ]−[Similarity( T 1 ,T 2 )]),

where T 1 and T 2 are the response and the correct response to the task, respectively, [T 1 ] and [T 2 ] are a number of graphemes in the response and the correct response to the task, respectively, and [Similarity(T 1 , T 2 )] is a number of graphemes in a similarity value for the response, the similarity value being based on a longest set of characters that appears in sequential order in both the response and the correct response.

8. The computer-implemented method of claim 1 , wherein the dissimilarity value is based on
a longest common substring, the longest common substring being a longest set of characters that appears in sequential order in both the response and the correct response, wherein the longest set of characters are consecutive characters within the response and the correct response.

9. The computer-implemented method of claim 1 ,
wherein the scoring of the response modifies a score previously assigned to the response.

10. The computer-implemented method of claim 9 , wherein the scoring of the response modifies the score previously assigned to the response
if the response has a high similarity to the correct response or a low dissimilarity to the correct response.

11. The computer-implemented method of claim 1 , wherein the character-by-character highlighting task includes a stimulus that is given to a test-taker and directions to select portions of the stimulus to respond to a prompt.

12. The computer-implemented method of claim 1 , wherein the threshold dissimilarity value is based on:
for each response i of the first set:
determining if the human grader scored the response i as being correct or incorrect,
based on a det

Assignees

• Educational Testing Service

Inventors

• Yamamoto Kentaro
• Sukkarieh Jana
• Von Davier Matthias

Classifications

• G09B7/02 Primary

  of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student · CPC title

• G09B19/06

  Foreign languages (with audible presentation of material to be studied G09B5/04) · CPC title

• G06F16/3334 Primary

  Selection or weighting of terms from queries, including natural language queries · CPC title

• G06F17/30663 Primary

  Physics · mapped topic