System and method for automated scoring of a summary-writing task

Patent metadata

Field	Value
Publication number	US-9576249-B2
Application number	US-201414218309-A
Country	US
Kind code	B2
Filing date	Mar 18, 2014
Priority date	Mar 18, 2013
Publication date	Feb 21, 2017
Grant date	Feb 21, 2017

Abstract

In accordance with the teachings described herein, systems and methods are provided for measuring a user's comprehension of subject matter of a text. A summary generated by the user is received, where the summary summarizes the text. The summary is processed to determine a first numerical measure indicative of a similarity between the summary and a reference summary. The summary is processed to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text. The summary is processed to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text. A numerical model is applied to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text.

First claim

It is claimed:

1. A computer-implemented method of measuring a user's comprehension of subject matter of a text, the method comprising:
receiving a summary generated by the user, the summary being a constructed response that summarizes a text;
parsing the summary with a processing system to identify a number of sentences contained in the summary and to identify in the summary a plurality of multi-word sequences;
processing the summary and a reference summary with the processing system to determine a first numerical measure indicative of a similarity between the summary and a reference summary, the reference summary having been designated as representative of the subject matter of the text;
processing the summary with the processing system to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text;
processing the summary and the text with the processing system to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text; and
applying a numerical model to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text, the numerical model including
a first variable and an associated first weighting factor, the first variable receiving a value of the first numerical measure,
a second variable and an associated second weighting factor, the first variable receiving a value of the second numerical measure, and
a third variable and an associated third weighting factor, the third variable receiving a value of the third numerical measure.

2. The computer-implemented method of claim 1 , wherein the determining of the third numerical measure includes:
determining a first metric for the summary, the first metric being a ratio between a first value and a second value, wherein the first value is a sum of lengths of all three-word or longer phrases from the text that are included in the summary, and wherein the second value is a length of the summary;
determining a second metric for the summary, the second metric being a ratio between the first value and a third value, wherein the third value is a length of the text; and
determining a third metric for the summary, the third metric being a length of a longest word sequence from the text that is included in the summary.

3. The computer-implemented method of claim 1 comprising:
processing the summary with the processing system to determine a fourth numerical measure indicative of a length of the summary; and
applying the numerical model to the fourth numerical measure to determine the score for the summary, the numerical model including a fourth variable and an associated fourth weighting factor, the fourth variable receiving a value of the fourth numerical measure.

4. The computer-implemented method of claim 1 comprising:
processing the summary with the processing system to determine a fourth numerical measure indicative of a number of discourse markers included in the summary; and
applying the numerical model to the fourth numerical measure to determine the score for the summary, the numerical model including a fourth variable and an associated fourth weighting factor, the fourth variable receiving a value of the fourth numerical measure.

5. The computer-implemented method of claim 1 , wherein the determining of the second numerical measure includes determining a number of sentences of the text from which the single sentence of the summary reproduces two-word or longer sequences.

6. A system for measuring a user's comprehension of subject matter of a text, the system comprising:
a processing system; and
computer-readable memory in communication with the processing system encoded with instructions for commanding the processing system to execute steps comprising:
receiving a summary generated by the user, the summary being a constructed response that summarizes a text;
parsing the summary with the processing system to identify a number of sentences contained in the summary and to identify in the summary a plurality of multi-word sequences;
processing the summary and a reference summary with the processing system to determine a first numerical measure indicative of a similarity between the summary and a reference summary, the reference summary having been designated as representative of the subject matter of the text;
processing the summary with the processing system to determine a second numerical measure indicative of a degree to which a single sentence of the summary summarizes an entirety of the text;
processing the summary and the text with the processing system to determine a third numerical measure indicative of a degree of copying in the summary of multi-word sequences present in the text; and
applying a numerical model to the first numerical measure, the second numerical measure and the third numerical measure to determine a score for the summary indicative of the user's comprehension of the subject matter of the text, the numerical model including
a first variable and an associated first weighting factor, the first variable receiving a value of the first numerical measure,
a second variable and an associated second weighting factor, the first variable receiving a value of the second numerical measure, and
a third variable and an associated third weighting factor, the third variable receiving a value of the third numerical measure.

7. The system of claim 6 , wherein the determining of the third numerical measure includes:
determining a first metric for the summary, the first metric being a ratio between a first value and a second value, wherein the first value is a sum of lengths of all three-word or longer phrases from the text that are included in the summary, and wherein the second value is a length of the summary;
determining a second metric for the summary, the second metric being a ratio between the first value and a third value, wherein the third value is a length of the text; and
determining a third metric for the summary, the third metric being a length of a longest word sequence from the text that is included in the summary.

8. The system of claim 6 , wherein the instructions command the processing system to execute the steps comprising:
processing the summary with the processing system to determine a fourth numerical measure indicative of a length of the summary; and
applying the numerical model to the fourth numerical measure to determine the score for the summary, the numerical model including a fourth variable and an associated fourth weighting factor, the fourth variable receiving a value of the fourth numerical measure.

9. The system of claim 6 , wherein the instructions command the processing system to execute the steps comprising:
processing the summary with the processing system to determine a fourth numerical measure indicative of a number of discourse markers included in the summary; and
applying the numerical model to the fourth numerical measure to determine the score for the summary, the numerical model including a fourth variable and an associated fourth weighting factor, the fourth variable receiving a value of the fourth numerical measure.

10. The system of claim 6 , wherein the determining of the second numerical measure includes determining a number of sentences of the text from which the single sentence of the summary reproduces two-word or longer sequences.

11. A non-transitory computer-readable storage medium for measuring a user's comprehension of subject matter of a text, the computer-readable st

Assignees

• Educational Testing Service

Inventors

• Madnani Nitin
• Burstein Jill

Classifications

• G09B7/00

  Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) · CPC title

• G06Q10/10

  Office automation; Time management · CPC title

• G06F40/258

  Heading extraction; Automatic titling; Numbering · CPC title

• G09B7/02 Primary

  of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student · CPC title

• G06N99/005 Primary

  Physics · mapped topic