Cluster analysis of participant responses for test generation or teaching

Patent metadata

Field	Value
Publication number	US-10388177-B2
Application number	US-201313871627-A
Country	US
Kind code	B2
Filing date	Apr 26, 2013
Priority date	Apr 27, 2012
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

Textual responses to open-ended (i.e., free-response) items provided by participants (e.g., by means of mobile wireless devices) are automatically classified, enabling an instructor to assess the responses in a convenient, organized fashion and adjust instruction accordingly.

First claim

What is claimed is:

1. A method of determining a plurality of textual responses to generate a future test option to correspond with a response to a free response item, the method comprising the steps of:
determining, at a server computer, one or more participant devices and one or more instructor devices, wherein the one or more participant devices and the one or more instructor devices are geographically dispersed and communicate via a network, wherein each of the one or more participant devices is operated by a respective participant and each respective participant is associated with a respective characteristic;
receiving, at the server computer, the free response item from one of the one or more instructor devices;
receiving, at the server computer, the plurality of textual responses from the one or more participant devices;
removing punctuation from the plurality of textual responses;
translating into a numerical dataset the plurality of textual responses resulting from the removing punctuation step;
applying, at the server computer, a clustering algorithm to the numerical dataset to generate one or more clusters, wherein the clustering algorithm is based at least in part on the respective characteristic;
generating the one or more clusters from the clustering algorithm;
determining a single cluster as a weighted average of the one or more clusters, wherein the determined single cluster is used to generate the future test option based at least in part on the plurality of textual responses and the determined single cluster; and
wirelessly providing the future test option by the server computer to the one or more instructor devices.

2. The method of claim 1 , further comprising:
after generating the one or more clusters from the clustering algorithm, determining a distance metric associated with the one or more clusters to identify similarities between the one or more clusters, wherein the one or more clusters include at least one incorrect answer to the free response item.

3. The method of claim 2 , wherein the distance metric analyzes a number of the plurality of textual responses not found in the same cluster.

4. The method of claim 2 , wherein the distance metric is associated with a number between zero and log(a number of the one or more clusters).

5. The method of claim 2 , further comprising:
ranking the one or more clusters based in part on the distance metric; and
transmitting the ranked clusters to the one or more instructor devices.

6. The method of claim 1 , wherein the plurality of textual responses are translated to the numerical dataset by:
stemming one or more words in the plurality of textual responses;
computing 1, 2, 3, 4, and 5-grams;
removing stemmed n-grams that appear in fewer than 1-percent or more than 99-percent of the plurality of textual responses; and
generating the stemmed n-grams as a vector.

7. The method of claim 1 , further comprising:
removing a second cluster from the one or more clusters, wherein the distance metric associated with the second cluster exceeds a threshold.

8. The method of claim 1 , further comprising:
storing, by the server computer, participant device information in a participant database.

9. The method of claim 1 , wherein four clusters are generated by the clustering algorithm to the numerical dataset.

10. One or more non-transitory machine-readable medium having machine-executable instructions configured to perform the machine-implementable method for:
determining, at a server computer, one or more participant devices and one or more instructor devices, wherein the one or more participant devices and the one or more instructor devices are geographically dispersed and communicate via a network, wherein each of the one or more participant devices is operated by a respective participant and each respective participant is associated with a respective characteristic;
receiving, at the server computer, a free response item from one of the one or more instructor devices;
receiving, at the server computer, the plurality of textual responses from the one or more participant devices;
removing punctuation from the plurality of textual responses;
translating into a numerical dataset the plurality of textual responses resulting from the removing punctuation step;
applying, at the server computer, a clustering algorithm to the numerical dataset to generate one or more clusters, wherein the clustering algorithm is based at least in part on the respective characteristic;
generating the one or more clusters from the clustering algorithm;
determining a single cluster as a weighted average of the one or more clusters, wherein the determined single cluster is used to generate the future test option based at least in part on the plurality of textual responses and the determined single cluster; and
wirelessly providing the future test option by the server computer to the one or more instructor devices.

11. The one or more non-transitory machine-readable medium of claim 10 , wherein the method further comprises:
after generating the one or more clusters from the clustering algorithm, determining a distance metric associated with the one or more clusters to identify similarities between the one or more clusters, wherein the one or more clusters include at least one incorrect answer to the free response item.

12. The one or more non-transitory machine-readable medium of claim 11 , wherein the distance metric analyzes a number of the plurality of textual responses not found in the same cluster.

13. The one or more non-transitory machine-readable medium of claim 11 , wherein the distance metric is associated with a number between zero and log(a number of the one or more clusters).

14. The one or more non-transitory machine-readable medium of claim 11 , wherein the method further comprises:
ranking the one or more clusters based in part on the distance metric; and
transmitting the ranked clusters to the one or more instructor devices.

15. The one or more non-transitory machine-readable medium of claim 10 , wherein the plurality of textual responses are translated to the numerical dataset by:
stemming one or more words in the plurality of textual responses;
computing 1, 2, 3, 4, and 5-grams;
removing stemmed n-grams that appear in fewer than 1-percent or more than 99-percent of the plurality of textual responses; and
generating the stemmed n-grams as a vector.

16. A computer system for determining a plurality of textual responses to generate a future test option to correspond with a response to a free response item, the system comprising:
a processor; and
one or memories coupled with said one or more processors, wherein the one or more processors and one or more memories are configured to;
determine, at a server computer, one or more participant devices and one or more instructor devices, wherein the one or more participant devices and the one or more instructor devices are geographically dispersed and communicate via a network, wherein each of the one or more participant devices is operated by a respective participant and each respective participant is associated with a respective characteristic;
receive, at the server computer, the free response item from one of the one or more instructor devices; receive, at the server computer, the plurality of textual responses from the one or more participant devices;
remove punctuation from the plurality of textual responses;
translate into a numerical dataset the plurality of textual responses resulting from the remove punctuation step;
apply, at the server computer, a

Assignees

• Harvard College

Inventors

• King Gary
• Lukoff Brian
• Mazur Eric

Classifications

• G09B7/00 Primary

  Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) · CPC title