Cognitive essay annotation

Patent metadata

Field	Value
Publication number	US-10957212-B2
Application number	US-201815945500-A
Country	US
Kind code	B2
Filing date	Apr 4, 2018
Priority date	Apr 4, 2018
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

Methods, computer program products, and systems are presented. The methods include, for instance: obtaining sample essays, sample annotations corresponding to the sample essays, and a subject content for building a subject domain comprehension model and an essay annotation model, by use of one or more neural network. The nodes of the subject domain comprehension model and the essay annotation model are interconnected based on respective relevancies for automatically annotating student works according to a standard of review corresponding to submitting students.

First claim

What is claimed is:

1. A computer implemented method for providing automated annotation for one or more student works, comprising:
obtaining, by one or more processors running one or more neural networks, training data for the one or more neural networks including a plurality of sample essays, a plurality of sample annotations respectively corresponding to the plurality of sample essays, and a subject content including respective knowledgebases per topic on which the plurality of sample essays had been written;
labeling, by the one or more processors running the one or more neural networks, by use of natural language classification tools operatively coupled to the one or more processors, the plurality of sample essays for gradable components and the plurality of sample annotations for each of annotation types;
building, by the one or more processors running the one or more neural networks, a subject domain comprehension model based on the subject content, by use of the one or more neural networks;
forming, by the one or more processors running the one or more neural networks, an essay annotation model, based on the gradable components of the plurality of sample essays and the annotation types of the plurality of sample annotations;
linking, by the one or more processors running the one or more neural networks, nodes of the subject domain comprehension model respectively representing preconfigured essay elements in the plurality of sample essays that show respective writers of the plurality of sample essays understand respective topics of the plurality of sample essays and nodes of the essay annotation model respectively representing preconfigured annotation elements in the plurality of sample annotations that evaluate the plurality of sample essays, based on respective relevancies between the nodes of the subject domain comprehension model and the nodes of the essay annotation model to thereby automatically provide one of the preconfigured annotation elements, represented by a first node of the subject domain comprehension model, for one of the preconfigured essay elements, and a first node amongst the nodes of the essay annotation model based on an edge between the first node of the subject domain comprehension model and the first node of the essay annotation model that is made as a result of the linking; and
producing, by the one or more processors, interconnected models resulting from the linking based on determining that the one or more neural networks has been trained for automatically annotating the one or more student works, including an essay, which had been submitted for evaluation to a computerized educational tool utilizing the interconnected models according to a standard of review corresponding to a preconfigured group of students who had written the one or more student works.

2. The computer implemented method of claim 1 , wherein the annotation types include corrections in texts, corrections in editorial marks, suggestions for activities, references, techniques, and combinations thereof, compliments, and grade letters.

3. The computer implemented method of claim 1 , wherein the gradable components for the sample essays include respective structures for each of the sample essays, each paragraph, each sentence, relevant facts, arguments presented in each of teh sample essays, assessments on the arguments on a logical flow, clarity of the arguments, and evidentiary supports for the arguments, evidence against the arguments, persuasiveness in expression for the arguments, clarity in expression for the arguments, general readability, originality, mechanics including grammar, spelling, and word choices.

4. The computer implemented method of claim 1 , wherein the subject domain comprehension model corresponds to respective student cohorts, as being classified based on respective academic development levels as represented by school-year grades of students, academic performance levels, and combinations thereof.

5. The computer implemented method of claim 1 , further comprising:
obtaining, from the student, the essay amongst the one or more student works that had been submitted for a review, wherein the student is a member in the preconfigured group of students corresponding to the standard of review;
annotating the essay by use of the interconnected models from the producing according to the standard of review; and
presenting the essay with annotations resulting from the annotating to the student and to an instructor, designated to receive the essay.

6. The computer implemented method of claim 5 , further comprising
comparing the annotations in the essay with a preset performance threshold for a quality of the annotations; and
adjusting automatically the interconnected models from the producing, based on ascertaining that the annotations do not satisfy the preset performance threshold.

7. The computer implemented method of claim 1 , further comprising:
obtaining a debate script for review as being generated from a live debate by use of a speech-to-text tool;
annotating the debate script by use of the interconnected models from the producing; and
presenting the debate script with annotations in real-time to students participating in the live debate and to an instructor, wherein the debate script is utilized for advancing arguments spontaneously, and for checking facts commented in the debate script.

8. A computer program product comprising:
a computer readable storage medium readable by one or more processor and storing instructions for execution by the one or more processor for performing a method for providing automated annotation for one or more student works, comprising:
obtaining, by one or more processors running one or more neural networks, training data for the one or more neural networks including a plurality of sample essays, a plurality of sample annotations respectively corresponding to the plurality of sample essays, and a subject content including respective knowledgebases per topic on which the plurality of sample essays had been written;
labeling, by the one or more processors running the one or more neural networks, by use of natural language classification tools operatively coupled to the one or more processor, the plurality of sample essays for gradable components and the plurality of sample annotations for each of annotation types;
building, by the one or more processors running the one or more neural networks, a subject domain comprehension model based on the subject content, by use of the one or more neural networks;
forming, by the one or more processors running the one or more neural networks, an essay annotation model, based on the gradable components of the plurality of sample essays and the annotation types of the plurality of sample annotations;
linking, by the one or more processors running the one or more neural networks, nodes of the subject domain comprehension model respectively representing preconfigured essay elements in the plurality of sample essays that show respective writers of the plurality of sample essays understand respective topics of the plurality of sample essays and nodes of the essay annotation model respectively representing preconfigured annotation elements in the plurality of sample annotations that evaluate the plurality of sample essays, based on respective relevancies between the nodes of the subject domain comprehension model and the nodes of the essay annotation model to thereby automatically provide one of the preconfigured annotation elements, represented by a first node of the subject domain comprehension model, for one of the preconfigured essay elements, and a first node amongst the nodes of the essay annotation model based on an edge between the first node of the subject domain compr

Assignees

• IBM

Inventors

• Plant Laurence
• Cleaver James D
• Harrer Stefan

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title