Systems and methods for automatically assessing constructed recommendations based on sentiment and specificity measures

Patent metadata

Field	Value
Publication number	US-10515153-B2
Application number	US-201414280088-A
Country	US
Kind code	B2
Filing date	May 16, 2014
Priority date	May 16, 2013
Publication date	Dec 24, 2019
Grant date	Dec 24, 2019

Abstract

A computer-implemented method of training an assessment model for assessing constructed texts expressing opinions on subjects includes accessing a plurality of training texts, which are constructed texts. The training texts are analyzed with the processing system to derive values of a plurality of linguistic features of an assessment model. At least one of the plurality of linguistic features relates to sentiment and at least one of the plurality of linguistic feature relates to specificity. The assessment model is trained with the processing system based on the values of the plurality of linguistic features. Based on the training, a weight for each of the plurality of linguistic features is determined. The assessment model is calibrated to include the weights for at least some of the plurality of linguistic features such that the assessment model is configured to generate assessment measures for constructed texts expressing opinions on subjects.

First claim

What is claimed is:

1. A computer-implemented method of training an assessment model for assessing constructed texts expressing opinions on subjects of education or employment, comprising:
receiving, using a processing system, a plurality of training texts, each of the training texts being a constructed text having one or more sentences, the one or more sentences expressing opinions on a subject of education or employment, wherein each sentence is annotated by a content type that defines whether the sentence is relevant to an evaluation on subjects of education or employment;
iteratively analyzing the training texts, using the processing system after the receiving, to derive feature values of a plurality of linguistic features of the constructed text, the feature values numerically representing each of the plurality of linguistic features, and determine the plurality of linguistic features based on the content type being relevant to the evaluation on subjects of education or employment, wherein at least one of the plurality of linguistic features relates to sentiment and at least one of the plurality of linguistic features relates to specificity;
training an assessment model, using the processing system after the iteratively analyzing, based on (i) the feature values of the plurality of linguistic features analyzed from the training texts, (ii) assigned measures of specificity of the opinions expressed in the training texts, and (iii) assigned measures of sentiment of the opinions expressed in the training texts;
determining, using the processing system after the training, a weight for each of the plurality of linguistic features based on the training and based on regression analysis of the feature values of plurality of linguistic features;
calibrating, using the processing system after the determining, the assessment model to include a combination of the feature values and the weights for at least some of the plurality of linguistic features such that the assessment model is configured to generate assessment measures for constructed texts expressing opinions on subjects of education or employment; and
outputting, using the processing system after the calibrating, outcome data associated with each training text in the plurality of training texts or in a second plurality of training texts, wherein the outcome data measures a level of success experienced by the subject of the training text based on application of the weights to the feature values of the plurality of linguistic features.

2. The computer-implemented method of claim 1 , wherein the assessment model comprises a specificity model and a sentiment model, wherein the specificity model is trained based on assigned measures of specificity for the training texts and at least some of the feature values of the plurality of linguistic features, wherein the sentiment model is trained based on assigned measures of sentiment for the training texts and at least some of the feature values of the plurality of linguistic features, wherein the specificity model and the sentiment model are trained separately.

3. The computer-implemented method of claim 1 , wherein each training text is associated with an assigned rating, and wherein the training of the assessment model is further based on the assigned ratings.

4. The computer-implemented method of claim 3 , wherein the assessment model comprises a rating model for determining ratings for constructed texts, and wherein the rating model is trained based on the assigned ratings and at least some of the feature values of the plurality of linguistic features.

5. The computer-implemented method of claim 1 , wherein each training text includes one or more text segments, each of which is associated with an assigned content type, wherein the assessment model comprises a content-type model for determining content types of text segments of constructed texts, wherein the content-type model is trained based on the assigned content types of text segments of the training texts.

6. The computer-implemented method of claim 1 , wherein each training text includes one or more text segments, wherein the training of the assessment model is further based on a proportion of the one or more text segments in each training text that are sufficiently related to the subject of the training text.

7. The computer-implemented method of claim 1 , further comprising:
predicting, using the processing system, correlations based on the level of success experienced by the subject of the training text;
wherein the training of the assessment model is further based on the outcome data associated with each training text and the predicted correlations;
wherein the assessment measures generated by the assessment model represent predictions of outcomes.

8. The computer-implemented method of claim 7 , wherein the training of the assessment model is further based on one or more external measures of proficiency associated with each training text in the plurality of training texts or in a second plurality of training texts.

9. The computer-implemented method of claim 1 , wherein training of the assessment model is further based on a quantity of instances where the training texts mention their respective subjects.

10. The computer-implemented method of claim 1 , wherein training of the assessment model is further based on instances of hedge words appearing in the training texts.

11. A computer-implemented method of assessing constructed texts expressing opinions on subjects of education or employment, comprising:
iteratively analyzing, using the processing system, a constructed text having one or more sentences, the one or more sentences expressing an opinion on a subject of education or employment, wherein each sentence is annotated by a content type that defines whether the sentence is relevant to an evaluation on subjects of education or employment to derive feature values of a plurality of linguistic features of the constructed text, the feature values numerically representing each of the plurality of linguistic features, and determine the plurality of linguistic features of the constructed text based on the content type being relevant to the evaluation on subjects of education or employment, wherein the plurality of linguistic features correspond to independent variables in an assessment model, wherein at least some of the feature values of the plurality of linguistic features are used to determine a level of specificity of the opinion expressed in the constructed text and a level of sentiment of the opinion expressed in the constructed text;
calculating, using the processing system after the iteratively analyzing, an assessment value based on application of weights to the feature values of the plurality of linguistic features according to the assessment model, wherein the weights are determined based on regression analysis of the feature values of the plurality of linguistic features; and
outputting, using the processing system after the calculating, an assessment of the constructed text based on the assessment value.

12. A non-transitory computer-readable medium for training an assessment model for assessing constructed texts expressing opinions on subjects of education or employment encoded with instructions for causing a processing system to execute steps comprising:
receiving, using a processing system, a plurality of training texts, each of the training texts being a constructed text having one or more sentences, the one or more sentences expressing opinions on a subject of education or employment, wherein each sentence is annotated by a content type that defines whether the sentence is relevant to an evaluation on subjects of

Assignees

• Educational Testing Service

Inventors

• Heilman Michael
• Breyer F Jay
• Flor Michael

Classifications

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F17/2785 Primary

  Physics · mapped topic