Learning emotional states using personalized calibration tasks

Patent metadata

Field	Value
Publication number	US-9767349-B1
Application number	US-201615149284-A
Country	US
Kind code	B1
Filing date	May 9, 2016
Priority date	May 9, 2016
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

A method for determining an emotional state of a subject taking an assessment. The method includes eliciting predicted facial expressions from a subject administered questions each intended to elicit a certain facial expression that conveys a baseline characteristic of the subject; receiving a video sequence capturing the subject answering the questions; determining an observable physical behavior experienced by the subject across a series of frames corresponding to the sample question; associating the observed behavior with the emotional state that corresponds with the facial expression; and training a classifier using the associations. The method includes receiving a second video sequence capturing the subject during an assessment and applying features extracted from the second image data to the classifier for determining the emotional state of the subject in response to an assessment item administered during the assessment.

First claim

What is claimed is:

1. A method for determining an emotional state of a subject taking an assessment, the method comprising:
generating a calibration task to elicit predicted responses from an associated subject administered the task, wherein each portion of the task is intended to elicit a certain emotional response that conveys a baseline characteristic of the associated subject;
receiving video data capturing the associated subject performing the calibration task, wherein each frame of the video data is synchronized within the task to correspond to a portion of the task;
processing the video data for determining an observable physical behavior experienced by the associated subject across a series of frames during the each portion of the task;
detecting an emotional response experienced by the associated subject across the series of frames corresponding to the each portion of the task;
associating (or tagging) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the emotional responses; and,
training a classifier using the features extracted from the image data, where each class is one of the categories associated with the observed behavior.

2. The method of claim 1 , wherein the calibration task that is used for training includes:
a first class including a set of elicited behaviors associate associated with a neutral and relaxed emotional state; and
a second class that includes all other elicited behaviors,
wherein a baseline is established for the associated subject by training the classifier on the features extracted from the image data from both the first and second classes to learn the neutral and relaxed emotional state of the associated subject.

3. The method of claim 1 , wherein the calibration task includes multiple questions across a range of difficulty levels, wherein at least two questions are different levels of difficulty, wherein each level of difficulty corresponds with a different one of the predicted responses and the each portion of the task is intended to elicit the certain emotional response.

4. The method of claim 1 further comprising:
using a mobile computing device including the image capture device, monitoring the associated subject during an administration of the calibration task to the associated subject.

5. The method of claim 1 , wherein each certain emotional response is a predicted facial expression that conveys the emotional state of the associated subject.

6. The method of claim 1 further comprising:
receiving second image data capturing the associated subject during an assessment administered after the calibration task;
applying features extracted from the second image data to the classifier for determining the emotional state of the associated subject in response to an assessment item administered during the assessment.

7. The method of claim 1 further comprising:
using a sensor, monitoring the associated subject during the calibration task, wherein the sensor measures a biometric indicative of the emotional state of the associated subject.

8. The method of claim 1 further comprising:
receiving the image data capturing at least two associated subjects performing the calibration task, wherein the calibration task includes questions covering at least two topics;
detecting observable physical behaviors experienced by the at least two associated subjects during the each portion of the task;
in response to determining a pattern of similarity between the observable physical behaviors, associating (or tagging) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the intended emotional responses;
training the classifier using the associations.

9. The method of claim 1 , wherein the processing the image data includes:
detecting the associated subject in an image frame of the image data;
applying a landmark detection (or Viola-Jones facial detector) algorithm to the image frame to determine a facial region;
applying a deformable part-based model (DPM) to determine a region of interest in the facial region corresponding to a facial landmark of the associated subject;
extracting features in the region of interest;
associating the features with an emotional category; and
training the classifier using the association.

10. The method of claim 9 , further comprising:
generating feature histograms from one or more of the extracted features;
concatenating the extracted features to generate a final feature descriptor; and,
training the classifier using the final feature descriptor.

11. The method of claim 9 , wherein the processing the image data further includes:
performing a coordinate transform on the region of interest across multiple image frames.

12. A method for determining an emotional state of a subject taking an assessment, the method comprising:
generating sample questions to elicit predicted facial expressions from an associated subject administered the questions, wherein each question is intended to elicit a certain facial expression that conveys a baseline characteristic of the associated subject;
receiving a video sequence capturing the associated subject answering the questions, wherein each frame of the video sequence is synchronized within a sample question;
determining an observable physical behavior experienced by the associated subject across a series of frames corresponding to the sample question;
detecting a facial expression conveyed by the associated subject across the series of frames corresponding to the question;
associating (or tagging) the observed behavior with the emotional state that corresponds with the facial expression;
training a classifier using the associations;
receiving a second video sequence capturing the associated subject during an assessment administered after the sample questions;
applying features extracted from the second image data to the classifier for determining the emotional state of the associated subject in response to an assessment item administered during the assessment.

13. A calibration system for determining an emotional state of a subject taking an assessment, the system comprising:
a processor, and
a non-transitory computer readable memory storing instructions that are executable by the processor to perform the operations of:
generating a calibration task to elicit predicted responses from an associated subject administered the task, wherein each portion of the task is intended to elicit a certain emotional response that conveys a baseline characteristic of the associated subject;
receiving image data from an image capture device capturing the associated subject performing the calibration task, wherein each frame of the image data is synchronized to correspond to a portion of the task;
determining an observable physical behavior experienced by the associated subject across a series of frames during the portion of the task;
detecting an emotional response experienced by the associated subject across the series of frames corresponding to the portion of the task;
associating (or tag) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the emotional responses;
training a classifier using the associations;
receiving second image data capturing the associated subject during an assessment administered after the calibration task; and,
applying features extracted from the second image data to the classifier for determining the emotional state of the associated subject in response

Assignees

• Xerox Corp

Inventors

• Shreve Matthew Adam
• Kumar Jayant
• Bala Raja
• Emmett Phillip J
• Clar Megan
• Subramanian Jeyasri
• Harte Eric

Classifications

• G06K9/00335 Primary

  Physics · mapped topic

• G06K9/00315

  Physics · mapped topic

• G06K9/6269

  Physics · mapped topic

• G06K9/00268

  Physics · mapped topic

• G06K9/00228

  Physics · mapped topic