Systems and methods for self-learning, content-aware affect recognition

Patent metadata

Field	Value
Publication number	US-2016180722-A1
Application number	US-201414578623-A
Country	US
Kind code	A1
Filing date	Dec 22, 2014
Priority date	Dec 22, 2014
Publication date	Jun 23, 2016
Grant date	—

Abstract

Systems and methods are disclosed for determining an affective state of a user. A user behavior characteristic is detected in response to content provided to the user. Content metadata indicates a context of the content provided to the user and a probability of the user experiencing at least one expected emotion in response to an interaction with the content. Based on the context and the at least one expected emotion indicated in the content metadata, one or more rules are applied to map the detected user behavior characteristic to an affective state of the user.

First claim

1 . A system to determine an affective state of a user, the system comprising:
a behavior feature extraction module to process information from one or more sensors to detect a user behavior characteristic, the user behavior characteristic generated in response to content provided to the user; and an affective state recognition module to:
receive content metadata indicating a context of the content provided to the user and a probability of the user experiencing at least one expected emotion in response to an interaction with the content;
based on the context and the at least one expected emotion indicated in the content metadata, apply one or more rules to map the detected user behavior characteristic to an affective state of the user; and
output or store the affective state of the user.

2 . The system of claim 1 , wherein the affective state recognition module is further configured to:
receive interaction metadata indicating an interaction between the user and an application or machine configured to present the content to the user; and based on the interaction metadata, update the rules to map the detected user behavior characteristic to the affective state.

3 . The system of claim 2 , wherein the content comprises a plurality of content intervals, and wherein the interaction metadata defines contextual sub-divisions within the content intervals.

4 . The system of claim 1 , wherein the affective state recognition module comprises a content metadata parser to receive the content metadata, and to separate the content metadata into a set of expected affective state and/or emotion labels, and a set of content types with associated content timeframes, and wherein the set of expected affective state and/or emotion labels are associated with a probability within each content timeframe.

5 . The system of claim 4 , wherein the affective state recognition module further comprises a learning module configured to:
receive data comprising the user behavior characteristic, the set of expected affective state and/or emotion labels, and the set of content types with associated content timeframes; process the received data to modify predefined behavior-to-emotion mapping rules to generate a profile for the user comprising a personalized emotion map; and apply the personalized emotion map to the detected user behavior characteristic and the at least one expected emotion to infer the affective state of the user.

6 . The system of claim 5 , wherein the learning module is further configured to update the personalized emotion map based on the detected user behavior characteristic and the at least one expected emotion.

7 . The system of claim 5 , wherein the learning module is configured to execute a transductive learning phase, the learning module comprising:
a real-time data collection module to process the user behavior characteristics, the set of expected affective-state or emotion labels, and the set of content types with associated content timeframes using a vector quantization algorithm to generate accumulated interval features; and a transductive learning module to generate an initial model for emotion mapping, the transductive learning module using a transductive learning algorithm to process the accumulated interval features and the behavior-to-emotion mapping rules.

8 . The system of claim 7 , wherein the learning module is further configured to execute an inductive learning phase, the learning module further comprising:
an inductive learning module to update the personalized emotion map using a machine learning algorithm to process the initial model generated by the transductive learning module, the user behavior characteristics, and the set of expected affective-state and/or emotion labels.

9 . A computer-implemented method of determining an affective state of a user, the method comprising:
receiving information from one or more sensors; processing, on one or more computing devices, the information from the one or more sensors to detect a user behavior as the user consumes content or interacts with a machine; receiving content metadata indicating a context of the content provided to the user and a probability of the user experiencing at least one expected emotion as the user consumes the content or interacts with the machine; based on the context and the at least one expected emotion indicated in the content metadata, applying one or more rules to map the detected user behavior to an affective state of the user.

10 . The computer-implemented method of claim 9 , wherein receiving the content metadata comprises receiving the content metadata from a provider of the content.

11 . The computer-implemented method of claim 9 , further comprising:
receiving interaction metadata indicating an interaction between the user and an application configured to present the content to the user; and based on the interaction metadata, updating the rules to map the detected user behavior to the affective state.

12 . The computer-implemented method of claim 11 , further comprising: processing the interaction metadata to determine a plurality of contextual sub-divisions within content intervals of the content.

13 . The computer-implemented method of claim 9 , further comprising:
parsing the content metadata into a set of expected affective state and/or emotion labels, and a set of content types with associated content timeframes, wherein the set of expected affective state and/or emotion labels are associated with a probability within each content timeframe.

14 . The computer-implemented method of claim 13 , further comprising:
receiving data comprising the user behavior, the set of expected affective state and/or emotion labels, and the set of content types with associated content timeframes; processing the received data to modify predefined behavior-to-emotion mapping rules to generate a profile for the user comprising a personalized emotion map; and applying the personalized emotion map to the detected user behavior and the at least one expected emotion to infer the affective state of the user.

15 . The computer-implemented method of claim 14 , further comprising:
executing a transductive learning phase comprising:
processing the user behavior, the set of expected affective-state or emotion labels, and the set of content types with associated content timeframes using a vector quantization algorithm to generate accumulated interval features; and
generating an initial model for emotion mapping using a transductive learning algorithm to process the accumulated interval features and the behavior-to-emotion mapping rules.

16 . The computer-implemented method of claim 15 , further comprising:
executing an inductive learning phase comprising updating the personalized emotion map using a machine learning algorithm to process the initial model, the user behavior, and the set of expected affective-state and/or emotion labels.

17 . At least one computer-readable storage medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving information from one or more sensors; processing, on one or more computing devices, the information from the one or more sensors to detect a user behavior as the user consumes content or interacts with a machine; receiving content metadata indicating a context of the content provided to the user and a probability of the user experiencing at least one expected emotion as the user consumes the content or interacts with the machine; based on the context

Assignees

• Intel Corp

Inventors

• Yehezkel Raanan
• Stanhill David
• Rond Eyal

Classifications

• G09B5/00 Primary

  Electrically-operated educational appliances (working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F) · CPC title

• G09B19/00

  Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) · CPC title