Systems and methods for determining emotions based on user gestures

Patent metadata

Field	Value
Publication number	US-10514766-B2
Application number	US-201514735060-A
Country	US
Kind code	B2
Filing date	Jun 9, 2015
Priority date	Jun 9, 2015
Publication date	Dec 24, 2019
Grant date	Dec 24, 2019

Abstract

Various embodiments of the invention allow to detect and analyze gestures, such as tapping and swiping patterns, that a user performs in the process of interacting with a computing device to determine the user's mood therefrom, so as to initiate an appropriate response. In certain embodiments, this is accomplished, without requiring labeled training data, by monitoring a user-device interaction via sensors and analyzing the sensor data based on contextual data via a processor to determine a gesture and one or more properties associated with an emotional state of the user. A response is generated based on the identified emotional state.

First claim

What is claimed is:

1. A method for determining user emotional states, the method comprising:
receiving, via one or more standard inputs of a device, an initial set of input data comprising a first set of contextual data and gesture data relating to a user-device interaction, the initial set of input data comprising generic data and being unlabeled;
generating derived data from the generic data;
based on at least one of the generic data or the derived data, identifying a first set of patterns;
assigning to the first set of patterns a first set of characteristics indicative of a generic user emotional state;
applying the first set of characteristics to a generic model to generate a generic emotional state profile;
collecting user-specific input data comprising a second set of contextual data and gesture data regarding interactions between a user and a computing device;
clustering the user-specific input data based on, at least, a second set of characteristics;
applying the second set of characteristics to a user-specific model to generate a user-specific emotional state profile regarding the interactions, wherein generating the user-specific emotional state profile comprises comparing the user's behavior against the generic model;
feeding back the clustered user-specific input data as inputs to the user-specific model to refine the user-specific emotional state profile by adjusting a threshold value in the user-specific model by assigning to a parameter of the user-specific emotional state profile a weighing factor that is based on a relationship between the gesture data and the second set of contextual data; and
using the user-specific emotional state profile and the second set of contextual data to identify a same user emotional state in at least two different environments.

2. The method according to claim 1 , further comprising correlating at least some of the gesture data and at least some of the contextual data with a user emotional state.

3. The method according to claim 1 further comprising inputting collected input data comprising contextual data and gesture data into the user emotional state profile to determine a user emotional state during a time period in which at least some of the collected input data was obtained.

4. The method according to claim 1 , further comprising updating the user-specific emotional state profile based on a second user-specific profile that is assigned to another user.

5. The method according to claim 1 , wherein the user-specific emotional state profile comprises a trained set of characteristics.

6. The method according to claim 1 , wherein the gesture data comprises information about at least one of a swipe pattern, a pressure exerted by a finger, a finger size, a speed of the gesture, a start or stop location, and a change in intensity of the gesture, and wherein contextual data comprises information about at least one of an activity of a user, a location of a user, and other nearby users.

7. The method according to claim 1 , further comprising triggering a reference emotional state in a training session.

8. The method according to claim 7 , wherein triggering the reference emotional state comprises one of preventing the computing device from responding to a user input and causing a delay in responding to the user input.

9. The method according to claim 1 , further comprising applying a testing procedure to evaluate one of a cognitive ability and a performance prior to permitting a user-device interaction.

10. A method for determining a user emotional state, the method comprising:
receiving, via one or more standard inputs of a device, an initial set of input data comprising a first set of contextual data and gesture data relating to a user-device interaction, the initial set of input data comprising generic data and being unlabeled;
based on at least the generic data, identifying a first set of patterns associated with a first set of characteristics that are indicative of a generic user emotional state;
applying the first set of characteristics to a generic model to generate a generic emotional state profile;
collecting user-specific input data comprising a second set of contextual data and gesture data regarding interactions between a user and a computing device;
clustering the user-specific input data based on, at least, a second set of characteristics;
applying the second set of characteristics to a user-specific model to generate a user-specific emotional state profile, wherein generating the user-specific emotional state profile comprises comparing the user's behavior against the generic model;
feeding back the clustered user-specific input data as inputs to the user-specific model to refine the user-specific emotional state profile by adjusting a threshold value in the user-specific model;
using the user-specific emotional state profile and the second set of contextual data to identify a user emotional state;
outputting the user emotional state; and
generating a response based on the user emotional state.

11. The method according to claim 10 , wherein the response is designed to do at least one of affecting a change in the user's mood, reengaging the user with a task, and adjusting a difficulty level of the task.

12. The method according to claim 10 , further comprising monitoring the user-device interaction by using sensors communicatively coupled to an analysis module.

13. The method according to claim 10 , wherein the user-specific emotional state profile is generated in a training session by clustering the input data based on characteristics associated with emotional states.

14. The method according to claim 10 , further comprising refining the user-specific emotional state profile based on feedback data.

15. A system for determining an emotional state of a user of a computing system, the system comprising:
one or more standard inputs of a device to collect an initial set of input data related to gesture data and a first set of contextual data, the gesture data relates to a user-device interaction between a user and a computing device, the initial set of input data comprising generic data and requiring no labeling;
an analysis module that performs steps comprising:
based on at least the generic data, identifying a first set of patterns associated with a first set of characteristics that are indicative of a generic user emotional state;
applying the first set of characteristics to a generic model to generate a generic emotional state profile;
collecting user-specific input data comprising a second set of contextual data and gesture data regarding interactions between a user and a computing device;
clustering the user-specific input data based on, at least, a second set of characteristics;
applying the second set of characteristics to a user-specific model to generate a user-specific emotional state profile, wherein generating the user-specific emotional state profile comprises comparing the user's behavior against the generic model;
feeding back the clustered user-specific input data as inputs to the user-specific model to refine the user-specific emotional state profile by adjusting a threshold value in the user-specific model; and
using the user-specific emotional state profile and the second set of contextual data to identify user emotional state; and

a response generator coupled to the analysis module to receive the user emotional state and generate a response based on the user emotional state.

16. The system according to claim 15 , wherein the analysis module further comprises a processing module coupled to the

Assignees

• Dell Products Lp

Inventors

• Gates Carrie Elaine
• Silberman Gabriel Mauricio

Classifications

• G06F2203/011

  Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title