Determining emotions using camera-based sensing

Patent metadata

Field	Value
Publication number	US-10874309-B2
Application number	US-201815951068-A
Country	US
Kind code	B2
Filing date	Apr 11, 2018
Priority date	May 1, 2017
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

In one embodiment, a computer-readable non-transitory storage medium embodies software that is operable when executed to, in real time, capture, by a single sensor, a number of images of a user; determine, based on the number of images, one or more short-term cardiological signals of the user during a period of time; estimate, based on the cardiological signals, a first short-term emotional state of the user; determine, based on the number of images, one or more short-term neurological signals of the user during the period of time; estimate, based on the neurological signals, a second short-term emotional state of the user; compare the first estimated emotional state to the second estimated emotional state; and in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine a short-term emotion of the user during the period of time.

First claim

What is claimed is:

1. One or more computer-readable non-transitory storage media embodying software that is operable when executed to, in real time:
capture, by one or more optical cameras, a plurality of images of a user's face;
determine, based on a first analysis of the plurality of images of the user's face, one or more short-term cardiological signals of the user during a period of time;
estimate, based on the cardiological signals, a first short-term emotional state of the user;
determine, based on a second analysis of the plurality of images of the user's face, one or more short-term neurological signals of the user during the period of time;
estimate, based on the neurological signals, a second short-term emotional state of the user;
compare the first estimated emotional state to the second estimated emotional state; and
in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine, based on the first estimated emotional state and second estimated emotional state, a short-term emotion of the user during the period of time.

2. The media of claim 1 , wherein the software is further operable to:
detect the face of the user in the plurality of images of the user's face;
determine, based on photoplethysmogram (PPG) data obtained from the plurality of images of the user's face, one or more of the short-term cardiological signals of the user;
determine, based on facial action coding system (FACS) data obtained from the plurality of images of the user's face, one or more of the short-term neurological signals of the user;
determine, one or more neuroanatomical signals of the user based on the plurality of images of the user's face; and
confirm the short-term emotion of the user based on a consistency between the neuroanatomical signals and the short-term emotion.

3. The media of claim 2 , wherein the FACS data is based on detected characteristics of the user's eye brow, lip width, mouth, teeth visibility, chin lines, nasolabial lines, nose lines, or cheek-based impression of smile lines.

4. The media of claim 2 , wherein one or more of the neuroanatomical signals are detected physical attributes of the user.

5. The media of claim 1 , wherein the software is further operable to determine one or more of the cardiological signals using a linear model based on a sympathovagal balance (SVB) of the user.

6. The media of claim 5 , wherein the software is further operable to:
determine one or more of the cardiological signals using a non-linear statistical model based on heartrate dynamics; and
determine that the short-term emotion corresponds to a positive emotion based on concurrently sensing the SVB and heartrate dynamics and based on determining that the SVB and the heartrate dynamics are consistent with positive emotion.

7. The media of claim 1 wherein the determination of the cardiological and neurological signals are performed concurrently.

8. A method comprising, in real time:
capturing, by one or more optical cameras, a plurality of images of a user's face;
determining, based on a first analysis of the plurality of images of the user's face, one or more short-term cardiological signals of the user during a period of time;
estimating, based on the cardiological signals, a first short-term emotional state of the user;
determining, based on a second analysis of the plurality of images of the user's face, one or more short-term neurological signals of the user during the period of time;
estimating, based on the neurological signals, a second short-term emotional state of the user;
comparing the first estimated emotional state to the second estimated emotional state; and
in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determining, based on the first estimated emotional state and second estimated emotional state, a short-term emotion of the user during the period of time.

9. The method of claim 8 , further comprising:
detecting the face of the user in the plurality of images of the user's face;
determining, based on photoplethysmogram (PPG) data obtained from the plurality of images of the user's face, one or more of the short-term cardiological signals of the user;
determining, based on facial action coding system (FACS) data obtained from the plurality of images of the user's face, one or more of the short-term neurological signals of the user;
determining, one or more neuroanatomical signals of the user based on the plurality of images of the user's face; and
confirming the short-term emotion of the user based on a consistency between the neuroanatomical signals and the short-term emotion.

10. The method of claim 9 , wherein the FACS data is based on detected characteristics of the user's eye brow, lip width, mouth, teeth visibility, chin lines, nasolabial lines, nose lines, or cheek-based impression of smile lines.

11. The method of claim 9 , wherein one or more of the neuroanatomical signals are detected physical attributes of the user.

12. The method of claim 8 , wherein the determination of one or more of the cardiological signals comprises using a linear model based on a sympathovagal balance (SVB) of the user.

13. The method of claim 12 , further comprising:
determining one or more of the cardiological signals using a non-linear statistical model based on heartrate dynamics; and
determining that the short-term emotion corresponds to a positive emotion based on concurrently sensing the SVB and heartrate dynamics and based on determining that the SVB and the heartrate dynamics are consistent with positive emotion.

14. The method of claim 8 , further comprising concurrently performing the determination of the cardiological and neurological signals.

15. A system comprising:
one or more processors; and
a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to, in real-time:
capture, by one or more optical cameras, a plurality of images of a user's face;
determine, based on a first analysis of the plurality of images of the user's face, one or more short-term cardiological signals of the user during a period of time;
estimate, based on the cardiological signals, a first short-term emotional state of the user;
determine, based on a second analysis of the plurality of images of the user's face, one or more short-term neurological signals of the user during the period of time;
estimate, based on the neurological signals, a second short-term emotional state of the user;
compare the first estimated emotional state to the second estimated emotional state; and
in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine, based on the first estimated emotional state and second estimated emotional state, a short-term emotion of the user during the period of time.

16. The system of claim 15 , wherein the processors are further operable to:
detect the face of the user in the plurality of images of the user's face;
determine, based on photoplethysmogram (PPG) data obtained from the plurality of images of the user's face, one or more of the short-term cardiological signals of the user;
determine, based on facial action coding system (FACS) data obtained from the plurality of images of the user's face, one or more of the short-term neurological signals of the user;
determine, one or more neuroanatomical signals of the user based on

Assignees

• Samsung Electronics Co Ltd

Inventors

• Khwaja Ayesha
• Young James
• Wortham Cody
• Sadi Sajid
• Jain Jawahar

Classifications

• A61B5/349

  Detecting specific parameters of the electrocardiograph cycle · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06V40/15

  Biometric patterns based on physiological signals, e.g. heartbeat, blood flow · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V40/176

  Dynamic expression · CPC title