Collection of affect data from multiple mobile devices

Patent metadata

Field	Value
Publication number	US-9934425-B2
Application number	US-201314144413-A
Country	US
Kind code	B2
Filing date	Dec 30, 2013
Priority date	Jun 7, 2010
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

A user interacts with various pieces of technology to perform numerous tasks and activities. Reactions can be observed and mental states inferred from these performances. Multiple devices, including mobile devices, can observe and record or transmit a user's mental state data. The mental state data collected from the multiple devices can be used to analyze the mental states of the user. The mental state data can be in the form of facial expressions, electrodermal activity, movements, or other detectable manifestations. Multiple cameras on the multiple devices can be usefully employed to collect facial data. An output can be rendered based on an analysis of the mental state data.

First claim

What is claimed is:

1. A computer-implemented method for mental state analysis comprising:
obtaining mental state data which is collected on an individual from multiple sources wherein the multiple sources include at least two sources of facial image data, wherein at least one of the facial image data sources collects facial image data intermittently while the individual is looking toward a camera, and wherein the multiple sources include at least one mobile device, including a cell phone, a tablet computer, or a wearable device;
aggregating, using one or more processors, the mental state data from the multiple sources;
obtaining analysis of the mental state data which is aggregated from the multiple sources, wherein the analysis results in mental state analysis;
interpolating the mental state analysis where the mental state data collected is intermittent, wherein the interpolating generates interpolated data for time periods during which mental state data was not collected from one or more of the multiple sources; and
rendering an output based on the analysis of the mental state data.

2. The method of claim 1 wherein the mental state data from multiple sources is tagged.

3. The method of claim 2 wherein the mental state data is tagged with an identity value for the individual.

4. The method of claim 3 wherein the mental state data is tagged with information on context in which the mental state data was collected.

5. The method of claim 2 further comprising partitioning the mental state data based on tagging.

6. The method of claim 1 wherein the at least two sources of the facial image data include one or more of a webcam, a phone camera, a tablet camera, a wearable camera, a room camera, a mobile device, a cell phone, a tablet computer, or a laptop computer.

7. The method of claim 1 wherein the at least one mobile device includes a forward facing camera.

8. The method of claim 1 wherein the analysis of the mental state data is obtained from a web service.

9. The method of claim 1 further comprising performing face detection to determine whether the individual is looking toward the camera.

10. The method of claim 9 further comprising filtering out faces of one or more other people to determine whether the individual is looking toward the camera.

11. The method of claim 1 further comprising interpolating mental state data when the mental state data collected is intermittent.

12. The method of claim 1 further comprising imputing additional mental state data for one or more periods where no mental state data was collected.

13. The method of claim 1 further comprising determining contextual information.

14. The method of claim 13 wherein the contextual information is based on one or more of skin temperature, accelerometer data, a photograph, an email, a text message, a phone log, or GPS information.

15. The method of claim 1 further comprising inferring mental states based on the mental state data which was collected.

16. The method of claim 15 wherein the mental states inferred include one or more of frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, or satisfaction.

17. The method of claim 16 further comprising populating a calendar based on the mental states which were inferred.

18. The method of claim 1 further comprising posting information based on the analysis to a social network page.

19. The method of claim 1 wherein the mental state data includes one or more of a group including physiological data, facial image data, or accelerometer data.

20. The method of claim 19 wherein the physiological data includes one or more of electrodermal activity, heart rate, heart rate variability, skin temperature, or respiration.

21. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:
obtaining mental state data which is collected on an individual from multiple sources wherein the multiple sources include at least two sources of facial image data, wherein at least one of the facial image data sources collects facial image data intermittently while the individual is looking toward a camera, and wherein the multiple sources include at least one mobile device, including a cell phone, a tablet computer, or a wearable device;
aggregating the mental state data from the multiple sources;
obtaining analysis of the mental state data which is aggregated from multiple sources, wherein the analysis results in mental state analysis;
interpolating the mental state analysis where the mental state data collected is intermittent, wherein the interpolating generates interpolated data for time periods during which mental state data was not collected from one or more of the multiple sources; and
rendering an output based on the analysis of the mental state data.

22. A computer system for mental state analysis comprising:
a memory which stores instructions;
one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
obtain mental state data which is collected on an individual from multiple sources wherein the multiple sources include at least two sources of facial image data, wherein at least one of the facial image data sources collects facial image data intermittently while the individual is looking toward a camera, and wherein the multiple sources include at least one mobile device, including a cell phone, a tablet computer, or a wearable device;
aggregate the mental state data from the multiple sources;
obtain analysis of the mental state data which is aggregated from multiple sources, wherein the analysis results in mental state analysis;
interpolate the mental state analysis where the mental state data collected is intermittent, wherein the interpolation generates interpolated data for time periods during which mental state data was not collected from one or more of the multiple sources; and
render an output based on the analysis of the mental state data.

23. The method of claim 1 wherein the facial image data includes abstracted facial expression information.

24. The method of claim 1 wherein the facial image data is used to determine mental state information.

25. The method of claim 1 further comprising stitching mental state analysis from the mental state data from the multiple sources.

26. The method of claim 25 wherein the mental state analysis further comprises identifying a best view, from the multiple sources of facial image data, and stitching the mental state analysis together based on the best view that was identified for the mental state data that was collected intermittently.

27. The method of claim 5 wherein the partitioning of the mental state data is based on tags identifying a source of the intermittent mental state data.

28. The method of claim 2 further comprising analyzing the mental state data that was tagged to generate an emotigraphic profile for the individual.

29. The method of claim 5 further comprising separating the mental state data into two or more groups based on content of tags from the mental state data that was tagged.

Assignees

• Affectiva Inc

Inventors

• El Kaliouby Rana
• Bender Daniel Abraham
• Kodra Evan
• Nowak Oliver Ernst
• Sadowsky Richard Scott
• Senechal Thibaud
• Turcot Panu James

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• A61B5/02055

  Simultaneously evaluating both cardiovascular condition and temperature · CPC title

• A61B5/11

  Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb {(A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721)} · CPC title

• G06Q30/0271

  Personalized advertisement · CPC title

• A61B5/0533

  Measuring galvanic skin response · CPC title