Emotion-aw are reactive interface

Patent metadata

Field	Value
Publication number	US-11334376-B2
Application number	US-201816184601-A
Country	US
Kind code	B2
Filing date	Nov 8, 2018
Priority date	Feb 13, 2018
Publication date	May 17, 2022
Grant date	May 17, 2022

Abstract

A computer-implemented method of providing an emotion-aware reactive interface in an electronic device includes receiving an image of a user as an input and identifying a multi-modal non-verbal cue in the image. The method further includes interpreting the multi-modal non-verbal cue to determine a categorization and outputting a reactive interface event determined based on the categorization.

First claim

What is claimed is:

1. A computer-implemented method of providing an emotion-aware reactive interface in an electronic device, the method comprising:
receiving, by a processor, image data from an image captured by a camera of the electronic device as an input via a user interface of an application on a display of the electronic device;
analyzing the image data, wherein the image comprises facial features and a plurality of subjects in the image to identify one or more subjects associated with the facial features, wherein a plurality of multi-modal non-verbal cues are associated with the one or more subjects corresponding to the facial features;
prioritizing multi-modal non-verbal cues of the plurality of multi-modal non-verbal cues based on a size and a position of the one or more subjects associated with the multi-modal non-verbal cues within the captured image and determining a first multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image;
determining a categorization of the first multi-modal non-verbal cue, wherein the categorization is determined from a set of categorizations defined in an interpretation layer; and
outputting a reactive interface event determined based on the first multi-modal non-verbal cue and its categorization,
wherein the reactive interface event is consumed by the application to provide one or more selectable items of graphic content on the user interface of the application provided on the display of the electronic device.

2. The method of claim 1 , wherein the categorization of the first multi-modal non-verbal cue is associated with at least one of an emotion, a gesture, a text, a scene, or a context.

3. The method of claim 1 , further comprising:
determining a second multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image;
determining a second categorization of the second multi-modal non-verbal cue; and
outputting the reactive interface event determined based in part on the second multi-modal non-verbal cue and the second categorization.

4. The method of claim 1 , wherein the plurality of multi-modal non-verbal cues comprise at least one of a facial expression, a pose, a gesture, a text, an object, or a scene.

5. The method of claim 1 , wherein the reactive interface event is utilized by an application on the electronic device to provide at least one of a background effect, a pop-up image, an avatar, or a visualization in a visual interface of the application.

6. An apparatus comprising:
a display;
a camera; and
a processor configured to:
receive, by the processor, via a user interface of an application on the display, image data from an image captured by the camera of the apparatus as an input,
analyze the image data comprising facial features with a plurality of subjects in the image to identify one or more subjects associated with the facial features, wherein a plurality of multi-modal non-verbal cues are associated with the one or more subjects corresponding to the facial features,
prioritize multi-modal non-verbal cues of the plurality of multi-modal non-verbal cues based on a size and a position of the plurality of subjects associated with the multi-modal non-verbal cues within the captured image and determine a first multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image,
determine a categorization of the first multi-modal non-verbal cue, wherein the categorization is determined from a set of categorizations defined in an interpretation layer, and
output a reactive interface event determined based on the first multi-modal non-verbal cue and its categorization,

wherein the reactive interface event is consumed by the application to provide one or more selectable items of graphic content on the user interface of the application provided on the display.

7. The apparatus of claim 6 , wherein the categorization of the first multi-modal non-verbal cue is associated with at least one of an emotion, a gesture, a text, a scene, or a context.

8. The apparatus of claim 6 , wherein the processor is further configured to:
determine a second multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image,
determine a second categorization of the second multi-modal non-verbal cue, and
output the reactive interface event determined based in part on the second multi-modal non-verbal cue and the second categorization.

9. The apparatus of claim 6 , wherein the plurality of multi-modal non-verbal cues comprise at least one of a facial expression, a pose, a gesture, a text, an object, or a scene.

10. The apparatus of claim 6 , wherein the reactive interface event is utilized by an application on the apparatus to provide at least one of a background effect, a pop-up image, an avatar, or a visualization in a visual interface of the application.

11. A non-transitory computer-readable medium comprising program code, which when executed by a processor, causes the processor to:
receive, by the processor, via a user interface of an application on a display of an electronic device, image data from an image captured by a camera of the electronic device as an input,
analyze the image data comprising facial features with a plurality of subjects in the image to identify one or more subjects associated with the facial features, wherein a plurality of multi-modal non-verbal cues are associated with the one or more subjects corresponding to the facial features,
prioritize multi-modal non-verbal cues of the plurality of multi-modal non-verbal cues based on a size and a position of the one or more subjects associated with the multi-modal non-verbal cues within the captured image and determine a first multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image,
determine a categorization of the first multi-modal non-verbal cue, wherein the categorization is determined from a set of categorizations defined in an interpretation layer, and
output a reactive interface event determined based on the first multi-modal non-verbal cue and its categorization,
wherein the reactive interface event is consumed by the application to provide one or more selectable items of graphic content on the user interface of the application provided on the display.

12. The non-transitory computer-readable medium of claim 11 , wherein the categorization of the first multi-modal non-verbal cue is associated with at least one of an emotion, a gesture, a text, a scene, or a context.

13. The non-transitory computer-readable medium of claim 11 , further comprising program code, which, when executed by the processor, causes the processor to:
determine a second multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image,
determine a second categorization of the second multi-modal non-verbal cue, and
output the reactive interface event determined based in part on the second multi-modal non-verbal cue and the second categorization.

14. The non-transitory computer-readable medium of claim 11 , wherein the plurality of multi-modal non-verbal cues comprise at least one of a facial expression, a pose, a gesture, a text, an object, or a scene.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Jihee
• Bendale Abhijit
• Ying Zhihan
• Gibbs Simon
• Luo Lu

Classifications

• G06V40/174 Primary

  Facial expression recognition · CPC title

• G06F3/0481

  based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• H04M2250/12

  including a sensor for measuring a physical value, e.g. temperature or motion · CPC title

• G06F9/453 Primary

  Help systems · CPC title