Training template construction apparatus for facial expression recognition and method thereof

Patent metadata

Field	Value
Publication number	US-10699164-B2
Application number	US-201816164970-A
Country	US
Kind code	B2
Filing date	Oct 19, 2018
Priority date	Sep 27, 2018
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

A training template construction apparatus includes a gaze fixation point receiving unit for receiving gaze fixation points of a user that looks a facial picture that expresses random emotion, from an eye tracking apparatus that is operatively associated with the gaze fixation point receiving unit, a gaze pattern extraction unit for extracting a gaze pattern and gaze pattern information via machine-learning of the gaze fixation points received from the gaze fixation point receiving unit, a heat map deduction unit for deducing a heat map using the gaze pattern and the gaze pattern information that are extracted by the gaze pattern extraction unit, a difference heat map deduction unit for deducing a difference value between the heat map deduced from the heat map deduction unit and a heat map of a reference group based on pre-stored facial pictures that express the same emotion and for deducing a difference heat map to which the difference value is applied, and a controller for analyzing the gaze pattern and the difference heat map to generate a training template of a sequence, a time, and a path for user gaze treatment.

First claim

What is claimed is:

1. A training template construction apparatus for facial expression recognition, comprising:
a gaze fixation point receiving unit for receiving gaze fixation points of a user that looks a facial picture that expresses random emotion, from an eye tracking apparatus that is operatively associated with the gaze fixation point receiving unit;
a gaze pattern extraction unit for extracting a gaze pattern and gaze pattern information via machine-learning of the gaze fixation points received from the gaze fixation point receiving unit;
a heat map deduction unit for deducing a heat map using the gaze pattern and the gaze pattern information that are extracted by the gaze pattern extraction unit;
a difference heat map deduction unit for deducing a difference value between the heat map deduced from the heat map deduction unit and a heat map of a reference group based on pre-stored facial pictures that express the same emotion and for deducing a difference heat map to which the difference value is applied; and
a controller for analyzing the gaze pattern and the difference heat map to generate a training template of a sequence, a time, and a path for user gaze treatment.

2. The training template construction apparatus of claim 1 , wherein:
the gaze pattern extraction unit
extracts the gaze pattern and gaze pattern information including an average gaze fixing time when a gaze stays at a facial expression region in the facial picture and a path along which a gaze is moved, from the gaze fixation points.

3. The training template construction apparatus of claim of claim 1 , wherein:
the heat map deduction unit
applies a weight value to the gaze pattern and gaze pattern information to deduce a heat map indicating distribution of different colors in the facial picture depending on a degree of a gaze ratio through the gaze pattern to which the weight value is applied.

4. The training template construction apparatus of claim of claim 1 , wherein:
the difference heat map deduction unit
analyzes the deduced difference value, indicates a region on which a gaze concentrates in the user gaze pattern compared with a gaze pattern of a reference group, in first color, and indicates a region on which a gaze concentrates compared with the user gaze pattern in the gaze pattern of the reference group, in second color that is complementary color of the first color.

5. The training template construction apparatus of claim of claim 4 , wherein:
the controller
generates a training template including a time when a gaze is fixed and a path along which a gaze is moved to allow the gaze of the user to concentrate on a region indicated in the second color for a predetermined time or greater.

6. The training template construction apparatus of claim of claim 1 , wherein:
the controller sets a time when a gaze is fixed and a path along which a gaze is moved to dispose the region around the mouth at a rear portion of the path along which the gaze is moved or to pass or avoid the region around the mouth
when a user gaze pattern is locally indicated in the region around the mouth of a facial picture indicating specific emotion.

7. A method of a training template construction apparatus for facial expression recognition, the method comprising:
receiving gaze fixation points of a user that looks a facial picture that expresses random emotion, from an operatively associated eye tracking apparatus;
extracting a gaze pattern and gaze pattern information via machine-learning of the gaze fixation points;
deducing a heat map using the gaze pattern and the gaze pattern information;
deducing a difference value between the heat map and a heat map of a reference group based on pre-stored facial pictures that express the same emotion and deducing a difference heat map to which the difference value is applied; and
analyzing the gaze pattern and the difference heat map to generate a training template of a sequence, a time, and a path for user gaze treatment.

8. The method of claim 7 , wherein:
the extracting of the gaze pattern and the gaze pattern information includes
extracting the gaze pattern and gaze pattern information including an average gaze fixing time when a gaze stays at a facial expression region in the facial picture and a path along which a gaze is moved, from the gaze fixation points.

9. The method of claim 7 , wherein:
the deducing of the heat map includes
applying a weight value to the gaze pattern and gaze pattern information to deduce a heat map indicating distribution of different colors in the facial picture depending on a degree of a gaze ratio through the gaze pattern to which the weight value is applied.

10. The method of claim 7 , wherein:
the deducing of the difference heat map includes
analyzing the deduced difference value, indicating a region on which a gaze concentrates in the user gaze pattern compared with a gaze pattern of a reference group, in first color, and indicating a region on which a gaze concentrates compared with the user gaze pattern in the gaze pattern of the reference group, in second color that is complementary color of the first color.

11. The method of claim 10 , wherein:
the generating of the training template includes
generating a training template including a time when a gaze is fixed and a path along which a gaze is moved to allow the gaze of the user to concentrate on a region indicated in the second color.

12. The method of claim 7 , wherein:
the generating of the training template includes setting a time when a gaze is fixed and a path along which a gaze is moved to dispose the region around the mouth at a rear portion of the path along which the gaze is moved or to pass or avoid the region around the mouth
when a user gaze pattern is locally indicated in the region around the mouth of a facial picture indicating specific emotion.

13. A non-transitory computer readable recording medium having recorded thereon a program for executing a method of providing a training template generated through a training template construction apparatus, wherein the program executes a function of providing a facial picture that expresses random emotion, a function of receiving a word indicating emotion of the facial picture from a user, a function of determining whether the received word and the emotion of the facial picture correspond to each other, and a function of indicating a specific region of the facial picture and directing a gaze at the specific region of the facial picture for n seconds, or providing gaze movement path points based on the specific region of the facial picture, and directing a gaze form seconds for each gaze movement path when the received word and the emotion of the facial picture do not correspond to each other as a determination result.

14. The non-transitory computer readable recording medium of claim 13 , wherein: the training template is generated based on a difference heat map to which at a gaze pattern of a schizophrenia patient group and a difference value between a heat map of the schizophrenia patient group and a heat map of the reference group are applied; and the specific region is a region on which a gaze concentrates in the gaze pattern of the reference group compared with the gaze pattern of the schizophrenia patient group.

15. The non-transitory computer readable recording medium of claim 14 , wherein: the training template is generated to rapidly pass a limited region around a mouth of the facial picture or to dispose the region around the mouth at a rear portion of the path along which the gaze is moved when the facial picture is a picture

Assignees

• Iac In Nat Univ Chungnam
• Kim Taehyun

Inventors

• Kim Taehyun
• Noh Soo Rim

Classifications

• G06V40/174 Primary

  Facial expression recognition · CPC title

• A61B5/0077 Primary

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title