Goggle type display device, eye gaze detection method, and eye gaze detection system

Patent metadata

Field	Value
Publication number	US-2019082170-A1
Application number	US-201816123111-A
Country	US
Kind code	A1
Filing date	Sep 6, 2018
Priority date	Sep 8, 2017
Publication date	Mar 14, 2019
Grant date	—

Abstract

A goggle type display device with a goggle-shaped frame and a display mounted on said frame includes: an image-capturing unit disposed inside of the frame; and an eye gaze position estimator that estimates an eye gaze position on the basis of images of an eye and a ridge of a nose included in a facial image captured by said image-capturing unit.

First claim

What is claimed is:

1 . A goggle type display device with a goggle-shaped frame and a display mounted on said frame comprising:
an image-capturing unit disposed inside of said frame; and an eye gaze position estimator configured to estimate an eye gaze position on the basis of images of an eye and a ridge of a nose included in a facial image captured by said image-capturing unit.

2 . The goggle type display device according to claim 1 , wherein said eye gaze position estimator comprises:
a target site detector configured to detect an image of a pupil or an iris of said at least one of the eyes and an image of said ridge of the nose from said facial image; and an eye gaze coordinate generator configured to generate coordinates of said eye gaze position which is with reference to said display on the basis of said image of the pupil or the iris and said image of the ridge of the nose.

3 . The goggle type display device according to claim 2 , wherein said eye gaze coordinate generator generates coordinates of said eye gaze position on the basis of a shape of said image of the pupil or the iris and a position of said pupil or the iris with reference to said ridge of the nose.

4 . The goggle type display device according to claim 2 , wherein
said target site detector detects images of an upper lid and a lower lid of said at least one of the eyes from said facial image, and said eye gaze coordinate generator generates coordinates of said eye gaze position on the basis of angles of said upper lid and said lower lid which is with reference to said ridge of the nose.

5 . The goggle type display device according to claim 2 , wherein
said goggle-shaped frame includes a pair of temples, and said image-capturing unit is disposed on an inner side of at least one of said pair of temples.

6 . The goggle type display device according to claim 2 , further comprising a second image-capturing unit configured to obtain a second facial image which includes said at least one of the eyes and an outer corner of the eye or a contour of a cheek, wherein
said target site detector detects an image of a pupil or an iris of said at least one of the eyes and an image of said outer corner of the eye or said contour of the cheek from said second facial image, and said eye gaze coordinate generator generates coordinates of said eye gaze position on the basis of a shape of said image of the pupil or the iris and a position of said pupil or said iris with reference to said outer corner of the at least one of the eyes or said contour of the cheek.

7 . The goggle type display device according to claim 6 , wherein
said target site detector detects positions of an upper lid and a lower lid of said at least one of the eyes and a position of said outer corner of the eye or said contour of the cheek from said second facial image, and said eye gaze coordinate generator generates coordinates of said eye gaze position on the basis of angles of said upper lid and said lower lid which is with reference to said outer corner of the eye or said contour of the cheek.

8 . The goggle type display device according to claim 6 , wherein the second image-capturing unit is disposed in a bridge of said goggle-shaped frame.

9 . The goggle type display device according to claim 6 , wherein said eye gaze position estimator estimates said eye gaze position by synthesizing the coordinates of the eye gaze position generated on the basis of said facial image and the coordinates of the eye gaze position generated on the basis of said second facial image.

10 . An eye gaze detection method in a goggle type display device, the goggle type display device including a goggle-shaped frame and a display and capable of being worn on a head of a user, the method comprising:
a step of photographing a face of the user wearing said frame to obtain a facial image of said user including at least one of eyes and a ridge of a nose; and a step of estimating an eye gaze position on the basis of said facial image.

11 . The eye gaze detection method according to claim 10 , wherein said step of estimating the eye gaze position comprises:
a step of detecting an image of a pupil or an iris of said at least one of the eyes and an image of said ridge of the nose from said facial image; and a step of generating coordinates of said eye gaze position which is with reference to said display on the basis of said image of the pupil or the iris and said image of the ridge of the nose.

12 . The eye gaze detection method according to claim 11 , wherein in said step of generating coordinates of the eye gaze position, coordinates of the eye gaze position are generated on the basis of a shape of said image of the pupil or the iris and a position of said pupil or said iris which is with reference to said ridge of the nose.

13 . The eye gaze detection method according to claim 11 , wherein said step of generating coordinates of the eye gaze position comprises:
a step of detecting images of an upper lid and a lower lid of said at least one of the eyes; and a step of generating coordinates of the eye gaze position on the basis of angles of said upper lid and said lower lid which are with reference to said ridge of the nose.

14 . The eye gaze detection method according to claim 11 , further comprising:
a step of obtaining a second facial image including said at least one of the eyes and an outer corner of the eye or a contour of a cheek; and a step of detecting an image of said pupil or said iris of the at least one of the eyes and an image of said outer corner of the eye or said contour of the cheek from said second facial image, wherein in said step of generating coordinates of the eye gaze position, coordinates of the eye gaze position are generated on the basis of a shape of said image of the pupil or the iris and a position of said pupil or said iris which is with reference to said outer corner of the at least one of the eyes or said contour of the cheek.

15 . The eye gaze detection method according to claim 14 , wherein said step of generating coordinates of the eye gaze position comprises:
a step of detecting images of an upper lid and a lower lid of said at least one of the eyes and an image of said outer corner of the eye or said contour of the cheek from said second facial image; and a step of generating said coordinates of the eye gaze position on the basis of angles of said upper lid and said lower lid which are with reference to said outer corner of the eye or said contour of the cheek.

16 . The eye gaze detection method according to claim 14 , wherein said step of estimating the eye gaze position comprises a step of synthesizing the coordinates of the eye gaze position generated on the basis of said facial image and the coordinates of the eye gaze position generated on the basis of said second facial image to estimate said eye gaze position.

17 . An eye gaze detection system comprising:
an image-capturing unit configured to photograph a facial image including at least one of eyes and a ridge of a nose; a target site detector configured to detect an image of said at least one of the eyes and an image of said ridge of the nose from said facial image; and an eye gaze position detector configured to detect an eye gaze position on the basis of said image of the at least one of the eyes and said image of the ridge of the nose.

18 . The eye gaze detection system according to claim 17 , wherein
said target site detector detects an image of a pupil or an iris of said at least one of the eyes,

Assignees

• Lapis Semiconductor Co Ltd

Inventors

• Akahori Hiroji

Classifications

• G06V40/193 Primary

  Preprocessing; Feature extraction · CPC title

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title

• H04N23/57

  Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices · CPC title

• A61F9/06 Primary

  Masks, shields or hoods for welders · CPC title

• H04N13/383 Primary

  for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes · CPC title