Line-of-sight measurement device

Patent metadata

Field	Value
Publication number	US-11571125-B2
Application number	US-202016787332-A
Country	US
Kind code	B2
Filing date	Feb 11, 2020
Priority date	Mar 28, 2019
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

A line-of-sight measurement device includes: an imaging unit that images a face of a subject; a light illumination unit that illuminates light to an eye of the subject; a camera coordinate system eyeball center coordinate calculation unit that estimates coordinates of an eyeball center, from a face image imaged by the imaging unit; a pupil center calculation unit that estimates coordinates of an apparent pupil center, from a pupil center position on the face image; an eyeball position orientation estimation unit that calculates an optical axis vector toward the pupil center from the eyeball center on the basis of the coordinates of the eyeball center and the apparent pupil center; a corneal reflection image calculation unit that obtains coordinates of a corneal reflection image on the basis of the coordinates of the eyeball center, the optical axis vector, and a predetermined eyeball model; and an image coordinate calculation unit that estimates image coordinates of a corneal reflection image on the face image, from the coordinates of the corneal reflection image.

First claim

What is claimed is:

1. A line-of-sight measurement device comprising:
an imaging unit that images a face of a subject to be observed;
a light illumination unit that illuminates light to an eye of the subject to be observed;
a camera coordinate system eyeball center coordinate calculation unit that estimates three-dimensional coordinates of an eyeball center in a camera coordinate system, from a face image representing the face imaged by the imaging unit;
a pupil center calculation unit that estimates three-dimensional coordinates of an apparent pupil center in the camera coordinate system, from a pupil center position of the eye on the face image;
an eyeball position orientation estimation unit that calculates a three-dimensional optical axis vector toward a three-dimensional position of the pupil center from a three-dimensional position of the eyeball center, in the camera coordinate system, on the basis of the three-dimensional coordinates of the eyeball center in the camera coordinate system and the three-dimensional coordinates of the apparent pupil center in the camera coordinate system;
a corneal reflection image calculation unit that obtains three-dimensional coordinates of a corneal reflection image in the camera coordinate system, on the basis of the three-dimensional coordinates of the eyeball center in the camera coordinate system, the optical axis vector, and a predetermined three-dimensional eyeball model; and
an image coordinate calculation unit that estimates image coordinates of a corneal reflection image on the face image, from the three-dimensional coordinates of the corneal reflection image in the camera coordinate system.

2. The line-of-sight measurement device according to claim 1 , wherein
a positional relationship between the imaging unit and the light illumination unit, a positional relationship between the imaging unit and the eye, and parameters relating to the imaging unit satisfy a predetermined constraint for determining that an imaging direction of the imaging unit and a light illumination direction of the light illumination unit are coaxial.

3. The line-of-sight measurement device according to claim 2 , wherein
the eyeball position orientation estimation unit
calculates an angle formed by a line segment connecting the three-dimensional coordinates of the apparent pupil center in the camera coordinate system and the three-dimensional coordinates of the eyeball center in the camera coordinate system and a line segment connecting the three-dimensional coordinates of the eyeball center in the camera coordinate system and three-dimensional coordinates of the imaging unit,
acquires a predetermined angle correction amount corresponding to the angle, and
calculates the three-dimensional optical axis vector toward the three-dimensional position of the pupil center from the three-dimensional position of the eyeball center in the camera coordinate system, on the basis of a vector connecting the three-dimensional coordinates of the eyeball center in the camera coordinate system and the three-dimensional coordinates of the apparent pupil center in the camera coordinate system, the acquired angle correction amount, the calculated angle, and the three-dimensional coordinates of the eyeball center in the camera coordinate system.

4. A line-of-sight measurement device comprising:
an imaging unit that images a face of a subject to be observed;
a light illumination unit that illuminates light to an eye of the subject to be observed;
a camera coordinate system eyeball center coordinate calculation unit that estimates three-dimensional coordinates of an eyeball center in a camera coordinate system, from a face image representing the face imaged by the imaging unit;
a corneal reflection image calculation unit that estimates three-dimensional coordinates of a corneal reflection image in the camera coordinate system, from a position of the corneal reflection image of the eye on the face image;
an eyeball position orientation estimation unit that calculates a three-dimensional optical axis vector toward a three-dimensional position of a corneal curvature center from a three-dimensional position of the eyeball center, in the camera coordinate system, on the basis of the three-dimensional coordinates of the eyeball center in the camera coordinate system and the three-dimensional coordinates of the corneal reflection image in the camera coordinate system;
a pupil center calculation unit that obtains three-dimensional coordinates of an apparent pupil center in the camera coordinate system, on the basis of the three-dimensional coordinates of the eyeball center in the camera coordinate system, the optical axis vector, and a predetermined three-dimensional eyeball model; and
an image coordinate calculation unit that estimates image coordinates of a pupil center on the face image, from the three-dimensional coordinates of the apparent pupil center in the camera coordinate system.

5. The line-of-sight measurement device according to claim 4 , wherein
a positional relationship between the imaging unit and the light illumination unit, a positional relationship between the imaging unit and the eye, and parameters relating to the imaging unit satisfy a predetermined constraint for determining that an imaging direction of the imaging unit and a light illumination direction of the light illumination unit are coaxial.

6. The line-of-sight measurement device according to claim 5 , wherein
the pupil center calculation unit
calculates an angle formed by a line segment connecting the three-dimensional coordinates of the eyeball center in the camera coordinate system and the three-dimensional coordinates of the corneal curvature center of the camera coordinate system and a line segment connecting the three-dimensional coordinates of the eyeball center in the camera coordinate system and three-dimensional coordinates of the imaging unit, on the basis of the optical axis vector and the three-dimensional coordinates of the eyeball center in the camera coordinate system,
acquires a predetermined angle correction amount corresponding to the calculated angle,
calculates a vector connecting the three-dimensional coordinates of the eyeball center in the camera coordinate system and the three-dimensional coordinates of the apparent pupil center in the camera coordinate system, on the basis of the three-dimensional coordinates of the eyeball center in the camera coordinate system, the optical axis vector, and the angle correction amount, and
obtains the three-dimensional coordinates of the apparent pupil center in the camera coordinate system, on the basis of the calculated vector, the three-dimensional coordinates of the eyeball center in the camera coordinate system, and a predetermined three-dimensional eyeball model.

Assignees

• Aisin Corp
• Toyota Motor Co Ltd

Inventors

• Kojima Shin-Ichi
• Osuga Shin
• Kato Takashi
• Yamada Yuya
• Matsumura Takeshi

Classifications

• A61B3/0008

  provided with illuminating means (A61B3/117, A61B3/14 take precedence) · CPC title

• A61B3/158 Primary

  of corneal reflection · CPC title

• G06V40/19

  Sensors therefor · CPC title

• A61B3/113 Primary

  for determining or recording eye movement · CPC title