Image processing method, image processing device, and non-transitory computer readable storage medium

Patent metadata

Field	Value
Publication number	US-12374158-B2
Application number	US-202217741623-A
Country	US
Kind code	B2
Filing date	May 11, 2022
Priority date	Nov 14, 2019
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

An image processing method includes: detecting a position of a facial feature point of a person from the image data; detecting a center position of a pupil of an eye of the person from the image data; detecting a horizontal component and a vertical component of an orientation of the face based on the position of the feature point; estimating a horizontal component of an eye gaze direction of the person with respect to an optical axis of the imaging device based on the horizontal component of the orientation of the face and a distance between the center position of the pupil and the position of the feature point; estimating a vertical component of the eye gaze direction based on at least the vertical component of the orientation of the face; and outputting eye gaze information including the horizontal component and the vertical component of the eye gaze direction.

First claim

The invention claimed is:

1. An image processing method in an image processing device, the image processing method comprising:
acquiring image data captured by an imaging device;
detecting a position of a facial feature point of a person from the image data;
detecting a center position of a pupil of an eye of the person from the image data;
detecting a horizontal component and a vertical component of an orientation of the face based on the position of the feature point;
estimating a horizontal component of an eye gaze direction of the person with respect to an optical axis of the imaging device based on the horizontal component of the orientation of the face and a distance between the center position of the pupil and the position of the feature point;
estimating a vertical component of the eye gaze direction based on at least the vertical component of the orientation of the face; and
outputting eye gaze information including the horizontal component and the vertical component of the eye gaze direction,
wherein the feature point includes an outer corner of the eye and an inner corner of the eye of the person,
in estimation of the horizontal component of the eye gaze direction, a horizontal component of a pupil direction that is a direction of the pupil with respect to the orientation of the face is estimated based on a first ratio of a first distance between the center position of the pupil and a position of the outer corner of the eye with respect to a distance between the outer corner of the eye and the inner corner of the eye and a second ratio of a second distance between the center position of the pupil and a position of the inner corner of the eye with respect to the distance between the outer corner of the eye and the inner corner of the eye, and the horizontal component of the eye gaze direction is estimated by synthesizing the horizontal component of the orientation of the face and the horizontal component of the pupil direction, and
in estimation of the horizontal component of the eye gaze direction, it is determined that the pupil of a right eye is set on the right in response to the first ratio being smaller than the second ratio and the pupil of the right eye is set on the left in response to the second ratio being smaller than the first ratio.

2. The image processing method according to claim 1 , wherein
the feature point includes at least one of an eyebrow and a corner of a mouth, and
a degree of interest of the person is estimated based on the eye gaze information and at least one of a position of the eyebrow and a position of the corner of the mouth.

3. The image processing method according to claim 2 , comprising:
estimating an expression of the person based on at least one of the position of the eyebrow and the position of the corner of the mouth, and
estimating the degree of interest of the person based on the eye gaze information and information indicating the estimated expression.

4. The image processing method according to claim 1 , wherein
the eye gaze information includes information indicating an eye gaze plane that is a region of a predetermined range with respect to an eye gaze point of the person on a predetermined target plane.

5. The image processing method according to claim 1 , wherein
the image data is captured by a visible light camera, and
the center position of the pupil is a center position of an iris.

6. The image processing method according to claim 1 , wherein
the image data is captured by an infrared light camera, and
the center position of the pupil is a center position of a pupil outer edge.

7. The image processing method according to claim 1 , wherein the image data is image data captured by an infrared light camera using infrared light in a predetermined second wavelength band in which a spectral intensity of sunlight is attenuated more than a predetermined first wavelength.

8. An image processing device comprising:
an acquisition unit that acquires image data captured by an imaging device;
a feature point detection unit that detects a position of a facial feature point of a person from the image data;
a pupil detection unit that detects a center position of a pupil of an eye of the person from the image data;
a face orientation detection unit that detects a horizontal component and a vertical component of an orientation of the face based on the position of the feature point;
a horizontal eye gaze estimation unit that estimates a horizontal component of an eye gaze direction of the person with respect to an optical axis of the imaging device based on the horizontal component of the orientation of the face and a distance between the center position of the pupil and the position of the feature point;
a vertical eye gaze estimation unit that estimates a vertical component of the eye gaze direction based on at least the vertical component of the orientation of the face; and
an output unit that outputs eye gaze information including the horizontal component and the vertical component of the eye gaze direction
wherein the feature point includes an outer corner of the eye and an inner corner of the eye of the person,
the horizontal eye gaze estimation unit estimates a horizontal component of a pupil direction that is a direction of the pupil with respect to the orientation of the face based on a first ratio of a first distance between the center position of the pupil and a position of the outer corner of the eye with respect to a distance between the outer corner of the eye and the inner corner of the eye and a second ratio of a second distance between the center position of the pupil and a position of the inner corner of the eye with respect to the distance between the outer corner of the eye and the inner corner of the eye, and estimates the horizontal component of the eye gaze direction by synthesizing the horizontal component of the orientation of the face and the horizontal component of the pupil direction, and
the horizontal eye gaze estimation unit determines that the pupil of a right eye is set on the right in response to the first ratio being smaller than the second ratio and the pupil of the right eye is set on the left in response to the second ratio being smaller than the first ratio.

9. A non-transitory computer readable storage medium storing an image processing program that causes a computer to function as
an acquisition unit that acquires image data captured by an imaging device,
a feature point detection unit that detects a position of a facial feature point of a person from the image data,
a pupil detection unit that detects a center position of a pupil of an eye of the person from the image data,
a face orientation detection unit that detects a horizontal component and a vertical component of an orientation of the face based on the position of the feature point,
a horizontal eye gaze estimation unit that estimates a horizontal component of an eye gaze direction of the person with respect to an optical axis of the imaging device based on the horizontal component of the orientation of the face and a distance between the center position of the pupil and the position of the feature point,
a vertical eye gaze estimation unit that estimates a vertical component of the eye gaze direction based on at least the vertical component of the orientation of the face, and
an output unit that outputs eye gaze information including the horizontal component and the vertical component of the eye gaze direction
wherein the feature point includes an outer corner of the eye and an inner corner of the eye of the person,
the horizontal eye gaze estimation unit estimates a horizontal component of a pupil direction that is a direction of the pupil with respect t

Assignees

• Swallow Incubate Co Ltd
• Panasonic Holdings Corp

Inventors

• Ohno Toshikazu

Classifications

• G06T2207/30201

  Face · CPC title

• G06T2207/10048

  Infrared image · CPC title

• G06V40/166

  using acquisition arrangements · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title