Image processing method, image processing device, and non-transitory computer readable storage medium

Patent metadata

Field	Value
Publication number	US-12260676-B2
Application number	US-202117527382-A
Country	US
Kind code	B2
Filing date	Nov 16, 2021
Priority date	Sep 13, 2019
Publication date	Mar 25, 2025
Grant date	Mar 25, 2025

Abstract

An image processing method includes: acquiring image data; detecting a face region including at least a part of a face of a person from the image data; setting a first region used for detection of an eye of the person in the face region having been detected; setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion; detecting pupil information indicating the pupil or the iris in the second region; and outputting the pupil information having been detected.

First claim

The invention claimed is:

1. An image processing method of detecting pupil information in an image processing device, the image processing method comprising:
acquiring image data;
detecting a face region including at least a part of a face of a person from the image data;
setting a first region used for detection of an eye of the person in the face region having been detected;
setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;
detecting pupil information indicating the pupil or the iris in the second region;
outputting the pupil information having been detected; and
detecting face orientation information indicating an orientation of a face of the person from the face region;
wherein the predetermined criterion is a criterion for removing, in the first region, a predetermined amount of area from a boundary side of the first region, and
wherein in setting of the second region,
as a rightward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a right boundary in the first region is made narrower and an exclusion width from a left boundary in the first region is made wider, and
as a leftward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a left boundary in the first region is made narrower and an exclusion width from a right boundary in the first region is made wider.

2. The image processing method according to claim 1 , further comprising:
detecting a facial feature point from the face region having been detected;
detecting face orientation information indicating an orientation of a face of the person based on the facial feature point;
detecting eye gaze information indicating an eye gaze of the person based on the pupil information and the face orientation information having been detected; and
outputting the eye gaze information having been detected.

3. The image processing method according to claim 2 , wherein the eye gaze information includes information indicating an eye gaze plane that is a region of a predetermined range with respect to an eye gaze point of the person on a predetermined target plane.

4. The image processing method according to claim 1 , wherein
the image data is captured by a visible light camera, and
the pupil information includes information indicating a center position of the iris and a size of the iris.

5. The image processing method according to claim 1 , wherein
the image data is captured by an infrared light camera, and
the pupil information includes at least one of information indicating a center position of the pupil and a size of the pupil and information indicating a center position of the iris and a size of the iris.

6. The image processing method according to claim 5 , wherein the image data is image data captured by an infrared light camera using infrared light in a predetermined second wavelength band in which a spectral intensity of sunlight is attenuated more than a predetermined first wavelength.

7. An image processing method of detecting pupil information in an image processing device, the image processing method comprising:
acquiring image data;
detecting a face region including at least a part of a face of a person from the image data;
setting a first region used for detection of an eye of the person in the face region having been detected;
setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;
detecting pupil information indicating the pupil or the iris in the second region;
outputting the pupil information having been detected;
detecting an outer corner of an eye and an inner corner of an eye from the face region having been detected, wherein
when the outer corner of the eye and the inner corner of the eye have been detected,
a region that surrounds the outer corner of the eye and the inner corner of the eye and is smaller than a first region set in a case of having failed to detect the outer corner of the eye and the inner corner of the eye is set as the first region in setting of the first region, and
the pupil information is detected in the first region in detection of the pupil information.

8. An image processing method of detecting pupil information in an image processing device, the image processing method comprising:
acquiring image data;
detecting a face region including at least a part of a face of a person from the image data;
setting a first region used for detection of an eye of the person in the face region having been detected;
setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;
detecting pupil information indicating the pupil or the iris in the second region; and
outputting the pupil information having been detected;
detecting a facial feature point from the face region having been detected;
detecting face orientation information indicating an orientation of a face of the person based on the facial feature point;
detecting eye gaze information indicating an eye gaze of the person based on the pupil information and the face orientation information having been detected;
outputting the eye gaze information having been detected detecting at least one of an eyebrow and a corner of a mouth from the face region having been detected; and
estimating a degree of interest of the person based on the eye gaze information and at least one of a position of the eyebrow having been detected and a position of the corner of the mouth having been detected.

9. The image processing method according to claim 8 , further comprising:
estimating an expression of the person based on at least one of a position of the eyebrow having been detected and a position of the corner of the mouth having been detected; and
estimating a degree of interest of the person based on the eye gaze information and information indicating the estimated expression.

10. An image processing device comprising:
a memory; and
a processor that executes a program from the memory so as to operate as:
an acquisition unit acquiring image data;
a face detection unit detecting a face region including an entire or part of a face of a person from the image data;
a first setting unit setting a first region used for detection of an eye of the person in the face region having been detected;
a second setting unit setting a second region estimated to include a pupil or an iris by narrowing the first region based on a predetermined criterion;
a face orientation detection unit detecting face orientation information indicating an orientation of a face of the person from the face region; and
a pupil information detection unit detecting pupil information indicating the pupil or the iris in the second region, and outputting the pupil information,
wherein the predetermined criterion is a criterion for removing, in the first region, a predetermined amount of area from a boundary side of the first region, and
wherein in setting of the second region,
as a rightward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a right boundary in the first region is made narrower and an exclusion width from a left boundary in the first region is made wider, and
as a leftward orientation degree of the face indicated by the face orientation information becomes greater, an exclusion width from a left boundary in the first region is made narrower and an exclusion width from a right boundary in the first region is made wider.

Assignees

• Swallow Incubate Co Ltd
• Panasonic Corp
• Panasonic Holdings Corp

Inventors

• Ohno Toshikazu

Classifications

• G06T2207/10048

  Infrared image · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V40/197

  Matching; Classification · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V40/193 Primary

  Preprocessing; Feature extraction · CPC title