Method for detecting opening and closing of eye, device for detecting opening and closing of eye, and non-transitory computer-readable recording medium recording program for detecting opening and closing of eye

Patent metadata

Field	Value
Publication number	US-12333860-B2
Application number	US-202217841841-A
Country	US
Kind code	B2
Filing date	Jun 16, 2022
Priority date	Dec 22, 2020
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

A device for detecting opening and closing of an eye acquires a first image taken by an image taking device, including a face of a person; generates a second image including an eye region of the person from the first image; generates a third image in which pixels having a gradation value smaller than a threshold are represented at a first luminance value and pixels having a gradation value not smaller than the threshold are represented at a second luminance value, by binarizing the second image; determines whether the eye of the person is opened or closed, on the basis of a height of the third image and a maximum height which is the maximum of a vertical distance between an upper pixel and a lower pixel in a first luminance region having the first luminance value; and outputs information indicative of a result of the determination.

First claim

The invention claimed is:

1. A method for detecting opening and closing of an eye, by a computer, comprising:
acquiring a first image taken by an image taking device, including a face of a person;
generating a second image including an eye region of the person from the first image;
generating a third image in which pixels having a gradation value smaller than a threshold are represented at a first luminance value and pixels having a gradation value not smaller than the threshold are represented at a second luminance value, by binarizing the second image;
determining whether the eye of the person is opened or closed, on the basis of a height of the third image and a maximum height which is the maximum of a vertical distance between an upper pixel and a lower pixel in a first luminance region having the first luminance value; and
outputting information indicative of a result of the determination,
wherein, in the determining:
it is determined whether the maximum height in the first luminance region is larger than a value obtained by multiplying the height of the third image by a first factor; and
it is determined that the eye is opened when the maximum height in the first luminance region is determined to be larger than the value obtained by multiplying the height of the third image by the first factor.

2. The method for detecting opening and closing of an eye according to claim 1 , wherein, in the determining:
it is determined whether the maximum height in the first luminance region is smaller than a value obtained by multiplying the height of the third image by a second factor smaller than the first factor when the maximum height in the first luminance region is determined to be not larger than the value obtained by multiplying the height of the third image by the first factor; and
it is determined that the eye is closed when the maximum height in the first luminance region is determined to be smaller than the value obtained by multiplying the height of the third image by the second factor.

3. The method for detecting opening and closing of an eye according to claim 2 , further comprising generating a fourth image indicative of a face region of the person from the first image, wherein, in the determining, it is determined whether the maximum height in the first luminance region is smaller than a value obtained by multiplying a height of the fourth image by the second factor and a third factor smaller than the second factor when the maximum height in the first luminance region is determined to be not smaller than the value obtained by multiplying the height of the third image by the second factor; and
the eye is determined to be closed when the maximum height in the first luminance region is smaller than the value obtained by multiplying the height of the fourth image by the second factor and the third factor; and the eye is determined to be opened when the maximum height in the first luminance region is not smaller than the value obtained by multiplying the height of the fourth image by the second factor and the third factor.

4. The method for detecting opening and closing of an eye according to claim 1 , further comprising generating a fourth image indicative of a face region of the person from the first image, wherein, in the determining, it is determined whether the maximum height in the first luminance region is smaller than a value obtained by multiplying a height of the fourth image by a second factor smaller than the first factor and a third factor smaller than the second factor when the maximum height in the first luminance region is determined to be not larger than the value obtained by multiplying the height of the third image by the first factor; and
the eye is determined to be closed when the maximum height in the first luminance region is determined to be smaller than the value obtained by multiplying the height of the fourth image by the second factor and the third factor; and the eye is determined to be opened when the maximum height in the first luminance region is not smaller than the value obtained by multiplying the height of the fourth image by the second factor and the third factor.

5. The method for detecting opening and closing of an eye according to claim 2 , wherein the second image has a rectangular shape; and
in the determining, it is determined whether the eye of the person is opened or closed on the basis of a ratio between a height and a width of the second image when the maximum height in the first luminance region is determined to be not smaller than the value obtained by multiplying the height of the third image by the second factor.

6. The method for detecting opening and closing of an eye according to claim 2 , wherein the second image is represented by a red component, a green component, and a blue component, further comprising:
generating from the second image a fifth image represented by a hue component, a saturation component, and a value component when the maximum height in the first luminance region is determined to be not smaller than the value obtained by multiplying the height of the third image by the second factor;
generating a sixth image in which a pixel having a gradation value smaller than a threshold is represented at a third luminance value and a pixel having a gradation value not smaller than the threshold is represented at a fourth luminance value, by binarizing the hue component of the fifth image; and
determining, on the basis of a third luminance region having the third luminance value, whether the eye of the person is opened or closed.

7. The method for detecting opening and closing of an eye according to claim 1 , further comprising:
detecting each position of an upper eyelid and a lower eyelid on the basis of the third image.

8. The method for detecting opening and closing of an eye according to claim 7 , wherein, in the detection of the position of the upper eyelid and the position of the lower eyelid, a morphological gradient operation on the third image is performed to detect the position of the upper eyelid and the position of the lower eyelid.

9. The method for detecting opening and closing of an eye according to claim 1 , wherein the third image is a binary image of one of a right eye and a left eye of the person, further comprising:
detecting each position of an eye outer corner and an eye inner corner on the basis of the third image.

10. The method for detecting opening and closing of an eye according to claim 9 , wherein, in the detection of the position of the eye outer corner and the position of the eye inner corner in the third image,
a position of a pixel which has the first luminance value and is on a left side in lateral directions is detected as the position of one of the eye outer corner and the eye inner corner; and
a position of a pixel which has the first luminance value and is on a right side in the lateral directions is detected as the position of the other of the eye outer corner and the eye inner corner.

11. The method for detecting opening and closing of an eye according to claim 1 , further comprising:
superimposing information indicating whether the eye is opened or closed on a face image of the person on a display.

12. A device for detecting opening and closing of an eye, comprising:
a memory; and
a processor that executes instructions stored on the memory in order to:
acquire a first image taken by an image taking device, including a face of a person;
generate a second image including an eye region of the person from the first image;
generate a third image in which pixels having a gradation value smaller than a threshold are represented at a first luminance valu

Assignees

• Swallow Incubate Co Ltd
• Panasonic Holdings Corp

Inventors

• Ohno Toshikazu

Classifications

• G06T2207/10024

  Color image · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V40/197

  Matching; Classification · CPC title

• G06V40/18 Primary

  Eye characteristics, e.g. of the iris · CPC title