Occupant monitoring device, occupant monitoring method, and occupant monitoring program

Patent metadata

Field	Value
Publication number	US-11501561-B2
Application number	US-201916580562-A
Country	US
Kind code	B2
Filing date	Sep 24, 2019
Priority date	Sep 27, 2018
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

An occupant monitoring device includes: an acquisition unit that acquires a captured image obtained by imaging a region in which there is a probability that a face of an occupant is present in a vehicle; a determination unit that determines whether the captured image acquired by the acquisition unit corresponds to a first image including the face a part of which is hidden by an accessory or a second image including the face a part of which is hidden by a non-accessory object other than the accessory; and a processing unit that detects face information regarding the face of the occupant based on the captured image in different modes according to a determination result in the determination unit, and monitors a state change of the face of the occupant based on a detection result.

First claim

What is claimed is:

1. An occupant monitoring device comprising:
a processor configured to implement:
an acquisition unit that acquires a captured image obtained by imaging a region in which there is a probability that a face of an occupant is present in a vehicle;
a determination unit that determines whether the captured image acquired by the acquisition unit corresponds to a first image including the face a part of which is hidden by an accessory or a second image including the face a part of which is hidden by a non-accessory object other than the accessory; and
a processing unit that detects face information regarding the face of the occupant based on the captured image in different modes according to a determination result in the determination unit, and monitors a state change of the face of the occupant based on a detection result,
wherein
in a case where the determination unit determines that the captured image corresponds to the first image, the processing unit detects the face information based on one or more feature points corresponding to a portion of the face not hidden by the accessory among a plurality of feature points set in advance for the whole face, and
in a case where the determination unit determines that the captured image corresponds to the second image, the processing unit detects the face information based on the plurality of feature points.

2. The occupant monitoring device according to claim 1 , wherein
in a case where the determination unit determines that the captured image corresponds to the second image, the processing unit detects separate face information according to a portion of the face hidden by the non-accessory object.

3. The occupant monitoring device according to claim 2 , wherein
the processing unit detects eye information regarding an eye as the face information in a case where the portion of the face hidden by the non-accessory object does not include the eye, and detects mouth information regarding a mouth as the face information in a case where the portion of the face hidden by the non-accessory object does not include the mouth.

4. The occupant monitoring device according to claim 3 , wherein
the processing unit detects expression information regarding an expression as the face information regardless of whether or not detection of the eye information and the mouth information is successful.

5. The occupant monitoring device according to claim 1 , wherein
in a case where the determination unit determines that the captured image corresponds to the first image, the processing unit detects the face information based on a result of fitting between a feature of the first image and a three-dimensional model representing a structure including a three-dimensional shape of the face of the occupant, and
in a case where the determination unit determines that the captured image corresponds to the second image, the processing unit detects the face information based on only a feature of the second image.

6. The occupant monitoring device according to claim 1 , wherein
the acquisition unit successively acquires the captured image a plurality of times,
in a case where the determination unit determines that the captured image corresponds to the first image, the processing unit detects the face information and then executes tracking of the face information, and
in a case where the determination unit determines that the captured image corresponds to the second image, the processing unit detects the face information and then detects the face information in different modes according to a determination result in the determination unit with respect to the captured image acquired next by the acquisition unit without executing tracking of the face information.

7. The occupant monitoring device according to claim 1 , wherein
the determination unit determines whether the captured image acquired by the acquisition unit corresponds to the first image or the second image based on a learned model generated by learning a learning image including information similar to the captured image and whether the learning image corresponds to the first image or the second image through machine learning.

8. An occupant monitoring method comprising:
an acquisition step of acquiring a captured image obtained by imaging a region in which there is a probability that a face of an occupant is present in a vehicle;
a determination step of determining whether the captured image acquired in the acquisition step corresponds to a first image including the face a part of which is hidden by an accessory or a second image including the face a part of which is hidden by a non-accessory object other than the accessory; and
a processing step of detecting face information regarding the face of the occupant based on the captured image in different modes according to a determination result in the determination step, and monitoring a state change of the face of the occupant based on a detection result,
wherein
in a case where the determination step determines that the captured image corresponds to the first image, the processing step detects the face information based on one or more feature points corresponding to a portion of the face not hidden by the accessory among a plurality of feature points set in advance for the whole face, and
in a case where the determination step determines that the captured image corresponds to the second image, the processing step detects the face information based on the plurality of feature points.

9. An occupant monitoring program stored on a non-transitory computer readable medium configured to cause a computer to execute:
an acquisition step of acquiring a captured image obtained by imaging a region in which there is a probability that a face of an occupant is present in a vehicle;
a determination step of determining whether the captured image acquired in the acquisition step corresponds to a first image including the face a part of which is hidden by an accessory or a second image including the face a part of which is hidden by a non-accessory object other than the accessory; and
a processing step of detecting face information regarding the face of the occupant based on the captured image in different modes according to a determination result in the determination step, and monitoring a state change of the face of the occupant based on a detection result,
wherein
in a case where the determination step determines that the captured image corresponds to the first image, the processing step detects the face information based on one or more feature points corresponding to a portion of the face not hidden by the accessory among a plurality of feature points set in advance for the whole face, and
in a case where the determination step determines that the captured image corresponds to the second image, the processing step detects the face information based on the plurality of feature points.

10. The occupant monitoring device according to claim 1 , wherein
in a case where the determination unit determines that the captured image corresponds to the second image, the processing unit detects separate face information according to a portion of the face hidden by the non-accessory object.

11. The occupant monitoring device according to claim 1 , wherein
in a case where the determination unit determines that the captured image corresponds to the first image, the processing unit detects the face information based on a result of fitting between a feature of the first image and a three-dimensional model representing a structure including a three-dimensional shape of the face of the occupant, and
in a case where the determination unit determines that the capture

Assignees

• Aisin Corp
• Toyota Motor Co Ltd

Inventors

• Osuga Shin
• Hattori Tetsuya
• Yamada Yoshiyuki
• Matsumura Takeshi

Classifications

• H04W4/48 Primary

  for in-vehicle communication · CPC title

• G06V40/165

  using facial parts and geometric relationships · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V40/166 Primary

  using acquisition arrangements · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title