Driver gaze detection system

Patent metadata

Field	Value
Publication number	US-9405982-B2
Application number	US-201314041016-A
Country	US
Kind code	B2
Filing date	Sep 30, 2013
Priority date	Jan 18, 2013
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

A method for detecting an eyes-off-the-road condition based on an estimated gaze direction of a driver of a vehicle includes monitoring facial feature points of the driver within image input data captured by an in-vehicle camera device. A location for each of a plurality of eye features for an eyeball of the driver is detected based on the monitored facial features. A head pose of the driver is estimated based on the monitored facial feature points. The gaze direction of the driver is estimated based on the detected location for each of the plurality of eye features and the estimated head pose.

First claim

The invention claimed is:

1. Method for detecting an eyes-off-the-road condition based on an estimated gaze direction of a driver of a vehicle, comprising:
monitoring facial feature points of the driver within image input data captured by an in-vehicle camera device;
detecting a location for each of a plurality of eye features for an eyeball of the driver based on the monitored facial feature points, comprising;
detecting a location of an iris;
detecting a location for a first eye corner indicative of an inner eye corner proximate to a nose bridge;
detecting a location for a second eye corner indicative of an outer eye corner;
for each corresponding eye feature of the plurality of eye features, training a classifier, comprising:
assigning image patches around a reference eye feature respective to the corresponding eye feature obtained from a database; and
identifying one or more of the assigned image patches that are centered around the reference eye feature;

estimating a head pose of the driver based on the monitored facial feature points; and
estimating the gaze direction of the driver based on the detected location information for each of the plurality of eye features and the estimated head pose.

2. The method of claim 1 , wherein the monitored facial feature points are extracted in accordance with the steps, comprising:
for a first image frame, detecting a face of the driver and extracting the facial feature points from the detected face;
for each consecutive image frame subsequent to the first image frame:
identifying a candidate region encompassing the facial feature points extracted from one or more previous image input frames; and
identifying the facial feature points only within the candidate region.

3. The method of claim 1 , further comprising:
for each corresponding eye feature within a first image frame:
selecting candidate pixels indicative of the corresponding eye feature using two statistical priors based on a magnitude of intensity and a detected edge strength;
calculating a confidence score for each candidate pixel based on a weighted sum of the two statistical priors;
selecting top candidate pixels each having a respective confidence score that is greater than a confidence score threshold;
comparing each of the top candidate pixels to the trained classifier to generate a classifier response respective to each of the top candidate pixels; and
detecting the location of the corresponding eye feature within the first image frame based on the generated classifier responses of the top candidate pixels.

4. The method of claim 3 , further comprising:
for each corresponding eye feature within each consecutive image frame subsequent to the first image frame:
identifying supplemental facial feature points surrounding the corresponding eye feature detected in an immediately preceding image frame;
tracking a location change for each of the supplemental facial feature points from the immediately preceding image frame to the corresponding image frame;
estimating an initial location of the corresponding eye feature in the corresponding image frame based on the tracked location change of the supporting feature points;
calculating the confidence score for each of a plurality of pixels within an area surrounding the estimated initial location of the corresponding eye feature;
selecting top candidate pixels within the area surrounding the estimated initial location, wherein each top candidate pixel has a respective confidence score that is greater than the confidence score threshold;
comparing the top candidate pixels to the trained classifier to generate the classifier response for each of the top candidate pixels; and
detecting the location of the corresponding eye feature within the corresponding image frame based on the top candidate pixel having the highest classifier response.

5. The method of claim 4 , wherein the plurality of pixels within the area surrounding the estimated initial location of the corresponding eye feature includes only pixels having detected edge responses exceeding a weak edge threshold.

6. The method of claim 1 , wherein estimating the head pose of the driver based on the monitored facial feature points, comprises:
in an output space, generating a plurality of uniformly-spaced yaw angles within a range of yaw angles;
in an input space:
generating a plurality of subspaces each associated with respective ones of the uniformly-spaced yaw angles and parameterized by a respective mean and basis,
identifying two candidate subspaces among the plurality of subspaces closest to the monitored facial feature points, and
selecting a neighboring subspace among the two candidate subspaces having a lowest reconstruction error associated with the monitored facial feature points; and

estimating the head pose of the driver based on the uniformly-spaced yaw angle in the output space associated with the selected neighboring sub-space in the input space.

7. The method of claim 6 , wherein estimating the head pose of the driver further comprises:
sampling a plurality of training images in the input space each associated a respective trained yaw angle, wherein the trained yaw angles are non-uniformly-spaced within the range of angles in the output space;
reconstructing each trained image and each associated trained yaw angle based on the generated plurality of uniformly-spaced yaw angles and the generated plurality of subspaces; and
estimating the head pose of the driver based on one of the reconstructed training images and the trained yaw angle associated therewith.

8. The method of claim 1 , further comprising:
prior to the monitoring facial feature points of the driver:
providing a detection region within the image input data encompassing a face location of the driver detected by a profile face detector;
using stored data of known face centers with respect to yaw angle, estimating a yaw angle of the detected face location of the driver within the detection region to generate a rectified face location; and

estimating the head pose of the driver within the image input data using the rectified face location.

9. The method of claim 1 , wherein estimating the gaze direction of the driver based on the detected location information for each of the plurality of eye features and the estimated head pose, comprises:
detecting a location for each of first and second eye corners of the eyeball;
calculating a midpoint between the detected location for each of the first and second eye corners of the eyeball;
calculating a center of the eye ball using the calculated midpoint and two corrections based on the estimated head pose;
calculating a scale of the face of the driver based on a distance between the detected first and second eye corners and the estimated head pose;
calculating a radius of the eye ball based on multiplying a normalized radius of the eye ball with the calculated scale of the face; and
estimating the gaze direction of the driver based on the calculated radius of the eye ball and the calculated center of the eye ball.

10. The method of claim 1 , further comprising:
identifying a gaze location of the driver corresponding to a point at which the estimated gaze direction intersects a windscreen plane of the vehicle;
comparing the identified gaze location of the driver to a road plane within the windscreen plane; and
detecting the eyes-off-the-road condition only when the identified gaze location is outside the road plane.

11. The method of claim 1 , wherein the in-vehicle camera device comprises a monocular camera device mounted proximate to a steering wheel on an interior of the vehicle, the monocular camera

Assignees

• Gm Global Tech Operations Llc
• Univ Carnegie Mellon

Inventors

• Zhang Wende
• Levi Dan
• Nachtegall Debbie E
• De La Torre Fernando
• Huang Dong

Classifications

• G06K9/0061

  Physics · mapped topic

• G06K9/00597

  Physics · mapped topic

• G06K9/00845 Primary

  Physics · mapped topic

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V20/597 Primary

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title