Eye gaze tracking utilizing surface normal identification

Patent metadata

Field	Value
Publication number	US-10088899-B2
Application number	US-201615365421-A
Country	US
Kind code	B2
Filing date	Nov 30, 2016
Priority date	May 20, 2014
Publication date	Oct 2, 2018
Grant date	Oct 2, 2018

Abstract

A gaze tracking system captures images of a vehicle operator. The gaze tracking system may detect facial features in the images and track the position of the facial features over time. The gaze tracking system may detect a triangle in an image, wherein the vertices of the triangle correspond to the facial features. The gaze tracking system may analyze the detected triangle to identify a surface normal for the triangle, and may track the surface normal (e.g., across multiple images) to track the eye gaze direction of the driver over time. The images may be captured and analyzed in near-real time. By tracking movement of the driver's head and eyes over time, the gaze analysis system may predict or estimate head position and/or gaze direction when one or more facial features are not detectable.

First claim

What is claimed:

1. A method comprising:
capturing, via an image sensor, a sequence of images of a driver of a vehicle over a time period;
in each of the sequence of images, detecting a triangle in three-dimensional (3D) space representing a relationship between three facial features including a first pupil, a second pupil, and a third feature of the driver depicted in the sequence of images, the triangle having vertices corresponding to the first pupil, the second pupil, and the third feature, wherein the third feature is selected from a group consisting of a nose, a mouth, and a chin;
for each of the sequence of images, analyzing the detected triangle in 3D space to identify a plane of the triangle in 3D space;
for each of the sequence of images, analyzing the identified plane of the triangle in 3D space to identify a surface normal for the identified plane in 3D space; and
tracking a change in an eye gaze direction of the driver over the time period by tracking a change in the identified surface normal for the plane in 3D space throughout the sequence of images, wherein the change in the eye gaze direction can be tracked regardless of whether the driver's head pose changes during the time period.

2. The method of claim 1 , further comprising:
calibrating the image sensor.

3. The method of claim 2 , wherein calibrating the image sensor comprises:
adjusting the image sensor to account for lighting conditions in the vehicle.

4. The method of claim 2 , wherein calibrating the image sensor comprises:
capturing a reference image of the driver gazing in a known direction with a known head pose; and
analyzing the reference image to detect the three facial features in the reference image;
wherein detecting the triangle in 3D space in each of the sequence of images includes: identifying, in each of the sequence of images, the three facial features detected in the reference image.

5. The method of claim 4 , wherein calibrating the image sensor further comprises:
displaying audio or visual instructions to instruct the driver to gaze in the known direction and to assume the known head pose.

6. The method of claim 4 , wherein calibrating the image sensor further comprises:
estimating that the driver is gazing in the known direction with the known head pose based on an analysis of a distribution of the facial features in one or more previously captured images of the driver;
wherein analyzing the reference image to detect the three facial features in the reference image occurs in response to estimating that the driver is gazing in the known direction with the known head pose.

7. The method of claim 1 , further comprising:
generating a report identifying one or more areas to which the tracked eye gaze direction is directed.

8. The method of claim 1 , further comprising:
generating an alarm when the tracked eye gaze direction is outside of a predefined range of safe gaze directions.

9. A system comprising:
an image sensor that captures a sequence of images of a driver of a vehicle over a time period;
one or more processors that are communicatively connected to the image sensor and that:
(i) detect, in each of the sequence of images captured by the image sensor, a triangle in three-dimensional (3D) space representing a relationship between three facial features including a first pupil, a second pupil, and a third feature of the driver of the vehicle depicted in the sequence of images, the triangle having vertices corresponding to the first pupil, the second pupil, and the third feature, wherein the third feature is selected from a group consisting of a nose, a mouth, and a chin;
(ii) for each of the sequence of images, analyze the detected triangle in 3D space to identify a plane of the triangle in 3D space;
(iii) for each of the sequence of images, analyze the identified plane of the triangle in 3D space to identify a surface normal for the identified plane in 3D space; and
(iv) track a change in an eye gaze direction of the driver over the time period by tracking a change in the identified surface normal for the plane in 3D space throughout the sequence of images, wherein the change in the eye gaze direction can be tracked regardless of whether the driver's head pose changes during the time period.

10. The system of claim 9 , wherein the image sensor is embedded in a dash or a console of the vehicle.

11. The system of claim 9 , wherein the one or more memory devices further comprise instructions to perform a calibration.

12. The system of claim 11 , wherein the instructions to perform the calibration comprise instructions to adjust the image sensor to account for lighting conditions in the vehicle.

13. The system of claim 11 , wherein the instructions to perform the calibration comprise:
instructions to cause the image sensor to capture a reference image of the driver gazing in a known direction with a known head pose; and
instructions to analyze the reference image to detect the three facial features in the reference image;
wherein the instructions to detect the triangle in 3D space includes: instructions to identify, in each of the sequence of images, the three facial features detected in the reference image.

14. The system of claim 13 , wherein the instructions to perform the calibration further comprise:
instructions display audio or visual instructions to instruct the driver to gaze in the known direction and to assume the known head pose.

15. The system of claim 13 , wherein the instructions to perform the calibration further comprise:
instructions to estimate that the driver is gazing in the known direction with the known head pose based on an analysis of a distribution of the facial features in one or more previously captured images of the driver;
wherein the instructions to analyze the reference image to detect the three facial features in the reference image occurs in response to estimating that the driver is gazing in the known direction with the known head pose.

16. The system of claim 9 , further comprising a speaker or display communicatively coupled to the one or more processors;
wherein the one or more processors generate an alarm via the speaker or display when the tracked eye gaze direction is outside of a predefined range of safe gaze directions.

17. A system comprising:
a means for capturing a sequence of images of a driver of a vehicle over a time period;
a means for detecting, in each of the sequence of images, a triangle in three-dimensional (3D) space representing a relationship between three facial features including a first pupil, a second pupil, and a third feature of the driver depicted in the sequence of images, the triangle having vertices corresponding to the first pupil, the second pupil, and the third feature, wherein the third feature is selected from a group consisting a nose, a mouth, and a chin;
a means for analyzing, for each of the sequence of images, the detected triangle in 3D space to identify a plane of the triangle in 3D space;
a means for analyzing, for each of the sequence of images, the identified plane of the triangle in 3D space to identify a surface normal for the identified plane in 3D space; and
a means for tracking a change in an eye gaze direction of the driver over the time period by tracking a change in the identified surface normal for the plane in 3D space throughout the sequence of images, wherein the change in the eye gaze direction can be tracked regardless of whether the driver's head pose changes during the time period.

18. The system of claim 17 , further com

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Plummer Bryan Allen
• Brown Parker Andrew
• Fields Brian Mark
• Roberson Steve

Classifications

• G06K9/00597

  Physics · mapped topic

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06K9/00268

  Physics · mapped topic

• G06K9/00845

  Physics · mapped topic

• G06V20/597 Primary

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title