Gaze tracking for a vehicle operator

Patent metadata

Field	Value
Publication number	US-9547798-B2
Application number	US-201414282437-A
Country	US
Kind code	B2
Filing date	May 20, 2014
Priority date	May 20, 2014
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

A gaze tracking system captures images of a vehicle operator's face using an image sensor device positioned within a vehicle. The gaze tracking system may be detect facial features from each image and track the position of facial features over time. The gaze tracking system may use the detected facial features to track the vehicle operator's head pose and gaze direction. The images may be captured and analyzed in near-real time. By tracking movement of the driver's head and eyes over time, the gaze analysis system may predict or estimate head position and/or gaze direction when one or more facial features are not detectable. The gaze tracking system may generate a report regarding the vehicle operator's gaze distribution. The report may be saved to memory and may be graphically displayed at a screen.

First claim

What is claimed is:

1. A gaze tracking system, the gaze tracking system comprising:
an image sensor device positioned on or within a vehicle such that the image sensor device may capture an image of an operator interface area in the vehicle;
one or more processors communicatively coupled to the image sensor device; and
one or more memory devices, communicatively coupled to the one or more processors, including instructions that when executed cause the one or more processors to (i) perform a calibration and (ii) perform gaze tracking;
the instructions to perform a calibration including instructions that when executed cause the one or more processors to:
(a) transmit a signal to cause the image sensor device to capture a first image of an operator interface area in the vehicle,
(b) receive the first image from the image sensor device,
(c) detect in the first image three facial features of a person, the three facial features including two-eye pupils,
(d) identify a first triangle including vertices positioned according to locations of the three facial features in the first image, and
(e) associate the first triangle with a first gaze direction;

the instructions to perform gaze tracking including instructions that when executed cause the one or more processors to:
(a) transmit a signal to cause the image sensor device to capture a second image of an operator interface area in the vehicle,
(b) receive the second image from the image sensor device,
(c) detect the three facial features in the second image,
(d) identify a second triangle including vertices positioned according to locations of the three facial features in the second image,
(e) determine a transformation from the first triangle to the second triangle,
(f) determine a second gaze direction based, at least in part, on the determined transformation from the first triangle to the second triangle by identifying a surface normal for the second triangle, wherein a change from the first gaze direction to the second gaze direction can be tracked regardless of whether the person's head pose changed from the first image to the second image, and
(g) store, at the one or more memory devices, report data including data representing the second gaze direction.

2. The gaze tracking system of claim 1 , further including a speaker or display.

3. The gaze tracking system of claim 1 , wherein the image sensor device is embedded in a dash or console of the vehicle.

4. The gaze tracking system of claim 2 , wherein the instructions to perform a calibration further include instructions to:
cause the one or more processors to provide, via the speaker or display, directions to a vehicle operator to (i) assume a particular head pose, or (ii) gaze in a particular direction.

5. The gaze tracking system of claim 2 , wherein the one or more memory devices further include instructions to:
cause the one or more processors to provide, via the speaker or display, a report based at least in part on the report data stored at the one or more memory devices;
wherein the report includes statistics regarding time spent gazing in the direction of one or more areas.

6. The gaze tracking system of claim 5 , wherein the report is provided in near real-time as a vehicle operator operators the vehicle.

7. The gaze tracking system of claim 2 , wherein the one or more memory devices further include instructions to cause the one or more processors to provide, via the speaker or display, an alarm when the second gaze direction is outside of a predefined range of safe gaze directions.

8. The gaze tracking system of claim 1 , wherein the image sensor device is configured to capture a stream of images at a rate of at least ten images per second, and
wherein the instructions to perform gaze tracking further include instructions to cause the one or more processors to perform gaze tracking in near real-time, wherein a gaze direction is determined for each of at least ten images every second.

9. The gaze tracking system of claim 1 , wherein the instructions to perform calibration further include instructions to cause the one or more processors to perform calibration automatically while the vehicle operator operates the vehicle.

10. The gaze tracking system of claim 1 , wherein the one or more processors and the one or more memory devices are part of an on-board system for the vehicle.

11. The gaze tracking system of claim 10 , further including a display embedded in a dash or console of the vehicle, wherein the one or more memory devices further include instructions to provide a report via the embedded display.

12. The gaze tracking system of claim 1 , wherein the one or more processors are communicatively coupled to the image sensor device via at least one wireless link.

13. The gaze tracking system of claim 12 , wherein the one or more processors and the one or more memory devices are part of a mobile device.

14. The gaze tracking system of claim 12 , wherein the one or more processors and the one or more memory devices are part of a stationary server system.

15. A computer-implemented method for operating a gaze tracking system, the method comprising:
causing one or more processors of a gaze tracking system to perform a calibration, wherein causing one or more processors of the gaze tracking system to perform the calibration includes causing one or more processors to:
receive a first image of an operator interface area within the vehicle,
detect in the first image three facial features of a person, the three facial features including two eye pupils,
identify a first triangle including vertices positioned according to locations of the three facial features in the first image, and
associate the first triangle with a first gaze direction; and
causing one or more processors of the gaze tracking system to perform gaze tracking, wherein causing one or more processors of the gaze tracking system to perform gaze tracking includes causing one or more processors of the gaze tracking system to:
receive a second image of the operator interface area within the vehicle,
detect the three facial features in the second image,
identify a second triangle including vertices positioned according to locations of the three facial features in the second image,
determine a transformation from the first triangle to the second triangle,
determine a second gaze direction based, at least in part, on the determined transformation from the first triangle to the second triangle by identifying a surface normal for the second triangle, wherein a change from the first gaze direction to the second gaze direction can be tracked between regardless of whether the person's head pose changed from the first image to the second image, and
store, at one or more memory devices, report data including data representing the second gaze direction.

16. The computer-implemented method of claim 15 , further comprising:
causing one or more processors of the gaze tracking system to receive a series of images of the operator interface area within the vehicle;
causing one or more processors of the gaze tracking system to detect the three facial features in each of the series of images;
causing one or more processors of the gaze tracking system to identify a triangle for each of the series of images, each triangle including vertices positioned according to locations of the three facial features detected in the corresponding image;
for each triangle, causing one or more processors of the gaze tracking system to determine a transformation relative to the first triangle;
for each determined transformat

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Plummer Bryan Allen
• Brown Parker Andrew
• Fields Brian Mark
• Roberson Steve

Classifications

• G06K9/00597

  Physics · mapped topic

• G06K9/00845 Primary

  Physics · mapped topic

• G06V20/597 Primary

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title