Vehicle occupant engagement using three-dimensional eye gaze vectors

Patent metadata

Field	Value
Publication number	US-11847858-B2
Application number	US-202218054102-A
Country	US
Kind code	B2
Filing date	Nov 9, 2022
Priority date	Jun 17, 2019
Publication date	Dec 19, 2023
Grant date	Dec 19, 2023

Abstract

According to the techniques of this disclosure, a method includes capturing, using a camera system of a vehicle, at least one image of an occupant of the vehicle, determining, based on the at least one image of the occupant, a location of one or more eyes of the occupant within the vehicle, and determining, based on the at least one image of the occupant, an eye gaze vector. The method may also include determining, based on the eye gaze vector, the location of the one or more eyes of the occupant, and a vehicle data file of the vehicle, a region of interest from a plurality of regions of interests of the vehicle at which the occupant is looking, wherein the vehicle data file specifies respective locations of each of the plurality of regions of interest, and selectively performing, based on the region of interest, an action.

First claim

What is claimed is:

1. A method comprising:
obtaining, via a camera system of a vehicle, at least one image of a driver of the vehicle;
determining, based on the at least one image of the driver, a location of one or more eyes of the driver within the vehicle;
determining, based on the at least one image of the driver, an eye gaze vector;
determining, based on the eye gaze vector, the location of the one or more eyes of the driver, and a vehicle data file of the vehicle, a region of interest in an interior of the vehicle from a plurality of regions of interests in the interior of the vehicle, wherein the vehicle data file specifies respective locations of each of the plurality of regions of interest in the interior of the vehicle, wherein the vehicle data file includes a respective set of coordinates for each region of interest from the plurality of regions of interest, and wherein each of the respective coordinate sets are defined relative to a centroid of a sphere that encompasses the interior of the vehicle, and wherein each of the respective sets of coordinate define a two-dimensional plane; and
responsive to determining that the region of interest is associated with a low level of driver engagement, outputting a warning message reminding the driver to pay attention to the road.

2. The method of claim 1 , wherein determining the eye gaze vector comprises:
identifying one or more facial landmarks in the at least one image;
determining, based on the one or more facial landmarks, a pitch angle, a roll angle, and a yaw angle of a facial plane of the driver; and
determining, based on the pitch angle, the roll angle, and the yaw angle, the eye gaze vector.

3. The method of claim 1 , wherein determining the eye gaze vector comprises:
determining, based on the at least one image, an angle of at least one pupil the driver; and
determining, based on the angle of the at least one pupil, the eye gaze vector.

4. The method of claim 1 , wherein determining the eye gaze vector comprises:
determining, based on the at least one image, a facial plane of the driver;
determining, based the facial plane, a first initial eye gaze vector;
determining, based on the image and using eye tracking, a second initial eye gaze vector; and
determining the eye gaze vector by at least combining the first initial eye gaze vector and the second initial eye gaze vector.

5. The method of claim 1 , wherein determining the eye gaze vector comprises:
applying at least one machine-learned model to the at least one image, wherein the machine-learned model outputs the eye gaze vector.

6. The method of claim 1 , wherein the at least one image comprises at least one respective image captured by each of two or more different cameras of the camera system, and wherein determining the location of the one or more eyes of the driver within the vehicle comprises:
determining, based on the at least one respective image captured by each of the two or more different cameras, a parallax angle;
determining, based on respective locations of each of the two or more different cameras and the parallax angle, a distance from at least one of the two or more different cameras to the one or more eyes of the driver; and
determining, based on the distance and the respective locations of each of the two or more different cameras, the location of the one or more eyes of the driver.

7. The method of claim 1 , wherein the at least one image comprises an image captured using an infrared camera of the camera system, and wherein determining the location of the one or more eyes of the driver within the vehicle comprises:
determining, based on distortion of the image, a distance from the infrared camera to the one or more eyes of the driver; and
determining, based on the location of the infrared camera and the distance, the location of the one or more eyes of the driver.

8. The method of claim 1 ,
wherein the location of the one or more eyes of the driver within the vehicle is specified using a camera-based coordinate system having one camera of the camera system as a centroid,
wherein the respective locations of each of the plurality of regions of interest are specified using a vehicle-based coordinate system having a centroid located in an interior of the vehicle and is different from the location of the one camera, and
wherein determining the region of interest at which the driver is looking comprises:
transforming the location of the one or more eyes from the camera-based coordinate system to the vehicle-based coordinate system;
determining whether a projection of the eye gaze vector from the location of the one or more eyes specified using the vehicle-based coordinate system intersects with any of the plurality of regions of interest; and
responsive to determining that the eye gaze vector intersects a particular region of interest from the plurality of regions of interest, determining that the particular region of interest is the region of interest at which the driver is looking.

9. The method of claim 1 , wherein the vehicle data file includes data structured in accordance with extensible markup language.

10. A computing system comprising:
at least one processor;
a camera system; and
memory comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
obtain, via the camera system, at least one image of a driver of a vehicle;
determine, based on the at least one image of the driver, a location of one or more eyes of the driver within the vehicle;
determine, based on the at least one image of the driver, an eye gaze vector;
determine, based on the eye gaze vector, the location of the one or more eyes of the driver, and a vehicle data file of the vehicle, a region of interest in an interior of the vehicle from a plurality of regions of interests in the interior of the vehicle, wherein the vehicle data file specifies respective locations of each of the plurality of regions of interest in the interior of the vehicle, wherein the vehicle data file includes a respective set of coordinates for each region of interest from the plurality of regions of interest, and wherein each of the respective coordinate sets are defined relative to a centroid of a sphere that encompasses the interior of the vehicle, and wherein each of the respective sets of coordinate define a two-dimensional plane; and
output a warning message reminding the driver to pay attention to the road in response to a determination that the region of interest is associated with a low level of driver engagement.

11. The computing system of claim 10 , wherein the instructions are executable by the at least one processor to determine the eye gaze vector by at least being executable to:
identify one or more facial landmarks in the at least one image;
determine, based on the one or more facial landmarks, a pitch angle, a roll angle, and a yaw angle of a facial plane of the driver; and
determine, based on the pitch angle, the roll angle, and the yaw angle, the eye gaze vector.

12. The computing system of claim 10 , wherein the instructions are executable by the at least one processor to determine the eye gaze vector by at least being executable to:
determine, based on the at least one image, an angle of at least one pupil the driver; and
determine, based on the angle of the at least one pupil, the eye gaze vector.

13. The computing system of claim 10 , wherein the instructions are executable by the at least one processor to determine the eye gaze vector by at least being executable to:
determine, based on the at least one image, a facial plane of the driver;
determ

Assignees

• Google Llc

Inventors

• Arora Ankit
• Keller Blaze Michael
• Nagarajan Guru

Classifications

• G06V40/19 Primary

  Sensors therefor · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V20/597

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title