Driver attention determination using gaze detection

Patent metadata

Field	Value
Publication number	US-12258024-B2
Application number	US-202217822353-A
Country	US
Kind code	B2
Filing date	Aug 25, 2022
Priority date	Aug 25, 2022
Publication date	Mar 25, 2025
Grant date	Mar 25, 2025

Abstract

This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing for vehicular monitoring operations. In a first aspect, a method of monitoring includes receiving first image data from a first camera oriented in a first direction with a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a field of view of the user; determining a set of regions of interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining an attentiveness score based on correspondence between the set of regions of interest and the gaze direction. Other aspects and features are also claimed and described.

First claim

What is claimed is:

1. A method, comprising:
receiving first image data from a first camera oriented in a first direction with a first field of view facing a user;
receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a field of view of the user;
determining a set of regions of interest based on the second image data;
determining a gaze direction of the user based on the first image data; and
determining an attentiveness score based on correspondence between the set of regions of interest and the gaze direction, the attentiveness score indicating whether the gaze direction of the user corresponds to the set of regions of interest for at least a threshold amount of
time, wherein determining the attentiveness score further comprises determining missed regi ons of interest among the set of regions of interest, each of the missed regions of interest indi cating a region of interest to which the gaze direction of the user does not correspond for at least a threshold amount of time; and
performing an action to alert the user in response to the missed regions of interest being higher than a threshold number.

2. The method of claim 1 , wherein the first image data comprises a first plurality of image frames captured over a duration of time, and wherein the second image data comprises a second plurality of image frames captured over the duration of time.

3. The method of claim 2 , wherein determining the attentiveness score comprises determining a fraction of the duration of time that the gaze direction of the user is correlated with the set of regions of interest.

4. The method of claim 1 , wherein determining the set of regions of interest comprises:
determining a risk map based on the second image data, wherein the set of regions of interest corresponds to portions of the risk map above a threshold risk level.

5. The method of claim 4 , wherein determining the risk map comprises determining the risk map using a machine learning (ML) algorithm.

6. The method of claim 1 , wherein determining the set of regions of interest comprises:
determining an object of interest in the second image data based on object detection, wherein the region of interest corresponds to the object of interest.

7. The method of claim 1 , further comprising:
determining whether the attentiveness score meets a first set of criteria; and
performing an action to alert the user when the attentiveness score meets the first set of criteria.

8. The method of claim 7 , wherein determining whether the attentiveness score meets a first set of criteria comprises determining whether the attentiveness score is a predetermined threshold amount below a historical attentiveness score corresponding to the user.

9. The method of claim 7 , wherein performing the action to alert the user comprises displaying the set of regions of interest on a heads-up display (HUD).

10. The method of claim 1 , further comprising transmitting the attentiveness score through a wireless communication link.

11. An apparatus, comprising:
a memory storing processor-readable code; and
at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
receiving first image data from a first camera oriented in a first direction with a first field of view facing a user;
receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a field of view of the user;
determining a set of regions of interest based on the second image data;
determining a gaze direction of the user based on the first image data; and
determining an attentiveness score based on correspondence between the set of regions of interest and the gaze direction, the attentiveness score indicating whether the gaze direction of the user corresponds to the set of regions of interest for at least a threshold amount of time, wherein determining the attentiveness score further comprises determining missed regions of interest among the set of regions of interest, each of the missed regions of interest indicating a region of interest to which the gaze direction of the user does not correspond for at least a threshold amount of time; and
performing an action to alert the user in response to the missed regions of interest being higher than a threshold number.

12. The apparatus of claim 11 , wherein the first image data comprises a first plurality of image frames captured over a duration of time, and wherein the second image data comprises a second plurality of image frames captured over the duration of time.

13. The apparatus of claim 12 , wherein determining the attentiveness score comprises determining a fraction of the duration of time that the gaze direction of the user is correlated with the set of regions of interest.

14. The apparatus of claim 11 , wherein determining the set of regions of interest comprises:
determining a risk map based on the second image data, wherein the set of regions of interest corresponds to portions of the risk map above a threshold risk level.

15. The apparatus of claim 14 , wherein determining the risk map comprises determining the risk map using a machine learning (ML) algorithm.

16. The apparatus of claim 11 , wherein determining the set of regions of interest comprises:
determining an object of interest in the second image data based on object detection, wherein the region of interest corresponds to the object of interest.

17. The apparatus of claim 11 , wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including:
determining whether the attentiveness score meets a first set of criteria; and
when the attentiveness score meets the first set of criteria, performing an action to alert the user.

18. The apparatus of claim 17 , wherein determining whether the attentiveness score meets a first set of criteria comprises determining whether the attentiveness score is a predetermined threshold amount below a historical attentiveness score corresponding to the user.

19. The apparatus of claim 17 , wherein performing the action to alert the user comprises displaying the set of regions of interest on a heads-up display (HUD).

20. The apparatus of claim 11 , wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including:
transmitting the attentiveness score through a wireless communication link.

21. A vehicular apparatus, comprising: a vehicle frame; a first camera attached to the vehicle frame and arranged to capture a first field of view comprising a portion of a cabin enclosed on at least some sides by the vehicle frame; a second camera attached to the vehicle frame and arranged to capture a second field of view corresponding to at least a portion of surroundings of the vehicle frame; and a processing system coupled to the first camera and to the second camera, wherein the processing system comprises: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor

Assignees

• Arriver Software Ab
• Qualcomm Inc

Inventors

• Persson Gustav Nils Ture
• Cronvall Per

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W2050/146

  Display means · CPC title

• B60W50/14

  Means for informing the driver, warning the driver or prompting a driver intervention · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V20/597 Primary

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title