Cognitive heat map: a model for driver situational awareness

Patent metadata

Field	Value
Publication number	US-11999356-B2
Application number	US-202117351611-A
Country	US
Kind code	B2
Filing date	Jun 18, 2021
Priority date	Nov 13, 2020
Publication date	Jun 4, 2024
Grant date	Jun 4, 2024

Abstract

A system includes a camera configured to capture image data of an environment, a monitoring system configured to generate a gaze sequences of a subject, and a computing device communicatively coupled to the camera and the monitoring system. The computing device is configured to receive the image data from the camera and the gaze sequences from the monitoring system, implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network, generate, with the machine learning model, a gaze probability density heat map, and generate, with the machine learning model, an attended awareness heat map.

First claim

The invention claimed is:

1. A system, comprising:
a camera configured to capture image data of an environment;
a monitoring system configured to generate a gaze sequences of a subject; and
a computing device communicatively coupled to the camera and the monitoring system, the computing device configured to:
receive the image data from the camera and the gaze sequences from the monitoring system,
implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network,
generate, with the machine learning model, a gaze probability density heat map, and
generate, with the machine learning model, an attended awareness heat map.

2. The system of claim 1 , wherein the machine learning model is configured to determine visual saliency of the environment from the image data and fuse the visual saliency of the environment with the gaze sequences to generate the gaze probability density heat map.

3. The system of claim 2 , wherein fusing the visual saliency of the environment with the gaze sequences reduces noise in the gaze sequences generated by the monitoring system such that the gaze probability density heat map provides a more accurate estimation of a gaze of the subject than the gaze sequences generated by the monitoring system.

4. The system of claim 1 , wherein the attended awareness heat map estimates levels of awareness of the subject corresponding to locations within the environment.

5. The system of claim 1 , wherein:
the image data received by the computing device is a video sequence spanning a period of time, and
the gaze sequences received by the computing device span the period of time.

6. The system of claim 5 , wherein the gaze probability density heat map provides an estimate of a gaze of the subject over the period of time.

7. The system of claim 5 , wherein the attended awareness heat map provides an overall level of awareness to locations within the environment at an end of the period of time.

8. The system of claim 1 , wherein the computing device is further configured to:
generate a visual saliency heat map of the environment, with the machine learning model, and
determine whether the subject is aware of salient regions in the environment defined by the visual saliency heat map based on a comparison of the gaze probability density heat map and the visual saliency heat map.

9. The system of claim 1 , wherein the machine learning model further comprises a first convolutional model and a second convolutional model configured to receive a latent map generated by the convolutional encoder-decoder neural network,
the first convolutional model generates the gaze probability density heat map, and
the second convolutional model generates the attended awareness heat map.

10. The system of claim 9 , wherein the second convolutional model includes a decay factor causing estimated levels of awareness defined in the attended awareness heat map to decay over time.

11. The system of claim 1 , wherein:
the decoder stage comprises a plurality of decoder units,
at least one decoder unit of the plurality of decoder units is configured to receive input from the side-channel corresponding to the at least one decoder unit and an output from a previous decoder unit of the plurality of decoder units, and
the at least one decoder unit is configured to concatenate the side-channel corresponding to the at least one decoder unit with the output from the previous decoder unit.

12. A method, comprising:
receiving image data of an environment from a camera and gaze sequences of a subject from a monitoring system;
implementing, with a computing device, a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network;
generating, with the machine learning model, a gaze probability density heat map; and
generating, with the machine learning model, an attended awareness heat map.

13. The method of claim 12 , further comprising:
determining, with the machine learning model, visual saliency of the environment from the image data; and
fusing the visual saliency of the environment with the gaze sequences to generate the gaze probability density heat map.

14. The method of claim 13 , wherein fusing the visual saliency of the environment with the gaze sequences reduces noise in the gaze sequences generated by the monitoring system such that the gaze probability density heat map provides a more accurate estimation of a gaze of the subject than the gaze sequences generated by the monitoring system.

15. The method of claim 12 , wherein the attended awareness heat map estimates levels of awareness of the subject corresponding to locations within the environment.

16. The method of claim 12 , wherein:
the image data received by the computing device is a video sequence spanning a period of time, and
the gaze sequences received by the computing device span the period of time.

17. The method of claim 16 , wherein the gaze probability density heat map provides an estimate of a gaze of the subject over the period of time and the attended awareness heat map provides an overall level of awareness to locations within the environment at an end of the period of time.

18. The method of claim 12 , further comprising:
generating a visual saliency heat map of the environment, with the machine learning model, and
determining whether the subject is aware of salient regions in the environment defined by the visual saliency heat map based on a comparison of the gaze probability density heat map and the visual saliency heat map.

19. The method of claim 12 , wherein the machine learning model further comprises a first convolutional model and a second convolutional model configured to receive a latent map generated by the convolutional encoder-decoder neural network,
the first convolutional model generates the gaze probability density heat map, and
the second convolutional model generates the attended awareness heat map.

20. A vehicle, comprising:
a camera configured to capture image data of an environment around the vehicle;
a monitoring system configured to generate a gaze sequences of a driver; and
a computing device communicatively coupled to the camera and the monitoring system, the computing device configured to:
receive the image data from the camera and the gaze sequences from the monitoring system,
implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network,
generate, with the machine learning model, a gaze probability density heat map, and
generate, with the machine learning model, an attended awareness heat map.

Assignees

• Toyota Res Inst Inc

Inventors

• Rosman Guy
• Stent Simon A I
• Fletcher Luke
• Leonard John
• Gopinath Deepak
• Terahata Katsuya

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• B60W40/08 Primary

  related to drivers or passengers · CPC title

• G06N3/045

  Combinations of networks · CPC title