Closed-loop real time SSVEP-based heads-up display to control in vehicle features using deep learning

Patent metadata

Field	Value
Publication number	US-12358370-B2
Application number	US-202218067473-A
Country	US
Kind code	B2
Filing date	Dec 16, 2022
Priority date	Dec 16, 2022
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

A vehicle system includes a controller programmed to display a plurality of icons on a heads-up-display (HUD) of the vehicle, receive electroencephalography (EEG) data from a driver of the vehicle, perform a Fast Fourier Transform of the EEG data to obtain an EEG spectrum, input the EEG spectrum into a trained machine learning model, determine which of the plurality of icons the driver is viewing based on an output of the trained machine learning model, and perform one or more vehicle operations based on the output of the trained machine learning model.

First claim

What is claimed is:

1. A vehicle system, comprising a controller programmed to:
display a plurality of icons on a heads-up-display (HUD) of a vehicle;
receive electroencephalography (EEG) data from a driver of the vehicle;
perform a Fast Fourier Transform (FFT) of the EEG data to obtain an EEG spectrum;
input the EEG spectrum into a trained machine learning model that outputs a prediction of an icon that the driver is looking at based on the EEG spectrum;
determine which of the plurality of icons the driver is viewing based on the output of the trained machine learning model; and
perform one or more vehicle operations based on the output of the trained machine learning model.

2. The vehicle system of claim 1 , wherein each of the plurality of icons has a different color.

3. The vehicle system of claim 1 , wherein each of the plurality of icons has a different shape.

4. The vehicle system of claim 1 , wherein the controller is further programmed to:
apply a band pass filter to the EEG data to obtain filtered EEG data; and
perform the FFT of the filtered EEG data.

5. The vehicle system of claim 1 , wherein the controller is further programmed to:
perform a data segmentation of the EEG data to obtain segmented EEG data; and
perform the FFT of the segmented EEG data.

6. The vehicle system of claim 1 , wherein the controller is further programmed to:
receive training data comprising EEG data collected from a plurality of individual subjects while each subject is viewing specific icons; and
train a machine learning model to predict which icon the individual subjects are viewing based on the training data to achieve the trained machine learning model.

7. The vehicle system of claim 1 , wherein the trained machine learning model comprises a convolutional neural network.

8. The vehicle system of claim 7 , wherein the convolutional neural network comprises a residual neural network architecture.

9. The vehicle system of claim 1 , wherein the trained machine learning model comprises one or more squeeze and excite (SE) blocks.

10. The vehicle system of claim 9 , wherein at least one of the SE blocks comprises a global max pooling layer, a first fully connected layer having a rectified linear unit activation function, and a second fully connected layer having a sigmoid activation function.

11. The vehicle system of claim 1 , wherein the trained machine learning model comprises two SE-Res blocks, wherein each SE-Res block comprises:
a two-dimensional convolutional layer;
a batch normalization layer;
an activation layer; and
an SE block.

12. The vehicle system of claim 11 , wherein an input to each SE-Res block is summed with an output of the SE-Res block.

13. The vehicle system of claim 11 , wherein the trained machine learning model further comprises:
a dropout layer; and
a Softmax classification layer.

14. A method, comprising:
displaying a plurality of icons on a heads-up-display (HUD) of a vehicle;
receiving electroencephalography (EEG) data from a driver of the vehicle;
performing a Fast Fourier Transform (FFT) of the EEG data to obtain an EEG spectrum;
inputting the EEG spectrum into a trained machine learning model that outputs a prediction of an icon that the driver is looking at based on the EEG spectrum;
determining which of the plurality of icons the driver is viewing based on the output of the trained machine learning model; and
performing one or more vehicle operations based on the output of the trained machine learning model.

15. The method of claim 14 , further comprising:
apply a band pass filter to the EEG data to obtain filtered EEG data;
performing a data segmentation of the filtered EEG data to obtain segmented EEG data; and
performing the FFT of the segmented EEG data.

16. The method of claim 14 , wherein the trained machine learning model comprises a convolutional neural network comprising:
two SE-Res blocks, wherein each SE-Res block comprises:
a two-dimensional convolutional layer;
a batch normalization layer;
an activation layer; and
an SE block.

17. The method of claim 16 , wherein the SE block comprises a global max pooling layer, a first fully connected layer having a rectified linear unit activation function, and a second fully connected layer having a sigmoid activation function.

18. The method of claim 16 , wherein the trained machine learning model further comprises:
a dropout layer; and
a Softmax classification layer.

19. A method, comprising:
receiving training data comprising EEG data collected from a plurality of individual subjects while each subject is viewing specific icons;
performing an FFT of the training data to obtain EEG spectrum data; and
training a machine learning model to predict which icon the individual subjects are viewing based on the EEG spectrum data.

20. The method of claim 19 , wherein the machine learning model comprises:
two SE-Res blocks, wherein each SE-Res block comprises:
a two-dimensional convolutional layer;
a batch normalization layer;
an activation layer;
an SE block; and
wherein the SE block comprises a global max pooling layer, a first fully connected layer having a rectified linear unit activation function, and a second fully connected layer having a sigmoid activation function.

Assignees

• Toyota Eng & Mfg North America
• Univ Of Connecticut Health Center

Inventors

• Shishavan Hossein Hamidi
• Kim Insoo
• Behzadi Mohammad
• Dede Ercan
• Lohan Danny

Classifications

• B60K35/00 Primary

  Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles · CPC title

• B60K2360/188

  Displaying information using colour changes · CPC title

• B60K2360/149

  Instrument input by detecting viewing direction not otherwise provided for · CPC title

• B60K2360/119

  Icons · CPC title

• B60K35/23

  Head-up displays [HUD] (optical aspects of head-up displays G02B27/01) · CPC title