Predictive impairment monitor system and method

Patent metadata

Field	Value
Publication number	US-11518408-B2
Application number	US-202017097580-A
Country	US
Kind code	B2
Filing date	Nov 13, 2020
Priority date	Nov 13, 2020
Publication date	Dec 6, 2022
Grant date	Dec 6, 2022

Abstract

A driver monitor system and method for predicting impairment of a user of a vehicle. The system includes video cameras, an input device for inputting a list of medications being taken by the driver. Processing circuitry predicts side effects of the medications based on the half-life of the medication, detecting eye gaze movement, eye lid position, and facial expression of the user using images from the video camera, predicting whether the user is transitioning into an impaired physical state that is a side effect of the medications, verifying the side effect of the medications, determining whether the user is fit to drive using the verified side effects of the medications, and outputting to the vehicle an instruction to operate the vehicle in a level of automation that makes up for the at least one side effect or to perform a safe pull over operation of the vehicle.

First claim

The invention claimed is:

1. A driver monitor system for predicting impairment of a user of a vehicle, the system comprising:
at least one video camera;
an input output device for inputting a list of at least one medication being taken by the user of the vehicle; and
processing circuitry configured to:
predict at least one side effect of the at least one medication based on the half-life of the at least one medication,
detect eye gaze movement, eye lid position, and facial expression of the user using images from the at least one video camera,
use the eye gaze movement, eye lid position, and facial expression to predict whether the user is transitioning into an impaired physical state that is a side effect of the at least one medication,
verify the at least one side effect of the at least one medication,
determine whether the user is fit to drive using the verified at least one side effect of the at least one medication, and
output to the vehicle an instruction to operate the vehicle in a level of automation that makes up for the at least one side effect or to perform a safe pull over operation of the vehicle.

2. The driver monitor system of claim 1 , wherein the input output device is configured to display a status list of the at least one medication and a most recent time that the at least one medication had been taken by the user.

3. The driver monitor system of claim 2 , wherein the input output device is configured to add a medication to the status list of the at least one medication.

4. The driver monitor system of claim 2 , wherein the processing circuitry is configured to predict at least one side effect of the at least one medication based on the half-life of the medication by determining if the half-life of the at least one medication has been reached using the most recent time that the at least one medication had been taken by the driver.

5. The driver monitor system of claim 1 , further including a machine learning device,
wherein the input output device outputs a verification request and receives a response to the verification request, and
wherein the eye gaze movement, the eye lid position, and the facial expression are fed back to the machine learning device which learns to predict whether the driver is transitioning into an impaired physical state.

6. The driver monitor system of claim 5 , wherein parameters of the machine learning device that are learned are stored in a memory as a profile associated with the user.

7. The driver monitor system of claim 6 , wherein independent profiles are stored in the memory in association with respective different users.

8. The driver monitor system of claim 5 , wherein the processing circuitry further monitors eye gaze movement, and
wherein the eye gaze movement is fed back to the machine learning device which learns to predict whether the driver is transitioning into an impaired physical state.

9. The driver monitor system of claim 5 , wherein the machine learning device learns by performing a reinforcement learning algorithm.

10. The driver monitor system of claim 1 , wherein the processing circuitry is configured to predict a side effect of the at least one medication including sorting side effects by expected degree of user impairment, and to select at least one side effect having a highest degree of user impairment.

11. A method of predicting impairment of a driver of a vehicle by a driver monitor system including at least one video camera, an input output device for inputting a list of at least one medication being taken by the driver of the vehicle, and processing circuitry, the method comprising:
predicting at least one side effect of the at least one medication based on the half-life of the at least one medication,
detecting, by the processing circuitry, eye gaze movement, eye lid position, and facial expression using images from the at least one video camera;
using the eye gaze movement, eye lid position, and facial expression to predict, by the processing circuitry, whether the user is transitioning into an impaired physical state;
verifying the at least one side effect of the at least one medication;
determining whether the user is fit to drive using the verified at least one side effect of the at least one medication; and
outputting to the vehicle an instruction to operate the vehicle in a level of automation that makes up for the at least one side effect or to perform a safe pull over operation of the vehicle.

12. The method of claim 11 , further comprising:
displaying, by the input output device, a status list of the at least one medication and a most recent time that the at least one medication had been taken by the user.

13. The method of claim 12 , further comprising:
adding, by the input output device, a medication to the status list of the at least one medication.

14. The method of claim 12 , further comprising:
predicting, by the processing circuitry, at least one side effect of the at least one medication based on the half-life of the medication by determining if the half-life of the at least one medication has been reached using the most recent time that the at least one medication had been taken by the user.

15. The method of claim 11 , the system further including a machine learning device, the method further comprising:
outputting, by the input output device, a verification request and receiving a response to the verification request; and
feeding back the eye gaze movement, the eye lid position, and the facial expression to the machine learning device which learns to predict whether the driver is transitioning into an impaired physical state.

16. The method of claim 15 , further comprising:
storing in a memory parameters of the machine learning device that are learned as a profile associated with the user.

17. The method of claim 16 , further comprising:
storing in the memory independent profiles in association with respective different users.

18. The method of claim 15 , further comprising:
monitoring, by the processing circuitry, eye gaze movement, and
feeding back the eye gaze movement to the machine learning device which learns to predict whether the driver is transitioning into an impaired physical state.

19. The method of claim 15 , wherein the machine learning device learns by performing a reinforcement learning algorithm.

20. The method of claim 11 , wherein the predicting a side effect of the at least one medication includes sorting side effects by expected degree of user impairment, and selecting at least one side effect having a highest degree of user impairment.

Assignees

• Toyota Res Inst Inc

Inventors

• Arechiga-Gonzalez Nikos
• Kong Soonho
• Decastro Jonathan
• Permenter Frank
• Park Dennis

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• B60W60/0015 Primary

  specially adapted for safety · CPC title

• B60W40/08

  related to drivers or passengers · CPC title

• B60W2540/225

  Direction of gaze · CPC title

• A61B5/4848

  Monitoring or testing the effects of treatment, e.g. of medication · CPC title