Driver monitoring and response system

Patent metadata

Field	Value
Publication number	US-11279279-B2
Application number	US-201716472760-A
Country	US
Kind code	B2
Filing date	Dec 19, 2017
Priority date	Dec 22, 2016
Publication date	Mar 22, 2022
Grant date	Mar 22, 2022

Abstract

An evaluation engine has two or more modules to assist a driver of a vehicle. A driver drowsiness module analyzes monitored features of the driver to recognize two or more levels of drowsiness of the driver of the vehicle. The driver drowsiness module evaluates drowsiness of the driver based on observed body language and facial analysis of the driver. The driver drowsiness module is configured to analyze live multi-modal sensor inputs from sensors against at least one of i) a trained artificial intelligence model and ii) a rules based model while the driver is driving the vehicle to produce an output comprising a driver drowsiness-level estimation. A driver assistance module provides one or more positive assistance mechanisms to the driver to return the driver to be at or above the designated level of drowsiness.

First claim

What is claimed is:

1. An evaluation engine having two or more modules to monitor a driver of a vehicle, comprising:
a driver drowsiness module that is configured to analyze monitored features of the driver to be capable to recognize two or more levels of drowsiness of the driver of the vehicle;
a facial analysis module that is configured to be capable to perform at least two of i) face tracking, ii) eye movement and iii) eye blink tracking on the driver of the vehicle to assist in detecting the levels of drowsiness of the driver of the vehicle, where an output analysis of the facial analysis module is supplied to the driver drowsiness module;
a sensor interface that is located among the two or more modules, including the facial analysis module and the driver drowsiness module, and one or more sensors, where the sensor interface is configured to receive input from the one or more sensors located in the vehicle including i) one or more cameras, and ii) a motion sensing device coupled with a speech user interface, to monitor the driver of the vehicle;
where the driver drowsiness module is configured to utilize the output of the facial analysis module to evaluate drowsiness of the driver based on at least one of observed body language and facial analysis of the driver, to detect and classify one or more levels of drowsiness of the driver of the vehicle when those states occur for the driver;
a driver assistance module that is configured to attempt to maintain the driver in a level selected from a group consisting of i) in a non-drowsiness level, ii) at or below a first level of drowsiness of the driver, and iii) any combination of both, based on an output from the driver drowsiness module; and, when the driver is not at least at or below the first level of drowsiness of the driver, then the driver assistance module is configured to provide one or more positive assistance mechanisms back to the driver to attempt to change the driver's level to the level of i) where the driver is the non-drowsiness level, ii) where the driver's level of drowsiness is lowered to a lower level of drowsiness, and iii) any combination of both, where the evaluation engine with its two or more modules is adaptive on providing a level of assistance that the driver needs in order to get back to any of i) the non-drowsiness level, and/or ii) one of the levels of drowsiness of the driver, based on feedback collected from the driver from the sensor interface, and an assessment by the driver assistance module of what effect a first level of assistance provided to the driver had on a current level of drowsiness of the driver.

2. The evaluation engine of claim 1 , where the driver assistance module is configured to monitor what effect the first level of assistance provided to the driver had on the level of drowsiness of the driver and determine whether additional or combinations of different types of assistance should further be presented to the driver to return the driver to at least a desired level of drowsiness.

3. The evaluation engine of claim 2 , where the driver assistance module is configured to provide one or more additional or combinations of different types of assistance back to the driver to attempt to change the driver's level of drowsiness including running a fan, running an air conditioner, changing a temperature in the car of the driver, shaking a seat of the driver, and changing a smell in a vehicle of the driver.

4. The evaluation engine of claim 1 , where one or more driver-drowsiness machine-learning models utilize ground truth correlations and deep learning machine learning algorithms to train the models, and where the one or more driver-drowsiness machine-learning models use a drowsiness level classification scheme that has at least three or more different levels of drowsiness of the driver, and
once the one or more driver-drowsiness machine-learning models are trained, they are used to analyze live multi-modal sensor inputs from the sensors while the driver is operating the vehicle to produce an output including a current level of drowsiness estimation specific to that driver.

5. The evaluation engine of claim 1 , where the sensor interface is configured to receive a multi-modal sensor input from at least three sensors including i) the motion sensing device coupled with the speech user interface that includes a microphone, ii) a hi-resolution InfraRed camera that is coupled to one or more InfraRed light sources in the vehicle that are positioned to narrowly focus on a face of the driver, and iii) a wide-angle lens camera positioned to capture a view of the driver's head and upper body.

6. The evaluation engine of claim 1 , where the facial analysis module has an ocular activity analysis module that is configured to cooperate with an infra-red light source to track a direction of a head of the driver relative a steering wheel of the vehicle and an angle of a gaze of the eyes of the driver of the vehicle, where the ocular activity analysis module implements a glint-based tracking mechanism that tracks corneal glints from the infra-red light source.

7. The evaluation engine of claim 1 , where the driver drowsiness module is configured to detect and classify three or more levels of drowsiness of the driver of the vehicle, including at least marginally drowsy, moderately drowsy, and significantly drowsy, when those states occur for the driver; and
where the evaluation engine with its two or more modules is configured to adapt an interactive conversation between itself and the driver in real time, based on the feedback from the driver and a current level of drowsiness of the driver from the three of more levels of drowsiness as indicated by an output from the driver activity tracking module.

8. The evaluation engine of claim 1 , where the driver assistance module is configured to provide a first positive assistance mechanism of engaging the driver with a personalized spoken summary as the first level of assistance, based on the driver's current level of drowsiness from the three of more levels of drowsiness, as determined by the driver drowsiness module, that is i) variable in decibel level, ii) selection of what kind of content of a document that the driver assistance module believes to be of interest to the driver, or iii) both variable in decibel level as well as what kind of content of the document that the system believes to be of interest to the driver.

9. The evaluation engine of claim 8 , where the driver assistance module utilizes a document summarization engine to produce an extractive summary of the content of the document, where a driver-specific preference model extracts driver preferences from texts, browsing habits, and input solicited from the user, where a text-to-speech subsystem that works with the driver assistance module is used to prepare the summarized content of the document to report to the driver through a speaker of the vehicle.

10. The evaluation engine of claim 8 , where the driver assistance module is further configured to monitor and evaluate i) the level of drowsiness of the driver as the personalized spoken summary is occurring and ii) what kind of content of the document is being presented as the level of drowsiness of the driver changes.

11. The evaluation engine of claim 1 , where a driver activity analysis module is configured to cooperate with the sensor interface to use a camera in the motion sensing device to track a driver's upper-body, where the driver activity analysis module is configured to track the driver's upper-body posture and movement using the motion sensor's data stream.

12. A method for an evaluation engine monitoring a driver of a vehicle, comprising:
anal

Assignees

• Stanford Res Inst Int
• Toyota Motor Corp

Inventors

• Tamrakar Amir
• Acharya Girish
• Okabe Makoto
• Byrnes John James

Classifications

• A61B5/163

  by tracking eye movement, gaze, or pupil change · CPC title

• A61B5/18 Primary

  for vehicle drivers {or machine operators} · CPC title

• A61B5/1114

  Tracking parts of the body · CPC title

• A61B5/741

  using synthesised speech · CPC title

• A61B5/1128

  using image analysis (A61B5/1127 takes precedence) · CPC title