Method for distress and road rage detection

Patent metadata

Field	Value
Publication number	US-10322728-B1
Application number	US-201815902729-A
Country	US
Kind code	B1
Filing date	Feb 22, 2018
Priority date	Feb 22, 2018
Publication date	Jun 18, 2019
Grant date	Jun 18, 2019

Abstract

A system for determining distress of a driver of a vehicle is provided, comprising a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning System (GPS) data sensors. The system also includes one or more processors configured to receive inputs from the plurality of sensors, and process the received inputs to obtain a driver heat change estimate, a driver expression estimate, a driver gesture estimate, an on-board diagnostics (OBD) estimate, and a GPS estimate. The one or more processors are further configured to store the estimates in a memory, use the stored estimates to generate deviation scores for each of the estimates, execute a machine learning algorithm to classify driver behavior as normal or impaired based on the deviation scores, and generate a warning based on the classification indicating impaired driver behavior.

First claim

What is claimed is:

1. A method for to determine distress of a driver of a vehicle, the method comprising:
receiving inputs by one or more processors from a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning System (GPS) data sensors;
processing the received inputs by the one or more processors to obtain a driver heat change estimate, a driver expression estimate, a driver gesture estimate, an on-board diagnostics (OBD) estimate, and a GPS estimate;
storing by the one or more processors the estimates in a memory;
using the stored estimates by the one or more processors to generate deviation scores for each of the driver heat change estimate, the driver expression estimate, the driver gesture estimate, the OBD estimate, and the GPS estimate, wherein generating the deviation score for the driver heat change estimate includes:
generating a normal driving model offline using normal driving thermal images of the driver;
comparing the normal driving model with real-time thermal image data of the driver to obtain a comparison result; and
applying a probability density function (PDF) to the comparison result to obtain the deviation score for the driver heat change estimate;

executing a machine learning algorithm by the one or more processors to classify driver behavior as normal or impaired based on the deviation scores; and
generating a warning by the one or more processors based on the classification indicating impaired driver behavior.

2. The method of claim 1 , wherein generating the deviation score for the driver expression estimate includes:
using detection-tracking-validation (DTV) to localize frontal face images of the driver;
constructing a face stream frame from a partitioned face region of the frontal face images;
applying a fully convolutional network (FCN) to the face stream frame using an encoder, including using multiple convolutional, pooling, batch normalization, and rectified linear unit (ReLU) layers;
reshaping a feature map of a last layer of the encoder into vector form to obtain an output, and applying the output to a recurrent neural network (RNN) to obtain a normal driving expression model using a Gaussian mixture model (GMM); and
comparing a real-time driver or passenger expression with the normal driving expression model to calculate the deviation score for the driver expression estimate.

3. The method of claim 1 , wherein generating the deviation score for the driver gesture estimate includes:
detecting driver gestures to obtain an image of a hands region of the driver;
constructing a two-layer hand stream from the image and normalizing the two-layer hand stream for size adjustment;
applying a fully convolutional network (FCN) to the two-layer hand stream using an encoder, including using multiple convolutional, pooling, batch normalization, and rectified linear unit (ReLU) layers;
reshaping a feature map of a last layer of the encoder into vector form to obtain an output, and applying the output to a recurrent neural network (RNN) to obtain a normal driving gesture model using a Gaussian mixture model (GMM); and
comparing a real-time driver gesture with the normal driving gesture model to calculate the deviation score for the driver gesture estimate.

4. The method of claim 1 , wherein generating the deviation score for the OBD estimate includes:
collecting normal driving data from OBD related to two or more of vehicle speed, steering wheel angle, steering wheel angle error, time to lane crossing, and time to collision;
using the normal driving data to generate a normal driving model for each of the two or more of vehicle speed, steering wheel angle, steering wheel angle error, time to lane crossing, and time to collision; and
comparing real-time data to the normal driving model for each of the two or more of vehicle speed, steering wheel angle, steering wheel angle error, time to lane crossing, and time to collision to generate a deviation score for each of the two or more of vehicle speed, steering wheel angle, steering wheel angle error, time to lane crossing, and time to collision.

5. The method of claim 1 , wherein the warning includes a visual alert.

6. The method of claim 1 , wherein the warning includes an audio output.

7. The method of claim 1 , wherein the warning includes a suggested corrective driver action using a display.

8. The method of claim 1 , wherein using the processor to execute the machine learning algorithm to classify the driver behavior as normal or impaired includes using a Gaussian mixture model (GMM).

9. The method of claim 8 , wherein expectation maximization is used to estimate model parameters of the GMM.

10. The method of claim 1 , wherein the processor is configured to generate a normal driving model offline for comparison to real-time driving data.

11. A system for determining distress of a driver of a vehicle, the system comprising:
a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning System (GPS) data sensors; and
one or more processors in communication with the plurality of sensors, the one or more processors configured to:
receive inputs from the plurality of sensors;
process the received inputs to obtain a driver heat change estimate, a driver expression estimate, a driver gesture estimate, an on-board diagnostics (OBD) estimate, and a GPS estimate;
store the estimates in a memory;
use the stored estimates to generate deviation scores for each of the driver heat change estimate, the driver expression estimate, the driver gesture estimate, the OBD estimate, and the GPS estimate, wherein generating the deviation score for the driver expression estimate includes:
using detection-tracking-validation (DTV) to localize frontal face images of the driver;
constructing a face stream frame from a partitioned face region of the frontal face images;
applying a fully convolutional network (FCN) to the face stream frame using an encoder, including using multiple convolutional, pooling, batch normalization, and rectified linear unit (ReLU) layers;
reshaping a feature map of a last layer of the encoder into vector form to obtain an output, and applying the output to a recurrent neural network (RNN) to obtain a normal driving expression model using a Gaussian mixture model (GMM); and
comparing a real-time driver or passenger expression with the normal driving expression model to calculate the deviation score for the driver expression estimate;

execute a machine learning algorithm to classify driver behavior as normal or impaired based on the deviation scores; and
generate a warning based on the classification indicating impaired driver behavior.

12. The system of claim 11 , wherein the plurality of sensors further includes exterior-facing sensors of the vehicle.

13. The system of claim 11 , wherein the processor is further configured to receive a traffic information input, including at least one of a speed limit and a lane direction.

14. The system of claim 11 , wherein the warning includes a suggested corrective driver action using a display.

15. A non-transitory computer-readable medium storing computer instructions to determine distress of a driver of a vehicle and provide a warning, that when executed by one or more processors, cause the one or more processors to perform steps of:
receiving inputs from a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning S

Assignees

• Futurewei Technologies Inc

Inventors

• Porikli Fatih
• Wu Yuzhu
• Bill Luis

Classifications

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V40/161 Primary

  Detection; Localisation; Normalisation · CPC title

• B60W50/14 Primary

  Means for informing the driver, warning the driver or prompting a driver intervention · CPC title