Driver and environment monitoring to predict human driving maneuvers and reduce human driving errors

Patent metadata

Field	Value
Publication number	US-12441371-B2
Application number	US-202117352560-A
Country	US
Kind code	B2
Filing date	Jun 21, 2021
Priority date	Jun 21, 2021
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

Techniques are disclosed to detect, inform, and automatically correct typical awareness-related human driver mistakes. This may include those that are caused by a misunderstanding of the current situation, a lack of focus or attention, and/or overconfidence in any currently-engaged assistance features. The disclosure is directed to the prediction of vehicle maneuvers using driver and external environment modeling. The consequence of executing a predicted maneuver is categorized based upon its risk or danger posed to the driving environment, and the vehicle may execute various actions based upon the categorization of a predicted riving maneuver to mitigate or eliminate that risk.

First claim

What is claimed is:

1. A vehicle, comprising:
maneuver prediction processing circuitry configured to:
receive a first data stream comprising driver state data that includes images of a driver of the vehicle;
receive a second data stream comprising vehicle state data that includes images of an exterior of the vehicle,
wherein the first data stream and the second data stream are received concurrently and within a sampling window of a predetermined time duration;
upon expiration of the predetermined time duration, (i) calculate, based upon an analysis of the images of the driver of the vehicle, a first probability from among a first set of probabilities of execution of each respective one of a first set of vehicle maneuvers, (ii) calculate, based upon an analysis of the images of the exterior of the vehicle, a second probability from among a second set of probabilities of execution of each respective one of a second set of vehicle maneuvers, and (iii) predict a vehicle maneuver based upon a weighted combination of the first probability and the second probability;

monitoring circuitry configured to categorize a consequence resulting from an execution of the predicted vehicle maneuver into one of a plurality of risk-based categories, each one of the plurality of risk-based categories being based upon a severity of danger resulting from the execution of the predicted vehicle maneuver in a driving environment of the vehicle; and
a controller configured to cause the vehicle to perform a corrective action that is based upon the one of the plurality of risk-based categories that the consequence resulting from the execution of the predicted vehicle maneuver is categorized.

2. The vehicle of claim 1 , wherein the plurality of risk-based categories includes a high-risk category associated with the predicted vehicle maneuver matching one of a set of predetermined vehicle maneuvers when executed, a mid-risk category associated with an occurrence of atypical driver behavior associated with the predicted vehicle maneuver being executed, and a low-risk category associated with an occurrence of a violation of a non-safety critical traffic rule associated with the predicted vehicle maneuver being executed.

3. The vehicle of claim 1 , wherein the plurality of risk-based categories includes a high-risk category associated with the predicted vehicle maneuver matching one of a set of predetermined vehicle maneuvers when executed, and
wherein the controller is configured to, when the consequence resulting from the execution of the predicted vehicle maneuver is categorized in the high-risk category, cause the vehicle to perform the corrective action corresponding to an intervening vehicle control operation to prevent the execution of the predicted vehicle maneuver.

4. The vehicle of claim 3 , wherein the controller is configured to cause the vehicle to perform the intervening vehicle control operation to provide a shared control of the vehicle between a driver and autonomous vehicle controls.

5. The vehicle of claim 4 , wherein the controller is configured to cause the vehicle to perform the intervening vehicle control operation to provide the shared control of the vehicle by generating a weighted function that includes a first weighting applied to driver vehicle control inputs, and a second weighting applied to autonomous vehicle control inputs.

6. The vehicle of claim 4 , wherein the controller is configured to cause the vehicle to maintain the intervening vehicle control operation to provide the shared control of the vehicle until an input is received indicating that the driver is capable of controlling the vehicle.

7. The vehicle of claim 6 , wherein the controller is further configured to cause the vehicle to maintain the intervening vehicle control operation to provide the shared control of the vehicle until the driver state data indicates that the driver has regained focus.

8. The vehicle of claim 1 , wherein the plurality of risk-based categories includes a mid-risk category associated with atypical driver behavior, the atypical driver behavior being associated with the predicted vehicle maneuver being executed, and
wherein the controller is configured to, when the consequence resulting from the execution of the predicted vehicle maneuver is categorized as the mid-risk category, cause the vehicle to perform the corrective action corresponding to a generation of a notification to the driver about the atypical driver behavior.

9. The vehicle of claim 8 , wherein the maneuver prediction processing circuitry is configured to predict an additional vehicle maneuver based upon the analysis of the driver state data after the notification is generated, and
wherein the monitoring circuitry is configured to categorize a consequence resulting from an execution of the additional predicted vehicle maneuver into one of the plurality of risk-based categories.

10. The vehicle of claim 1 , wherein the plurality of risk-based categories includes a low-risk category associated with an occurrence of a violation of a non-safety critical traffic rule associated with the predicted vehicle maneuver being executed, and
wherein the controller is configured to, when the consequence resulting from the execution of the predicted vehicle maneuver is categorized as the low-risk category, cause the vehicle to autonomously activate a vehicle component to correct for the occurrence of the violation of the non-safety critical traffic rule.

11. The vehicle of claim 1 , wherein the controller is configured to cause the vehicle to perform one of a set of different corrective actions, each one of the set of different corrective actions being assigned to a respective one of the plurality of risk-based categories that the consequence resulting from the execution of the predicted vehicle maneuver is categorized, and
wherein the set of different corrective actions comprise different levels of vehicle intervention with respect to vehicle control.

12. The vehicle of claim 1 , wherein the first probability comprises a maximum probability from among the first set of probabilities, and
wherein the second probability comprises a maximum probability from among the second set of probabilities.

13. A controller of a vehicle, comprising:
a data interface configured to:
provide a first data stream comprising driver state data that includes images of a driver of the vehicle;
provide a second data stream comprising vehicle state data that includes images of an exterior of the vehicle,
wherein the first data stream and the second data stream are received concurrently and within a sampling window of a predetermined time duration; and

one or more processors configured to:
upon expiration of the predetermined time duration, (i) calculate, based upon an analysis of the images of the driver of the vehicle, a first probability from among a first set of probabilities of execution of each respective one of a first set of vehicle maneuvers, (ii) calculate, based upon an analysis of the images of the exterior of the vehicle, a second probability from among a second set of probabilities of execution of each respective one of a second set of vehicle maneuvers, and (iii) predict a vehicle maneuver based upon a weighted combination of the first probability and the second probability;
categorize a consequence resulting from an execution of the predicted vehicle maneuver into one of a plurality of risk-based categories, each one of the plurality of risk-based categories being based upon a severity of danger resulting from the execution of the predicted vehicle maneuver in a driving environment of the vehicle; and
cause the vehicle

Assignees

• Intel Corp

Inventors

• Rosales Rafael
• Alvarez Ignacio J
• Geissler Florian
• Kose Cihangir Neslihan
• Paulitsch Michael

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V20/597

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title

• B60W40/08 Primary

  related to drivers or passengers · CPC title

• B60W60/0059

  Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity · CPC title

• B60W2420/40

  Photo, light or radio wave sensitive means, e.g. infrared sensors · CPC title