Ensemble neural network state machine for detecting distractions

Patent metadata

Field	Value
Publication number	US-11386325-B1
Application number	US-202117454799-A
Country	US
Kind code	B1
Filing date	Nov 12, 2021
Priority date	Nov 12, 2021
Publication date	Jul 12, 2022
Grant date	Jul 12, 2022

Abstract

A vehicle device may execute one or more neural networks (and/or other artificial intelligence), based on input from one or more of the cameras and/or other sensors, to intelligently detect safety events in real-time. The one or more neural networks may be an ensemble neural network that includes neural networks for detecting a head and hand of a user, neural networks for detecting hand actions of the user, neural networks for detecting the head pose of the user, neural networks for predicting an occurrence of an event, and neural networks for predicting a start time and end time of the event. Further, the neural networks can be segmented into a modular neural network based on metadata. The segmentation of the neural network can define a thin layer of the modular neural network to enable independent tuning of the thin layer of the modular neural network.

First claim

What is claimed is:

1. A vehicle device comprising:
a computer readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the vehicle device to:
access sensor data from one or more sensors associated with a vehicle, the sensor data associated with an image of a scene;
execute an ensemble neural network configured to detect occurrence of a safety event associated with a driver of the vehicle, wherein the safety event indicates a distracted state of the driver of the vehicle, the ensemble neural network comprising a plurality of models including:
a first model configured to detect a face and a hand of a user of the vehicle based at least in part on the sensor data,
a second model configured to detect one or more hand actions of the user based at least in part on the detected hand of the user,
a third model configured to detect a head pose of the user based at least in part on the detected face of the user,
a fourth model configured to detect a gaze of the user based at least in part on the detected face of the user, and
a fifth model configured to predict, based at least in part on the one or more hand actions, the head pose, and the gaze, a probability of a safety event associated with the user of the vehicle; and

in response to determining at least that the probability of the safety event exceeds a threshold, trigger an event alert indicative of occurrence of the safety event.

2. The vehicle device of claim 1 , wherein the ensemble neural network further comprises a plurality of layers, wherein the plurality of models are distributed across the plurality of layers.

3. The vehicle device of claim 2 , wherein a first layer of the plurality of layers of the ensemble neural network comprises the first model and the second model and a second layer of the plurality of layers of the ensemble neural network comprises the third model.

4. A vehicle device comprising:
a computer readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the vehicle device to:
access sensor data from one or more sensors associated with a vehicle;
execute an ensemble neural network configured to detect occurrence of a safety event associated with a driver of the vehicle, the ensemble neural network comprising a plurality of models including:
a first model configured to detect a face and a hand of a user of the vehicle based at least in part on the sensor data,
a second model configured to detect one or more hand actions of the user based at least in part on the detected hand of the user,
a third model configured to detect a head pose of the user based at least in part on the detected face of the user, and
a fourth model configured to predict, based at least in part on the one or more hand actions and the head pose, a probability of the safety event associated with the user of the vehicle; and

in response to determining at least that the probability of the safety event exceeds a threshold, trigger an event alert indicative of occurrence of the safety event.

5. The vehicle device of claim 4 , wherein the first model comprises a face and hand detection model, the second model comprises a hand action classification model, and the third model comprises a head pose classification model.

6. The vehicle device of claim 4 , wherein the plurality of models further includes a fifth model comprising a gaze detection model, the fifth model configured to detect one or more eye gaze angles based at least in part on the detected face of the user, wherein the fourth model is further configured to predict the probability of the safety event associated with the user of the vehicle further based at least in part on the one or more eye gaze angles.

7. The vehicle device of claim 4 , wherein to access the sensor data, the one or more processors are configured to execute the program instructions to further cause the vehicle device to obtain the sensor data from a plurality of sensors.

8. The vehicle device of claim 4 , wherein the sensor data comprises at least one of camera data, accelerometer data, audio data, or location data.

9. The vehicle device of claim 4 , wherein the one or more processors are configured to execute the program instructions to:
store the event alert to generate a stored event alert;
trigger an additional event alert indicative of occurrence of the safety event;
compare the stored event alert with the additional event alert; and
identify timing information associated with the event based at least in part on comparing the stored event alert with the additional event alert.

10. The vehicle device of claim 4 , wherein, to trigger the event alert, the one or more processors are configured to execute the program instructions to further cause the vehicle device to trigger the event alert at a frame by frame level.

11. The vehicle device of claim 4 , wherein the ensemble neural network comprises a frame classifier pipeline and a sequence detector pipeline.

12. The vehicle device of claim 4 , wherein the safety event indicates a distracted state of the user.

13. The vehicle device of claim 4 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to train the ensemble neural network.

14. The vehicle device of claim 4 , wherein the first model is further configured to detect the face by identifying one or more face bounding boxes and detect the hand by identifying one or more hand bounding boxes.

15. The vehicle device of claim 4 , wherein the third model is configured to detect the head pose by detecting one or more of a yaw, a pitch, or a roll angle.

16. The vehicle device of claim 4 , wherein the one or more hand actions comprise at least one of a neutral hand action, a hand interacting with a phone hand action, or a hand interacting with food hand action.

17. The vehicle device of claim 4 , wherein the fourth model is configured to predict the probability of the safety event based on outputs from at least the first model, the second model, and the third model.

18. The vehicle device of claim 4 , wherein the plurality of models includes a fifth model is configured to detect a start time and an end time of the safety event based at least in part on the probability of the safety event.

19. The vehicle device of claim 4 , wherein the sensor data comprises streaming sensor data.

20. A computer-implemented method comprising:
accessing sensor data from one or more sensors associated with a vehicle;
executing an ensemble neural network configured to detect occurrence of a safety event associated with a driver of the vehicle, the ensemble neural network comprising a plurality of models including:
a first model configured to detect a face and a hand of the user of the vehicle based at least in part on the sensor data,
a second model configured to detect one or more hand actions of the user based at least in part on the detected hand of the user,
a third model configured to detect a head pose of the user based at least in part on the detected face of the user, and
a fourth model configured to predict, based at least in part on the one or more hand actions and the head pose, a probability of the safety event; and

in response to determining at least that the probability of the safety event exceeds a threshold, trigger an event alert indicative of the occurrence of the ev

Assignees

• Samsara Inc

Inventors

• Srinivasan Sharan
• Tuan Brian
• Bicket John
• Wang Jing
• Akhtar Muhammad Ali
• Acevedo Abner Ayala
• Kellerman Bruce
• Shieh Vincent

Classifications

• B60W40/09 Primary

  Driving style or behaviour · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/098 Primary

  Distributed learning, e.g. federated learning · CPC title

• B60W2540/225

  Direction of gaze · CPC title