Tuning layers of a modular neural network

Patent metadata

Field	Value
Publication number	US-11352014-B1
Application number	US-202117454790-A
Country	US
Kind code	B1
Filing date	Nov 12, 2021
Priority date	Nov 12, 2021
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

A vehicle device may execute one or more neural networks (and/or other artificial intelligence), based on input from one or more of the cameras and/or other sensors, to intelligently detect safety events in real-time. The one or more neural networks may be an ensemble neural network that includes neural networks for detecting a head and hand of a user, neural networks for detecting hand actions of the user, neural networks for detecting the head pose of the user, neural networks for predicting an occurrence of an event, and neural networks for predicting a start time and end time of the event. Further, the neural networks can be segmented into a modular neural network based on metadata. The segmentation of the neural network can define a thin layer of the modular neural network to enable independent tuning of the thin layer of the modular neural network.

First claim

What is claimed is:

1. A vehicle device comprising:
a computer readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the vehicle device to:
access metadata associated with a user of a vehicle, wherein the metadata comprises at least one of one or more characteristics of the user or one or more characteristics of the vehicle;
compare the metadata with segmentation data, wherein the segmentation data identifies statistics associated with tuning particular models;
determine how to segment a neural network based on comparing the metadata with the segmentation data;
segment the neural network to generate a modular neural network based at least in part on determining how to segment the neural network, the modular neural network comprising a plurality of models, wherein each of the plurality of models is independently tunable and trainable to identify corresponding features of sensor data;
obtain sensor data, the sensor data identifying the user of the vehicle;
execute the modular neural network based on the sensor data, wherein the modular neural network outputs a probability of a distracted state of the user of the vehicle;
identify a particular model of the plurality of models based at least in part on the probability of the distracted state of the user of the vehicle; and
independently tune the particular model based at least in part on the probability of the distracted state of the user of the vehicle.

2. A vehicle device comprising:
a computer readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the vehicle device to:
access metadata associated with a user of a vehicle;
segment a neural network to generate a modular neural network based at least in part on the metadata associated with the user of the vehicle, the modular neural network comprising a plurality of models, wherein each of the plurality of models is independently tunable and trainable to identify corresponding features of sensor data;
obtain sensor data, the sensor data identifying the user of the vehicle;
execute the modular neural network based on the sensor data, wherein the modular neural network outputs a probability of an event associated with the user of the vehicle;
identify a particular model of the plurality of models based at least in part on the probability of the event; and
independently tune the particular model based at least in part on the probability of the event.

3. The vehicle device of claim 2 , wherein the plurality of models comprises a face detection model, a hand detection model, a hand action classification model, a head pose classification model, or a gaze detection model.

4. The vehicle device of claim 2 , wherein the modular neural network further comprises a plurality of layers, wherein the plurality of models are distributed across the plurality of layers.

5. The vehicle device of claim 2 , wherein the metadata is specific to the user of the vehicle and the user is a specific individual or a plurality of individuals.

6. The vehicle device of claim 2 , wherein the metadata is specific to the vehicle.

7. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
periodically update the metadata associated with the user of the vehicle.

8. The vehicle device of claim 2 , wherein to obtain the sensor data, the one or more processors are configured to execute the program instructions to further cause the vehicle device to obtain the sensor data from a plurality of sensors.

9. The vehicle device of claim 2 , wherein the sensor data comprises at least one of camera data, accelerometer data, audio data, or location data.

10. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
store the probability of the event to generate a stored probability of the event;
execute the modular neural network based on additional sensor data, wherein the modular neural network outputs an additional probability of the event associated with the user of the vehicle;
compare the stored probability of the event with the additional probability of the event; and
identify timing information associated with the event based at least in part on comparing the stored probability of the event with the additional probability of the event.

11. The vehicle device of claim 2 , wherein, to obtain the probability of the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to obtain the probability of the event at a frame by frame level.

12. The vehicle device of claim 2 , wherein the modular neural network comprises a frame classifier pipeline and a sequence detector pipeline, wherein the particular model comprises one or more of the frame classifier pipeline or the sequence detector pipeline.

13. The vehicle device of claim 2 , wherein to identify the particular model, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify an error associated with the particular model.

14. The vehicle device of claim 2 , wherein, to identify the particular model, the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
identify an error associated with the modular neural network, wherein the error indicates that the event is a false positive event;
determine the error is associated with the particular model.

15. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to train the modular neural network, wherein, to independently tune the particular model, the one or more processors are configured to execute the program instructions to further cause the vehicle device to independently train the particular model subsequent to training the modular neural network.

16. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to train the modular neural network, wherein, to independently tune the particular model, the one or more processors are configured to execute the program instructions to further cause the vehicle device to independently train the particular model without training at least one other model of the plurality of models subsequent to training the modular neural network.

17. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to obtain input from a user computing device, wherein, to identify the particular model, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify the particular model based at least in part on the input.

18. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to provide the probability of the event to a user computing device and obtain input from the user computing device based at least in part on the probability of the event, wherein, to identify the particular model, the one or more processors are configured to exe

Assignees

• Samsara Inc

Inventors

• Srinivasan Sharan
• Tuan Brian
• Bicket John
• Wang Jing
• Akhtar Muhammad Ali
• Acevedo Abner Ayala
• Kellerman Bruce
• Shieh Vincent

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V20/59 Primary

  inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title