Shared processing with deep neural networks

Patent metadata

Field	Value
Publication number	US-12299571-B2
Application number	US-202318351117-A
Country	US
Kind code	B2
Filing date	Jul 12, 2023
Priority date	Aug 25, 2017
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

A system includes a processor for performing one or more autonomous driving or assisted driving tasks based on a neural network. The neural network includes a base portion for performing feature extraction simultaneously for a plurality of tasks on a single set of input data. The neural network includes a plurality of subtask portions for performing the plurality of tasks based on feature extraction output from the base portion. Each of the plurality of subtask portions comprise nodes or layers of a neutral network trained on different sets of training data, and the base portion comprises nodes or layers of a neural network trained using each of the different sets of training data constrained by elastic weight consolidation to limit the base portion from forgetting a previously learned task.

First claim

The invention claimed is:

1. A method comprising:
extracting a feature from an input image with a common layer of a neural network;
simultaneously processing the extracted feature with a plurality of task layers of the neural network, wherein the plurality of task layers perform a plurality of different tasks, and wherein each of the plurality of task layers is trained to perform one assigned task of the plurality of tasks;
wherein the common layer and the plurality of task layers are independent layers of the neural network such that the plurality of task layers are sublayers of the common layer;
wherein only the common layer of the neural network is constrained using elastic weight consolidation (EWC) to retain memory for extracting the feature on the input image;
wherein each of the plurality of task layers is trained to perform one task of the plurality of different tasks after the common layer is trained to perform feature extraction; and
wherein each of the plurality of task layers is trained with a different dataset.

2. The method of claim 1 , further comprising training the common layer of the neural network, wherein training the common layer comprises:
training the common layer with a first training dataset comprising a plurality of images comprising a first label, wherein the first label is associated with a first task, and wherein a first task layer of the plurality of task layers is trained to execute the first task; and
training the common layer with a second training dataset comprising a plurality of images comprising a second label, wherein the second label is associated with a second task, and wherein a second task layer of the plurality of task layers is trained to execute the second task.

3. The method of claim 2 , wherein the first task layer executes the first task based on the extracted feature; and
wherein the second task layer executes the second task based on the extracted feature.

4. The method of claim 2 , further comprising training the first task layer of the neural network with the first training dataset and training the second task layer of the neural network with the second training dataset.

5. The method of claim 4 , wherein:
training the first task layer of the neural network comprises providing an output of the common layer of the neural network to the first task layer; and
training the second task layer of the neural network comprises providing the output of the common layer of the neural network to the second task layer;
the method further comprises connecting the first task layer and the second task layer to the common layer during testing or deployment;
the common layer simultaneously performs feature extraction for both the first task and the second task and provides output to both the first task layer and the second task layer; and
the output of the neural network is a result of feature extraction for both the first task and the second task.

6. The method of claim 1 , wherein the plurality of different tasks comprises object detection, image segmentation, lane detection, driving surface detection, and driving condition detection.

7. The method of claim 4 , wherein each of the plurality of task layers executes its respective assigned task simultaneously in response to a camera of a vehicle capturing the input image.

8. The method of claim 1 , wherein task outputs from each of the plurality of task layers are utilized by a vehicle controller to determine a driving maneuver for a vehicle.

9. The method of claim 1 , wherein the neural network is a convolutional neural network (CNN) or a recurrent neural network (RNN).

10. The method of claim 1 , wherein only the common layer of the neural network is constrained using the EWC, and wherein the common layer of the neural is constrained using the EWC during training of each of the plurality of task layers.

11. A system comprising one or more processors configured as a neural network to execute instructions comprising:
extracting a feature from an input image with a common layer of a neural network;
simultaneously processing the extracted feature with a plurality of task layers of the neural network, wherein the plurality of task layers perform a plurality of different tasks, and wherein each of the plurality of task layers is trained to perform one assigned task of the plurality of tasks;
wherein the common layer and the plurality of task layers are independent layers of the neural network such that the plurality of task layers are sublayers of the common layer;
wherein only the common layer of the neural network is constrained using elastic weight consolidation (EWC) to retain memory for extracting the feature on the input image;
wherein each of the plurality of task layers is trained to perform one task of the plurality of different tasks after the common layer is trained to perform feature extraction; and
wherein each of the plurality of task layers is trained with a different dataset.

12. The system of claim 11 , wherein training the common layer comprises:
training the common layer with a first training dataset comprising a plurality of images comprising a first label, wherein the first label is associated with a first task, and wherein a first task layer of the plurality of task layers is trained to execute the first task; and
training the common layer with a second training dataset comprising a plurality of images comprising a second label, wherein the second label is associated with a second task, and wherein a second task layer of the plurality of task layers is trained to execute the second task.

13. The system of claim 12 , wherein the neural network is configured such that the first task layer executes the first task based on the extracted feature; and
wherein the second task layer executes the second task based on the extracted feature.

14. The system of claim 12 , wherein training the neural network comprises training the first task layer of the neural network with the first training dataset and training the second task layer of the neural network with the second training dataset.

15. The system of claim 14 , wherein:
training the first task layer of the neural network comprises providing an output of the common layer of the neural network to the first task layer; and
training the second task layer of the neural network comprises providing the output of the common layer of the neural network to the second task layer;
the method further comprises connecting the first task layer and the second task layer to the common layer during testing or deployment;
the common layer simultaneously performs feature extraction for both the first task and the second task and provides output to both the first task layer and the second task layer; and
the output of the neural network is a result of feature extraction for both the first task and the second task.

16. The system of claim 11 , wherein the plurality of different tasks comprises object detection, image segmentation, lane detection, driving surface detection, and driving condition detection.

17. The system of claim 14 , wherein each of the plurality of task layers executes its respective assigned task simultaneously in response to a camera of a vehicle capturing the input image.

18. The system of claim 11 , further comprising a vehicle controller, wherein task outputs from each of the plurality of task layers are utilized by the vehicle controller to determine a driving maneuver for a vehicle.

19. The system of claim 11 , wherein the neural network is a convolutional

Assignees

• Ford Global Tech Llc

Inventors

• Hotson Guy
• Nariyambut Murali Vidya
• Puskorius Gintaras Vincent

Classifications

• G06N3/09 Primary

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title