Method of real time vehicle recognition with neuromorphic computing network for autonomous driving

Patent metadata

Field	Value
Publication number	US-11199839-B2
Application number	US-201916519814-A
Country	US
Kind code	B2
Filing date	Jul 23, 2019
Priority date	Jul 23, 2018
Publication date	Dec 14, 2021
Grant date	Dec 14, 2021

Abstract

Described is a system for online vehicle recognition in an autonomous driving environment. Using a learning network comprising an unsupervised learning component and a supervised learning component, images of moving vehicles extracted from videos captured in the autonomous driving environment are learned and classified. Vehicle feature data is extracted from input moving vehicle images. The extracted vehicle feature data is clustered into different vehicle classes using the unsupervised learning component. Vehicle class labels for the different vehicle classes are generated using the supervised learning component. Based on a vehicle class label for a moving vehicle in the autonomous driving environment, the system selects an action to be performed by the autonomous vehicle, and causes the selected action to be performed by the autonomous vehicle in the autonomous driving environment.

First claim

What is claimed is:

1. A system for online vehicle recognition in an autonomous driving environment, the system comprising:
one or more processors and a non-transitory memory having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of:
using a learning network comprising an unsupervised learning component and a supervised learning component, learning and classifying images of moving vehicles extracted from videos captured in the autonomous driving environment by:
using at least one of motion vectors and optical flows to detect moving vehicles in an input video sequence of the autonomous driving environment;
extracting vehicle feature data from the input video sequence by segmenting the motion vectors or optical flows to find bounding boxes of moving vehicles for extracting images of the moving vehicles from the input video sequence;
automatically clustering the extracted vehicle feature data into different vehicle classes using the unsupervised learning component; and
generating vehicle class labels for the different vehicle classes using the supervised learning component; and

selecting an action to be performed by an autonomous vehicle based on a vehicle class label for one or more moving vehicles in the autonomous driving environment; and
causing the selected action to be performed by the autonomous vehicle in the autonomous driving environment.

2. The system as set forth in claim 1 , wherein the unsupervised learning component is a spiking reservoir network comprising a plurality of neurons, and the supervised learning component is a linear neural network classifier comprising a plurality of neurons.

3. The system as set forth in claim 2 , wherein each neuron in the spiking reservoir network is fully connected to the plurality of neurons in the linear neural network classifier, and wherein a set of synaptic weights between the spiking reservoir network and the linear neural network classifier is trained by a supervised learning method.

4. The system as set forth in claim 2 , wherein the linear neural network classifier uses averaged spiking rates from the spiking reservoir network to generate the vehicle class labels.

5. The system as set forth in claim 1 , wherein upon extracting features from an unknown moving vehicle image, the one or more processors perform operations of:
generating an unknown vehicle identification signal;
saving any images of the unknown moving vehicle;
requesting a new vehicle class label for the unknown moving vehicle when a number of saved images of the unknown vehicle reaches a threshold value; and
learning the new vehicle class.

6. The system as set forth in claim 2 , wherein synaptic weights are saved following supervised learning, and wherein to learn new vehicle classes, the saved synaptic weights representing old vehicle classes, are used as initial weights for unsupervised learning.

7. The system as set forth in claim 6 , wherein synaptic weights learned from new vehicle classes are combined with the saved synaptic weights, and wherein the combined synaptic weights are used for classification of both old vehicle classes and new vehicle classes.

8. A computer implemented method for online vehicle recognition in an autonomous driving environment, the method comprising an act of:
causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
using a learning network comprising an unsupervised learning component and a supervised learning component, learning and classifying images of moving vehicles extracted from videos captured in the autonomous driving environment by:
using at least one of motion vectors and optical flows to detect moving vehicles in an input video sequence of the autonomous driving environment;
extracting vehicle feature data from the input video sequence by segmenting the motion vectors or optical flows to find bounding boxes of moving vehicles for extracting images of the moving vehicles from the input video sequence;
automatically clustering the extracted vehicle feature data into different vehicle classes using the unsupervised learning component; and
generating vehicle class labels for the different vehicle classes using the supervised learning component; and

selecting an action to be performed by an autonomous vehicle based on a vehicle class label for one or more moving vehicles in the autonomous driving environment; and
causing the selected action to be performed by the autonomous vehicle in the autonomous driving environment.

9. The method as set forth in claim 8 , wherein the unsupervised learning component is a spiking reservoir network comprising a plurality of neurons, and the supervised learning component is a linear neural network classifier comprising a plurality of neurons.

10. The method as set forth in claim 9 , wherein each neuron in the spiking reservoir network is fully connected to the plurality of neurons in the linear neural network classifier, and wherein a set of synaptic weights between the spiking reservoir network and the linear neural network classifier is trained by a supervised learning method.

11. The method as set forth in claim 9 , wherein the linear neural network classifier uses averaged spiking rates from the spiking reservoir network to generate the vehicle class labels.

12. The method as set forth in claim 8 , wherein upon extracting features from an unknown moving vehicle image, the one or more processors perform operations of:
generating an unknown vehicle identification signal;
saving any images of the unknown moving vehicle;
requesting a new vehicle class label for the unknown moving vehicle when a number of saved images of the unknown vehicle reaches a threshold value; and
learning the new vehicle class.

13. The method as set forth in claim 9 , wherein synaptic weights are saved following supervised learning, and wherein to learn new vehicle classes, the saved synaptic weights representing old vehicle classes, are used as initial weights for unsupervised learning.

14. The method as set forth in claim 13 , wherein synaptic weights learned from new vehicle classes are combined with the saved synaptic weights, and wherein the combined synaptic weights are used for classification of both old vehicle classes and new vehicle classes.

15. A computer program product for online vehicle recognition in an autonomous driving environment, the computer program product comprising:
computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors for causing the processor to perform operations of:
using a learning network comprising an unsupervised learning component and a supervised learning component, learning and classifying images of moving vehicles extracted from videos captured in the autonomous driving environment by:
using at least one of motion vectors and optical flows to detect moving vehicles in an input video sequence of the autonomous driving environment;
extracting vehicle feature data from the input video sequence by segmenting the motion vectors or optical flows to find bounding boxes of moving vehicles for extracting images of the moving vehicles from the input video sequence;
automatically clustering the extracted vehicle feature data into different vehicle classes using the unsupervised learning component; and
generating vehicle class labels for the different vehicle classes using the supervised learning component; a

Assignees

• Hrl Lab Llc

Inventors

• Jiang Qin
• Cho Youngkwan
• Stepp Nigel D
• Skorheim Steven W
• De Sapio Vincent
• Pilly Praveen K
• Scorcioni Ruggero

Classifications

• G06V10/7788

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06V10/451

  with interaction between the filter responses, e.g. cortical complex cells · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/82

  using neural networks · CPC title