Systems and methods for classifying detected objects in an image at an automated driving system

Patent metadata

Field	Value
Publication number	US-11954916-B2
Application number	US-202217665895-A
Country	US
Kind code	B2
Filing date	Feb 7, 2022
Priority date	Feb 7, 2022
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

An automated driving system includes an object detection system. A neural network image encoder generates image embeddings associated with an image including an object. A neural network text encoder generates concept embeddings associated with each of a plurality of concepts. Each of the plurality of concepts is associated with one of at least two object classes. A confidence score module generates a confidence score for each of the plurality of concepts based on the image embeddings and the concept embeddings associated with the concept. An object class prediction module generates a predicted object class of the object based on an association between a set of concepts of the plurality of concepts having at least two of the highest values of the generated confidence scores and the one of the at least two object classes associated with a majority of the set of concepts.

First claim

What is claimed is:

1. An automated driving system (ADS) of an autonomous vehicle including an object detection system, comprising:
a neural network image encoder configured to generate image embeddings associated with at least a portion of an image generated by a vehicle sensor system, the at least the portion of the image including an object;
a neural network text encoder configured to generate concept embeddings associated with each of a plurality of concepts, each of the plurality of concepts being associated with one of at least two object classes;
a confidence score module configured to generate a confidence score for each of the plurality of concepts based on the image embeddings and the concept embeddings associated with the concept;
an object class prediction module configured to generate a predicted object class of the object based on an association between a set of concepts of the plurality of concepts having at least two of the highest values of the generated confidence scores and the one of the at least two object classes associated with a majority of the set of concepts; and
a processor configured to manage performance of the autonomous vehicle based in part on the predicted object class of the object.

2. The system of claim 1 , wherein each of the plurality of concepts is associated with the one of the at least two object classes comprise a pre-defined concept.

3. The system of claim 1 , wherein the object classes of the at least two object classes comprise at least two of a car, a bus, a bicycle, a motorcycle, and a pedestrian.

4. The system of claim 1 , further comprising an object candidate module configured to detect the object in the image received from the vehicle sensor system and generate the at least the portion of the image based on a location of the detected object in the image.

5. The system of claim 4 , wherein the object candidate module is configured to implement one of a Fast Region-based Convolutional Network (Fast R-CNN) method and a You Look Only Once (YOLO) method.

6. The system of claim 1 , further comprising a feature extraction neural network including the neural network image encoder and the neural network text encoder, wherein the feature extraction neural network is configured to be pre-trained to connect the plurality of concepts with image data associated with each of the at least two object classes by a feature extraction neural network training system.

7. The system of claim 1 , wherein the confidence score module is configured to generate the confidence score for each of the plurality of concepts by applying a cosine similarity function to the image embeddings and the concept embeddings associated with the concept.

8. A non-transitory computer readable medium comprising instructions stored thereon for classifying a detected object in an image at an automated driving system (ADS), that upon execution by a processor, cause the processor to: generate image embeddings associated with at least a portion of an image generated by a vehicle sensor system, the at least the portion of the image including an object; generate concept embeddings associated with each of a plurality of concepts, each of the plurality of concepts being associated with one of at least two object classes; generate a confidence score for each of the plurality of concepts based on the image embeddings and the concept embeddings associated with the concept; generate a predicted object class of the object based on an association between a set of concepts of the plurality of concepts having at least two of the highest values of the generated confidence scores and the one of the at least two object classes associated with a majority of the set of concepts; and manage performance of the autonomous vehicle based in part on the predicted object class of the object.

9. The non-transitory computer readable medium of claim 8 , further comprising instructions to cause the processor to generate the concept embeddings associated with each of the plurality of concepts, each of the plurality of concepts associated with the one of the at least two object classes comprising a pre-defined concept.

10. The non-transitory computer readable medium of claim 8 , further comprising instructions to cause the processor to generate the concept embeddings associated with each of the plurality of concepts, each of the plurality of concepts being associated with the one of the at least two object classes comprising at least two of a car, a bus, a bicycle, a motorcycle, and a pedestrian.

11. The non-transitory computer readable medium of claim 8 , further comprising instructions to cause the processor to detect the object in the image received from the vehicle sensor system and generate the at least the portion of the image based on a location of the detected object in the image.

12. The non-transitory computer readable medium of claim 8 , further comprising instructions to cause the processor to detect the object in the image received from the vehicle sensor system using one of a Fast Region-based Convolutional Network (Fast R-CNN) method and a You Look Only Once (YOLO) method and generate the at least the portion of the image based on a location of the detected object in the image.

13. The non-transitory computer readable medium of claim 12 , further comprising instructions to cause the processor to pre-train a feature extraction neural network to connect the plurality of concepts with image data associated with each of the at least two object classes, the feature extraction neural network comprising a neural network image encoder configured to generate the image embeddings associated with the at least the portion of the image including the object and a neural network text encoder configured to generate the concept embeddings associated with each of the plurality of concepts.

14. The non-transitory computer readable medium of claim 8 , further comprising instructions to cause the processor to generate the confidence score for each of the plurality of concepts by applying a cosine similarity function to the image embeddings and the concept embeddings associated with the concept.

15. A method of classifying a detected object in an image at an automated driving system (ADS) of an autonomous vehicle comprising:
generating image embeddings associated with at least a portion of an image generated by a vehicle sensor system at a neural network image encoder, the at least the portion of the image including an object;
generating concept embeddings associated with each of a plurality of concepts at a neural network text encoder, each of the plurality of concepts being associated with one of at least two object classes;
generating a confidence score for each of the plurality of concepts based on the image embeddings and the concept embeddings associated with the concept at a confidence score module;
generating a predicted object class of the object based on an association between a set of concepts of the plurality of concepts having at least two of the highest values of the generated confidence scores and the one of the at least two object classes associated with a majority of the set of concepts at an object class prediction module; and
managing performance of the autonomous vehicle based in part on the predicted object class of the object at a controller.

16. The method of claim 15 , further comprising generating the concept embeddings associated with each of the plurality of concepts, each of the plurality of concepts associated with the one of the at least two object classes comprising a pre-defined concept.

17.

Assignees

• Gm Global Tech Operations Llc

Inventors

• Tong Wei
• Gupta Siddhartha
• Bond Jacob Alan
• Yuan Zhuoning

Classifications

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• B60W40/04

  Traffic conditions · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title