Deep learning based beam control for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-11701996-B2
Application number	US-202117528701-A
Country	US
Kind code	B2
Filing date	Nov 17, 2021
Priority date	Nov 17, 2021
Publication date	Jul 18, 2023
Grant date	Jul 18, 2023

Abstract

Provided are systems and methods for a deep learning based beam control. Sensor data associated with the environment and the corresponding detected objects from a perception system are obtained. Object features and image features are extracted. The extracted object features and image features are fused into fused features. A beam control status is predicted according to the fused features, wherein the beam control status indicates a high beam illumination intensity or a low beam illumination intensity of a light emitting device.

First claim

What is claimed is:

1. A system, comprising:
at least one processor; and
at least one memory storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to:
extract object features from a perception system output to identify objects in an environment using an object feature network, wherein the object feature network outputs object features associated with a map prior used to extract a distance associated with the objects, wherein the distance governs an illumination intensity output by a light emitting device;
extract image features from sensor data to identify environmental illumination information using an image feature network, wherein the image feature network outputs image features that comprise environmental information and data associated with an illumination of the environment;
fuse the object features and the image features into fused features using a feature fusion network that takes as input the object features and the image features and outputs fused features;
predict a beam control status according to the fused features, wherein the beam control status indicates a high beam illumination intensity or a low beam illumination intensity of the light emitting device; and

a control circuit communicatively coupled to the at least one processor, wherein the control circuit is configured to operate the light emitting device based upon the beam control status.

2. The system of claim 1 , wherein the at least one processor fuses the object features and the image features by applying a multilayer perceptron to concatenated object features and image features.

3. The system of claim 1 , wherein the object feature network is trained using data associated with manual driving and at least one corresponding output generated by the perception system.

4. The system of claim 1 , wherein the image feature network is trained using data associated with manual driving and at least one corresponding image sensor output generated by an image sensor.

5. The system of claim 1 , wherein the object feature network, image feature network, and feature fusion network are retrained during shadow mode to satisfy a predetermined safety confidence level.

6. The system of claim 1 , wherein the instructions stored in the memory further cause the at least one processor to:
determine a mismatch between a predicted beam control status of a fused feature and a corresponding manual driving data sample;
in response to a mismatch between the predicted beam control status and the corresponding manual driving data sample, identify the fused feature as a conflicting sample; and
modifying the object feature network, the image feature network, the fusion feature network, or any combinations thereof based on the conflicting sample.

7. A method, comprising:
extracting, using at least one processor, object features from a perception system output to identify objects in an environment using an object feature network, wherein the object feature network outputs object features associated with a map prior used to extract a distance associated with the objects, wherein the distance governs an illumination intensity output by a light emitting device;
extracting, using the at least one processor, image features from sensor data to identify environmental illumination information using an image feature network, wherein the image feature network outputs image features that comprise environmental information and data associated with an illumination of the environment;
fusing, using the at least one processor, the object features and the image features into fused features using a feature fusion network that takes as input the object features and the image features and outputs fused features;
predicting, using the at least one processor, a beam control status according to the fused features, wherein the beam control status indicates a high beam illumination intensity or a low beam illumination intensity of the light emitting device; and
operating, using the at least one processor, the light emitting device based upon the beam control status.

8. The method of claim 7 , wherein fusing the object features and the image features into fused features comprises applying a multilayer perceptron to concatenated object features and image features.

9. The method of claim 7 , wherein the object feature network is trained using data associated with manual driving and at least one corresponding output generated by the perception system.

10. The method of claim 7 , wherein the image feature network is trained using data associated with manual driving and at least one corresponding image sensor output generated by an image sensor.

11. The method of claim 7 , wherein the object feature network, image feature network, and feature fusion network are retrained during shadow mode to satisfy a predetermined safety confidence level.

12. The method of claim 7 , comprising:
determining, using the at least one processor, a mismatch between the predicted beam control status of a fused feature and a corresponding manual driving data sample;
in response to a mismatch between the predicted beam control status of the fused feature and the corresponding manual driving data sample, identifying, using the at least one processor, the fused feature as a conflicting sample; and
modifying, using the at least one processor, the object feature network, the image feature network, the fusion feature network, or any combinations thereof based on the conflicting sample.

13. A non-transitory computer program product storing instructions that, when executed by at least one programmable processor of a vehicle, cause the at least one programmable computer processor to perform operations comprising:
extracting object features from a perception system output to identify objects in an environment using an object feature network, wherein the object feature network outputs object features associated with a map prior used to extract a distance associated with the objects, wherein the distance governs an illumination intensity output by a light emitting device;
extracting image features from sensor data to identify environmental illumination information using an image feature network, wherein the image feature network outputs image features that comprise environmental information and data associated with an illumination of the environment;
fusing the object features and the image features into fused features using a feature fusion network that takes as input the object features and the image features and outputs fused features;
predicting a beam control status according to the fused features, wherein the beam control status indicates a high beam illumination intensity or a low beam illumination intensity of the light emitting device; and
operating the light emitting device based upon the beam control status.

14. The computer program product of claim 13 , wherein fusing the object features and the image features into fused features comprises applying a multilayer perceptron to concatenated object features and image features.

15. The computer program product of claim 13 , wherein the object feature network is trained using data associated with manual driving and at least one corresponding output generated by the perception system.

16. The computer program product of claim 13 , wherein the image feature network is trained using data associated with manual driving and at least one corresponding image sensor output generated by an image sensor.

17. The computer program product of claim 13 , wherein the object feature

Assignees

• Motional Ad Llc

Inventors

• Zhou Lubing
• Meng Xiaoli
• Shetti Karan Rajendra

Classifications

• B60Q1/085 Primary

  due to special conditions, e.g. adverse weather, type of road, badly illuminated road signs or potential dangers (B60Q1/10, B60Q1/12, B60Q1/1423 take precedence) · CPC title

• B60Q1/143

  combined with another condition, e.g. using vehicle recognition from camera images or activation of wipers · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title

• G06F2218/08

  Feature extraction · CPC title