Perception system diagnosis using predicted sensor data and perception results

Patent metadata

Field	Value
Publication number	US-11829128-B2
Application number	US-201916661126-A
Country	US
Kind code	B2
Filing date	Oct 23, 2019
Priority date	Oct 23, 2019
Publication date	Nov 28, 2023
Grant date	Nov 28, 2023

Abstract

A perception system includes a perception module configured to capture first sensor data that includes data from at least one of an external sensor and a camera captured in a first period, a prediction module configured to receive the first sensor data, generate, based on the first sensor data, predicted sensor data for a second period subsequent to the first period, receive second sensor data for the second period, and output results of a comparison between the predicted sensor data and the second sensor data, and a diagnostic module configured to selectively identify a fault in the perception system based on the results of the comparison.

First claim

What is claimed is:

1. A perception system for a vehicle, comprising:
a perception circuit configured to capture a first sensor measurement, wherein the first sensor measurement includes data directly received from a first sensor captured in a first period,
the first sensor being a camera of the vehicle;
a prediction circuit configured to:
(i) receive the first sensor measurement captured in the first period,
(ii) prior to capturing a second sensor measurement in a second period subsequent to the first period using the first sensor, generate a predicted sensor measurement from the first sensor for the second period using the first sensor measurement,
(iii) in the second period, directly receive the second sensor measurement captured by the first sensor in the second period, and
(iv) output results of a comparison between the predicted sensor measurement and the second sensor measurement, wherein the results include a mathematical difference involving a subtraction operation between the predicted sensor measurement and the second sensor measurement;

a diagnostic circuit configured to selectively identify a fault in the perception system based on the mathematical difference between the predicted sensor measurement and the second sensor measurement; and
at least one of:
an engine control module configured to control torque output of an engine based on input from the perception circuit;
a power inverter module configured to control power flow to and from an electric motor based on the input from the perception circuit;
a braking control module configured to adjust braking of the vehicle based on the input from the perception circuit; and
a steering control module configured to adjust steering based on the input from the perception circuit.

2. The perception system of claim 1 , wherein the first sensor measurement further includes historical sensor measurements captured in periods prior to the first period.

3. The perception system of claim 2 , wherein the prediction circuit is configured to generate the predicted sensor measurement further based on the historical sensor measurements.

4. The perception system of cairn , wherein the prediction circuit includes a convolutional long short term memory network.

5. The perception system of claim 1 , wherein the perception circuit is further configured to generate perception results based on the second sensor measurement, wherein the perception results include labeled objects and locations.

6. The perception system of claim 5 , wherein the prediction circuit is configured to generate predicted perception results corresponding to the predicted sensor measurement.

7. The perception system of claim 6 , wherein the prediction circuit is configured to (i) compare the predicted perception results to the perception results and (ii) provide results of the comparison between the predicted perception results and the perception results to the diagnostic circuit.

8. The perception system of claim 7 , wherein the diagnostic circuit is configured to selectively identify the fault in the perception system further based on the results of the comparison between the predicted perception results and the perception results.

9. The perception system of claim 1 , wherein the fault corresponds to a fault in the first sensor.

10. A method of operating a perception system for a vehicle, the method comprising:
capturing a first sensor measurement, wherein the first sensor measurement includes data directly received from a first sensor captured in a first period, wherein the first sensor is a camera of the vehicle;
prior to capturing a second sensor measurement in a second period subsequent to the first period using the first sensor, generating, a predicted sensor measurement from the first sensor for the second period using the first sensor measurement;
in the second period, directly receiving the second sensor measurement captured by the first sensor in the second period;
outputting results of a comparison between the predicted sensor measurement and the second sensor measurement, wherein the results include a mathematical difference involving a subtraction operation between the predicted sensor measurement and the second sensor measurement;
selectively identifying a fault in the perception system based on the mathematical difference between the predicted sensor measurement and the second sensor measurement; and
at least one of:
controlling torque output of an engine based on input from the perception system;
controlling power flow to and from an electric motor based on the input from the perception system;
adjusting braking of the vehicle based on the input from the perception system; and
adjusting steering based on the input from the perception system.

11. The method of claim 10 , wherein the first sensor measurement further includes historical sensor measurements captured in periods prior to the first period.

12. The method of claim 11 , further comprising generating the predicted sensor measurement further based on the historical sensor measurements.

13. The method of claim 10 , further comprising generating the predicted sensor measurement using a convolutional long short term memory network.

14. The method of claim 10 , further comprising generating perception results based on the second sensor measurement, wherein the perception results include labeled objects and locations.

15. The method of claim 14 , further comprising generating predicted perception results corresponding to the predicted sensor measurement.

16. The method of claim 15 , further comprising comparing the predicted perception results to the perception results and selectively identifying the fault in the perception system further based on the results of the comparison between the predicted perception results and the perception results.

17. The method of claim 10 , wherein the fault corresponds to a fault in the first sensor.

18. A perception system for a vehicle, the perception system comprising:
a first sensor arranged to provide data corresponding to an environment surrounding the vehicle, wherein the first sensor is a camera;
a perception circuit configured to
capture a first sensor measurement, wherein the first sensor measurement includes data directly captured from the first sensor in a first period,
capture a second sensor measurement, wherein the second sensor measurement includes data directly captured from the first sensor in a second period subsequent to the first period, and
generate perception results based on the second sensor measurement, wherein the perception results include labeled objects and locations;

a prediction circuit configured to (i) receive the first sensor measurement captured in the first period, the second sensor measurement captured in the second period, and the perception results, (ii) generate a predicted sensor measurement from the first sensor for the second period, (iii) generate, based on the predicted sensor measurement, predicted perception results, and (iv) output results of a first comparison between the predicted sensor measurement and the second sensor measurement and a second comparison between the predicted perception results and the perception results, wherein the results include a mathematical difference involving a subtraction operation between the predicted sensor measurement and the second sensor measurement;
a diagnostic circuit configured to selectively identify a fault in the perception system based on the results of the first comparison, the second comparison, and the mathematical di

Assignees

• Gm Global Tech Operations Llc

Inventors

• Hu Yao
• Tong Wei
• Lin Wen-Chiao

Classifications

• G05B23/024 Primary

  Quantitative history assessment, e.g. mathematical relationships between available data; Functions therefor; Principal component analysis [PCA]; Partial least square [PLS]; Statistical classifiers, e.g. Bayesian networks, linear regression or correlation analysis; Neural networks · CPC title

• G05B23/0283

  Predictive maintenance, e.g. involving the monitoring of a system and, based on the monitoring results, taking decisions on the maintenance schedule of the monitored system; Estimating remaining useful life [RUL] (preventive maintenance, i.e. planning maintenance according to the available resources without monitoring the system G06Q10/06) · CPC title

• G05D1/0055

  with safety arrangements · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0094

  involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target · CPC title