Sensor dirtiness detection

Patent metadata

Field	Value
Publication number	US-10836356-B2
Application number	US-201816136631-A
Country	US
Kind code	B2
Filing date	Sep 20, 2018
Priority date	Sep 20, 2018
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

An occluded area on a vehicle sensor is identified by localizing the vehicle in space. A discrepancy between historic image data and real-time image data from the sensor is determined, and a cleaning actuator is actuated based on the determined discrepancy.

First claim

What is claimed is:

1. A method, comprising:
receiving historic image data including a classification from a plurality of classifications for respective points in the image data, wherein the plurality of classifications include at least one of a flat surface, a human, a vehicle, a construction, an object, or a nature class;
identifying an occluded area on a vehicle sensor that collects real-time image data by localizing the vehicle in space;
determining the discrepancy between the historic image data and the real-time image data from the sensor based at least in part on the classification of a point in the real-time image data and a classification of a corresponding point in the historic image data; and
actuating a cleaning actuator based on the determined discrepancy.

2. The method of claim 1 , wherein the historic image data is based on data from at least on one of a second vehicle and a second sensor of the vehicle.

3. The method of claim 1 , wherein the historic image data includes 3D location coordinates.

4. The method of claim 1 , wherein determining the discrepancy further includes:
performing a perspective transformation of historic image data based on vehicle location coordinates and a vehicle orientation;
identifying first feature points in the real-time image data and second feature points in the historic image data;
performing a homography that includes a line-preserving projective mapping for the first and second feature points; and
identifying a first portion of the real-time image data matching a second portion of the historic image data.

5. The method of claim 1 , further comprising:
identifying a classification of each feature in the historic image data;
selecting static features based on the classification of the features;
determining false positive and true positive classifications of the static features based on the real-time image data; and
determine the discrepancy based on the determined true positive and false classifications and a confusion matrix including an average expected rate of misclassification for each class of features.

6. The method of claim 5 , further comprising:
determining a local discrepancy value for a location on a transparency of the sensor and a global discrepancy value for the transparency; and
actuating the cleaning actuator upon determining that a difference between an average of the local discrepancy and the global discrepancy exceeds a threshold.

7. The method of claim 5 , wherein the static feature is a feature of at least one of a flat, construction, and object classes.

8. A system, comprising a processor and a memory, the memory storing instructions executable by the processor to:
receive historic image data including a classification from a plurality of classifications for respective points in the image data, wherein the plurality of classifications include at least one of a flat surface, a human, a vehicle, a construction, an object, or a nature class;
identify an occluded area on a vehicle sensor that collects real-time image data by localizing the vehicle in space;
determine the discrepancy between the historic image data and the real-time image data from the sensor based at least in part on the classification of a point in the real-time image data and a classification of a corresponding point in the historic image data; and
actuate a cleaning actuator based on the determined discrepancy.

9. The system of claim 8 , wherein the vehicle sensor includes a camera sensor, and the instructions further include instructions to identify the occluded area in an optical path of the camera sensor.

10. The system of claim 9 , wherein the optical path includes at least one of a lens and a transparent exterior cover.

11. The system of claim 9 , wherein the occluded area is an area in the optical path of the vehicle sensor that is covered by at least one of fog, water, smudge, dust, and scratch.

12. The system of claim 9 , wherein the occluded area is an area of the optical path where an optical attribute of the optical path deviates from a specified optical property; the optical property including at least one of a focal point and a distortion.

13. A system, comprising:
a vehicle camera sensor having an optical path; and
a processor programmed to:
receive historic image data including a classification from a plurality of classifications for respective points in the image data, wherein the plurality of classifications include at least one of a flat surface, a human, a vehicle, a construction, an object, or a nature class;
identify identify an occluded area on a vehicle sensor that collects real-time image data by localizing the vehicle in space;
determine determine the discrepancy between the historic image data and the real-time image data from the sensor based at lease in part on the classification of a point in the real-time image data and a classification of a corresponding point in the historic image data; and
actuate a cleaning actuator based on the determined discrepancy.

14. The system of claim 13 , wherein the occluded area is an area in an optical path of the vehicle sensor that is covered by at least one of fog, water, smudge, dust, and scratch.

15. The system of claim 13 , wherein the occluded area is an area of the optical path where an optical attribute of the optical path deviates from a specified optical property; the optical property including at least one of a focal point and a distortion.

16. The system of claim 8 , wherein the instructions further instructions to:
identifying a classification of each feature in the historic image data;
selecting static features based on the classification of the features;
determining false positive and true positive classifications of the static features based on the real-time image data; and
determine the discrepancy based on the determined true positive and false classifications and a confusion matrix including an average expected rate of misclassification for each class of features.

17. The system of claim 16 , wherein the instructions further include instructions to:
determine a local discrepancy value for a location on a transparency of the sensor and a global discrepancy value for the transparency; and
actuate the cleaning actuator upon determining that a difference between an average of the local discrepancy and the global discrepancy exceeds a threshold.

18. The system of claim 16 , wherein the static feature is a feature of at least one of a flat, construction, and object classes.

Assignees

• Ford Global Tech Llc

Inventors

• Herman david michael
• Krishnan Venkatesh
• Patil Sunil

Classifications

• G02B27/0006 Primary

  with means to keep optical surfaces clean, e.g. by preventing or removing dirt, stains, contamination, condensation (G02B1/18 takes precedence; cleaning in general B08B) · CPC title

• B60S1/0848 Primary

  Cleaning devices for cameras on vehicle · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/993

  Evaluation of the quality of the acquired pattern · CPC title

• G06T2207/30168

  Image quality inspection · CPC title