Camera assessment techniques for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-11227409-B1
Application number	US-201816105084-A
Country	US
Kind code	B1
Filing date	Aug 20, 2018
Priority date	Aug 20, 2018
Publication date	Jan 18, 2022
Grant date	Jan 18, 2022

Abstract

The disclosure relates to assessing operation of a camera. In one instance, a volume of space corresponding to a first vehicle in an environment of a second vehicle may be identified using sensor data generated by a LIDAR system of the second vehicle. An image captured by a camera of the second vehicle may be identified. The camera may have an overlapping field of view of the LIDAR system at a time when the sensor data was generated. An area of the image corresponding to the volume of space may be identified and processed in order to identify a vehicle light. The operation of the camera may be assessed based on the processing.

First claim

The invention claimed is:

1. A method for assessing operation of a camera, the method comprising:
identifying, by one or more processors, a volume of space corresponding to a first vehicle in an environment of a second vehicle using sensor data generated by a light detection and ranging (LIDAR) system of the second vehicle;
identifying, by the one or more processors, an image captured by a camera of the second vehicle, the camera having a field of view that overlaps with a portion of a field of view of the LIDAR system at a time when the sensor data was generated;
projecting, by the one or more processors, a three-dimensional (3D) bounding box into the image in order to identify an area of the image corresponding to the volume of space;
determining, by the one or more processors based on the identified area of the image, a result indicating whether there is a difference in color of an edge of the 3D bounding box projected into the image that meets a color difference threshold, wherein groups of pixels on each side of the edge are processed to identify actual or average differences in color between the groups of pixels on each side of the edge; and
assessing, by the one or more processors, the operation of the camera based on the result.

2. The method of claim 1 , wherein the 3D bounding box corresponds to the volume of space, and the area is a two-dimensional (2D) polygon.

3. The method of claim 2 , wherein the 3D bounding box is projected into the image in order to identify the 2D polygon.

4. The method of claim 2 , wherein the polygon is not a rectangle.

5. The method of claim 1 , wherein the camera includes a neutral density (ND) filter.

6. The method of claim 1 , wherein the image includes a vehicle tail light.

7. The method of claim 1 , further comprising:
processing, by the one or more processors, one or more sub-areas within the area in order to identify a vehicle light based on a model of a vehicle selected from a plurality of models, wherein the selection of the model is based on a type of the vehicle, the model indicating expected areas of vehicle lights,
wherein the processing the one or more sub-areas within the area includes using an image classifier in order to identify one or more areas of local maximum brightness above a brightness threshold.

8. The method of claim 1 , further comprising:
processing, by the one or more processors, one or more sub-areas within the area in order to identify a vehicle light based on a model of a vehicle selected from a plurality of models, wherein the selection of the model is based on a type of the vehicle, the model indicating expected areas of vehicle lights,
wherein the processing the one or more sub-areas within the area includes identifying contrast areas indicative of bright objects in the area.

9. The method of claim 1 , wherein the identifying the image is further based on ambient lighting conditions when the sensor data was generated.

10. The method of claim 1 , wherein the identifying the image is further based on a time of day when the sensor data was generated.

11. The method of claim 1 , further comprising, based on the assessment, attempting to clean a lens of the camera.

12. The method of claim 1 , wherein the second vehicle is at least one of a car, a truck, a motorcycle, a bus or a recreational vehicle.

13. The method of claim 1 , further comprising:
reducing, by the one or more processors, a size of the image to include only portions of the image corresponding to the overlapping field of view of the camera,
wherein the size of the image is reduced by performing at least one of thumbnailing or cropping.

14. A computing device for assessing operation of a camera, the computing device comprising:
a memory configured to store instructions and data;
one or more processors coupled to the memory, the one or more processors configured to:
access the instructions and data stored in the memory;
identify a volume of space corresponding to a first vehicle in an environment of a second vehicle using sensor data generated by a light detection and ranging (LIDAR) system of the second vehicle;
identify an image captured by a camera of the second vehicle, the camera having a field of view that overlaps with a portion of a field of view of the LIDAR system at a time when the sensor data was generated;
project a three-dimensional (3D) bounding box into the image in order to identify an area of the image corresponding to the volume of space;
determine, based on the identified area of the image, a result indicating whether there is a difference in color of an edge of the 3D bounding box projected into the image that meets a color difference threshold, wherein groups of pixels on each side of the edge are processed to identify actual or average differences in color between the groups of pixels on each side of the edge; and
assess the operation of the camera based on the result.

15. The computing device of claim 14 , wherein the computing device is comprised by the second vehicle.

Assignees

• Waymo Llc

Inventors

• Wu Chen
• Craddock Carl Warren
• Wendel Andreas

Classifications

• G06F18/25

  Fusion techniques · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/931

  of land vehicles · CPC title

• H04N17/002

  for television cameras · CPC title