Associating LIDAR data and image data

Patent metadata

Field	Value
Publication number	US-11816852-B2
Application number	US-202016940216-A
Country	US
Kind code	B2
Filing date	Jul 27, 2020
Priority date	May 3, 2018
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

First claim

What is claimed is:

1. A method comprising:
receiving LIDAR data and image data associated with an environment;
determining, based at least in part on the image data, a region of interest identifying a portion of the image data as being associated with an object in the environment;
generating scores for LIDAR points associated with the region of interest, wherein generating a first score of the scores for a corresponding first LIDAR point is based, at least in part, on a distance from a first point in the image data within the region of interest to a second point in the image data associated with a projection of the first LIDAR point into the image data, and wherein the first score is non-binary;
determining, based at least in part on the scores, a weighted median of the LIDAR points; and
determining, based at least in part on the weighted median, a first depth estimate associated with a distance from a sensor to the object.

2. The method of claim 1 , wherein generating the first score comprises determining, by a machine-learned model, a probability distribution associated with a depth measurement associated with the first LIDAR point.

3. The method of claim 2 , wherein determining the probability distribution by the machine-learned model comprises:
providing at least one of an object detection or a classification associated with the object detection as input to the machine-learned model; and
receiving, from the machine-learned model, the probability distribution.

4. The method of claim 2 , further comprising:
determining, based at least in part on the probability distribution, a probability density associated with the depth measurement associated with the first LIDAR point; and
determining, based at least in part on a distance between the first LIDAR point projected into the image data and a center of the region of interest, a factor.

5. The method of claim 4 , wherein generating the first score associated with the first LIDAR point is based at least in part on the factor and the probability density.

6. The method of claim 1 , further comprising controlling an autonomous vehicle based at least in part on the first depth estimate.

7. An apparatus comprising:
one or more processors; and
a memory storing processor-executable instructions that, when executed by the one or more processors, cause the apparatus to perform operations comprising:
receiving LIDAR data and image data associated with an environment;
receiving a region of interest identifying a portion of the image data as being associated with an object in the environment;
generating scores for LIDAR points associated with the region of interest, wherein generating a first score for a corresponding first LIDAR point is based at least in part on a distance from a first point within the region of interest to a second point in the image data associated with the first LIDAR point, and wherein the first score increases or decreases based at least in part on distance; and
determining, based at least in part on the scores, a first depth estimate associated with a distance from a sensor to the object.

8. The apparatus of claim 7 , wherein generating the first score is based at least in part on:
providing at least one of an object detection or a classification associated with the object detection as input to a machine-learned model; and
receiving, from the machine-learned model, a probability distribution associating various depths with probabilities.

9. The apparatus of claim 8 , wherein generating the first score is based at least in part on a probability density and a factor, the operations further comprising:
determining, based at least in part on the probability distribution, the probability density associated with a depth measurement associated with the first LIDAR point; and
determining, based at least in part on a distance between the first LIDAR point projected into the image data and a center of the region of interest, the factor.

10. The apparatus of claim 7 , wherein determining the first depth estimate comprises:
sorting the LIDAR points by distance; and
determining, based at least in part on the sorted LIDAR points and the scores, a weighted median as the first depth estimate, wherein weights associated with the weighted median are based at least in part on the scores.

11. The apparatus of claim 7 , wherein the operations further comprise:
identifying a subset of LIDAR points associated with distance values outside a range of depth values that are based at least in part on the first depth estimate;
sorting the subset of LIDAR points by distances associated with the subset of LIDAR points;
determining, based at least in part on scores associated with the subset and the sorting, a second weighted median; and
identifying, as a secondary depth estimate, a depth measurement associated with the second weighted median.

12. The apparatus of claim 11 , wherein the operations further comprise identifying the secondary depth estimate as being associated with a second object in the environment.

13. The apparatus of claim 7 , wherein the operations further generating instructions for controlling an autonomous vehicle based at least in part on the first depth estimate.

14. The apparatus of claim 7 , wherein generating the first score comprises determining that a projection of the first LIDAR point is within the region of interest and determining the distance from the point within the region of interest to a projected point associated with the first LIDAR point.

15. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving LIDAR data and image data associated with an environment;
receiving a region of interest associated with an object in the environment;
generating scores for LIDAR points associated with the region of interest, wherein generating a first score for a corresponding first LIDAR point is based at least in part on a distance from a first point within the region of interest to a second point in the region of interest associated with the first LIDAR point, and wherein the first score increases or decreases based at least in part on distance; and
determining, based at least in part on the scores, a first depth estimate associated with a distance from a sensor to the object.

16. The non-transitory computer-readable medium of claim 15 , wherein generating the first score is based at least in part on:
providing at least one of a portion of image data associated with the region of interest or a classification associated with the portion of image data as input to a machine-learned model; and
receiving, from the machine-learned model, a probability distribution identifying at least a probability associated with a particular depth.

17. The non-transitory computer-readable medium of claim 16 , wherein generating the first score is based at least in part on a probability density and a factor, the operations further comprising:
determining, based at least in part on the probability distribution, the probability density associated with a depth measurement associated with the first LIDAR point; and
determining, based at least in part on a distance between the first LIDAR point projected into the image data and a center of the region of interest, the factor.

18. The non-transitory computer-readable medium of claim 15 , wherein determining the first depth estimate comprises:
sorting the LIDAR points by distance; and
d

Assignees

• Zoox Inc

Inventors

• Lee Tencia
• Pradhan Sabeek Mani
• Anguelov Dragomir Dimitrov

Classifications

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G01S2007/4975

  of sensor obstruction by, e.g. dirt- or ice-coating, e.g. by reflection measurement on front-screen · CPC title