Associating LIDAR data and image data

Patent metadata

Field	Value
Publication number	US-10726567-B2
Application number	US-201815970838-A
Country	US
Kind code	B2
Filing date	May 3, 2018
Priority date	May 3, 2018
Publication date	Jul 28, 2020
Grant date	Jul 28, 2020

Abstract

A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to:
receive an image of an environment from an image sensor;
determine, based at least in part on the image, a region of interest that identifies a portion of the image as representing an object in the environment;
receive LIDAR points from a LIDAR device, the LIDAR points associated with the region of interest and a time the image was captured;
generate scores for the LIDAR points, wherein generating a score for the LIDAR point comprises:
determining, based at least in part on a probability distribution generated by a monocular image model, a probability density associated with the depth measurement associated with the LIDAR point; and
determining a factor, based at least in part on a distance in pixels between the LIDAR point projected into the image and a center of the region of interest; and

determine, using a weighted median calculation, a primary depth estimate of the object, wherein weights associated with the weighted median calculation comprise the scores.

2. The system as claim 1 recites, the instructions further cause the system to:
select, as a subset of LIDAR points, LIDAR points that are associated with depth measurements that are within a range of the primary depth estimate;
determine a second weighted median of the sorted LIDAR points; and
determine, based at least in part on the second weighted median, a secondary depth estimate of the object.

3. The system as claim 2 recites, wherein the system comprises an autonomous vehicle, the camera and LIDAR being on the autonomous vehicle, and wherein the instructions further cause the system to:
identify, based at least in part on the primary depth estimate or the secondary depth estimate, a position of the object in the environment; and
generate, based at least in part on the position of the object, a trajectory for controlling motion of the autonomous vehicle.

4. The system as claim 2 recites, wherein the instructions further cause the system to:
compare the primary depth estimate and the secondary depth estimate to an output of a monocular image model;
compare a first density of LIDAR points associated with the primary depth estimate to a second density of LIDAR points associated with the secondary depth estimate; or
compare the primary depth estimate and the secondary depth estimate to an object track associated with the object.

5. The system as claim 1 recites, wherein generating the score for the LIDAR point comprises multiplying the probability density by the factor.

6. A computer-implemented method of determining a distance from an image plane to an object, the method comprising:
receiving LIDAR data and image data of an environment;
determining a region of interest associated with the object detected in the environment;
determining LIDAR points of the LIDAR data that correspond to the region of interest;
generating scores for the LIDAR points, wherein generating a score for a LIDAR point comprises:
determining a factor based at least in part on a distance from a center of the region of interest to a projection of the LIDAR point onto the image;
determining a probability density of a depth measurement associated with the LIDAR point; and
generating the score based at least in part on the probability density and the factor;

determining, based at least in part on the scores, a weighted median of the LIDAR points; and
identifying, as a primary depth estimate, a depth measurement associated with the weighted median.

7. The computer-implemented method as claim 6 recites, wherein determining the factor comprises evaluating a Gaussian centered at the center of the region of interest using the projection of the LIDAR point onto the image.

8. The computer-implemented method as claim 6 recites, wherein determining the probability density comprises generating a probability distribution over a range of depths, via a machine-learning model and based at least in part on a classification of the object.

9. The computer-implemented method of claim 6 , wherein generating the score comprises multiplying the probability density by the factor.

10. The computer-implemented method as claim 6 recites, further comprising:
identifying a subset of LIDAR points associated with distances that meet or exceed a range of depth values that comprises the primary depth estimate;
sorting the subset of LIDAR points by distances associated with the subset of LIDAR points;
determining, based at least in part on scores associated with the subset and the sorting, a second weighted median; and
identifying, as a secondary depth estimate, a depth measurement associated with the second weighted median.

11. The computer-implemented method as claim 10 recites, wherein the range of depth values varies from a point 0.8 meters less than the primary depth estimate to 1.6 meters more than the primary depth estimate.

12. The computer-implemented method as claim 10 recites, further comprising:
choosing, as an output depth, the primary depth estimate or the secondary depth estimate based at least in part on at least one of:
comparing a first probability density or a first probability associated with the first depth estimate by evaluating the probability distribution using the first depth estimate, to a second probability density or a second probability associated with the second depth estimate by evaluating the probability distribution using the second depth estimate;
comparing a first density of LIDAR points associated with the primary depth estimate to a second density of LIDAR points associated with the secondary depth; or
comparing the primary depth estimate and the secondary depth estimate to an object track associated with the object.

13. The computer-implemented method as claim 12 recites, wherein choosing the secondary depth estimate further comprises:
indicating an existence of an occluding object that occludes at least part of the object; and
associating the primary depth estimate with the occluding object and the secondary depth estimate with the object.

14. The computer-implemented method as claim 12 recites, further comprising:
sending the output depth to a controller of an autonomous vehicle; and
generating, based at least in part on the output depth, a trajectory, the trajectory configured to cause the autonomous vehicle to traverse a portion of the environment.

15. A non-transitory computer-readable medium having a set of instructions that, when executed, cause one or more processors to perform operations comprising:
receiving, from a camera, an image of an environment that comprises an object;
receiving a region of interest representing a location of the object in the image;
receiving, from a point cloud sensor, point cloud data;
determining, from the point cloud data, point cloud points that correspond with the region of interest;
determining, based at least in part on the image, a probability distribution of depths;
generating, based at least in part on relative coordinates of the point cloud points in an image space associated with the image and based at least in part on a position of the point cloud points relative to depths specified by the probability distribution, scores for the point cloud points;
determining, by a weighted median calculation, a weighted median based at least in part on the scores; and
identifying a depth measurement associated with the weighted median as a

Assignees

• Zoox Inc

Inventors

• Lee Tencia
• Pradhan Sabeek Mani
• Anguelov Dragomir Dimitrov

Classifications

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G01S2007/4975

  of sensor obstruction by, e.g. dirt- or ice-coating, e.g. by reflection measurement on front-screen · CPC title