Image-based depth data and localization

Patent metadata

Field	Value
Publication number	US-11087494-B1
Application number	US-201916408407-A
Country	US
Kind code	B1
Filing date	May 9, 2019
Priority date	May 9, 2019
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
accessing lidar-based map data of an environment, the lidar-based map data comprising a three-dimensional mesh associated with the environment;
capturing, by a sensor of an autonomous vehicle, image data associated with the environment;
inputting the image data to a machine-learned model;
receiving, from the machine-learned model, depth data associated with the image data, wherein the depth data comprising a point cloud associated with the environment; and
determining a location of the autonomous vehicle in the environment by comparing a first portion of the depth data to a second portion of the lidar-based map data.

2. The system of claim 1 , the operations further comprising:
controlling, based at least in part on the location, the autonomous vehicle.

3. The system of claim 1 , wherein the depth data is first depth data, and wherein the operations further comprise:
receiving second depth data captured by a lidar sensor.

4. The system of claim 1 , wherein the depth data is first depth data, the operations further comprising:
determining, based at least in part on the image data, a third portion of the image data representing an object in the environment;
determining a classification associated with the object;
determining, based at least in part on the classification, a fourth portion of the first depth data; and
determining, based at least in part on discarding the fourth portion of the depth data, second depth data corresponding to the third portion of the image data,
wherein determining the location of the autonomous vehicle comprises comparing the second depth data to the second portion of the lidar-based map data.

5. The system of claim 4 , wherein determining the classification associated with the object comprises:
determining an object type associated with the object, the object type comprising at least one of a vehicle, a cyclist, a pedestrian, or an animal.

6. A method comprising:
accessing map data of an environment;
receiving image data from an image sensor on a vehicle;
inputting at least a first portion of the image data to a machine-learned model trained to determine depth data based on the first portion of the image data;
receiving, from the machine-learned model, the depth data comprising a point cloud associated with the environment; and
determining, based at least in part on comparing a second portion of the depth data to a third portion of the map data, a location of the vehicle in the environment;
wherein the machine-learned model is trained based at least in part on inputting captured depth data captured by a depth sensor, and wherein the captured depth data corresponds to ground truth data.

7. The method of claim 6 , wherein determining the location of the vehicle is based at least in part on a localization algorithm comprising at least one of:
an iterative closest point algorithm,
a robust point matching algorithm,
a kernel correlation algorithm,
a coherent point drift algorithm, or
a sorting correspondence space algorithm.

8. The method of claim 6 , wherein the vehicle is an autonomous vehicle, the method further comprising:
controlling, based at least in part on the location, the autonomous vehicle.

9. The method of claim 6 , wherein the map data comprises at least one of:
mesh map data, or
voxel-based map data.

10. The method of claim 6 , wherein the depth data further comprises: surface normal data associated with a static object of the environment.

11. The method of claim 6 , wherein the depth data is based at least in part on a discrete depth bin and a continuous offset associated with the discrete depth bin.

12. The method of claim 6 , wherein the location is a first location, the method further comprising:
receiving, from a lidar sensor on the vehicle, lidar data;
determining, based at least in part on at least a portion of the lidar data to the map data, a second location of the vehicle in the environment;
determining a difference between the first location and the second location; and
determining that the difference meets or exceeds a threshold value.

13. The method of claim 12 , further comprising:
determining, based at least in part on the difference meeting or exceeding the threshold value, a calibration adjustment associated with the image sensor on the vehicle.

14. A non-transitory computer-readable medium storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving lidar-based map data of an environment;
receiving, from a sensor on a vehicle, image data;
determining, based at least in part on the image data, depth data associated with the image data; and
determining, based at least in part on the depth data and the lidar-based map data, a location of the vehicle in the environment.

15. The non-transitory computer-readable medium of claim 14 , wherein the depth data comprises:
surface normal data that indicates a vector associated with a surface of a static object of the environment.

16. The non-transitory computer-readable medium of claim 14 , the operations further comprising:
inputting at least a portion of the image data to a machine-learned model; and
receiving, from the machine-learned model, the depth data.

17. The non-transitory computer-readable medium of claim 14 , wherein the depth data is based at least in part on a discrete depth bin and a continuous offset associated with the discrete depth bin, wherein the discrete depth bin is based at least in part on a non-linear scale.

18. The non-transitory computer-readable medium of claim 14 , wherein the sensor is a first sensor and the location is a first location, the operations further comprising:
receiving, from a second sensor on the vehicle, lidar data;
determining, based at least in part on at least a portion of the lidar data and the lidar-based map data, a second location of the vehicle in the environment;
determining a difference between the first location and the second location; and
determining that the difference meets or exceeds a threshold value.

19. The non-transitory computer-readable medium of claim 18 , the operations further comprising:
determining, based at least in part on the difference, a calibration adjustment associated with at least one of the first sensor or the second sensor.

Assignees

• Zoox Inc

Inventors

• Srinivasan Praveen

Classifications

• G06V20/584

  of vehicle lights or traffic lights · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/30244

  Camera pose · CPC title