Autonomous vehicle localization using image analysis and manipulation

Patent metadata

Field	Value
Publication number	US-10678262-B2
Application number	US-201715640340-A
Country	US
Kind code	B2
Filing date	Jun 30, 2017
Priority date	Jul 1, 2016
Publication date	Jun 9, 2020
Grant date	Jun 9, 2020

Abstract

A system to use submaps to control operation of a vehicle is disclosed. A storage system may be provided with a vehicle to store a collection of submaps that represent a geographic area where the vehicle may be driven. A programmatic interface may be provided to receive submaps and submap updates independently of other submaps.

First claim

What is claimed is:

1. A method for performing localization for a vehicle in motion, the method being implemented by one or more processors and comprising:
accessing a collection of imagelets for a geographic region that encompasses multiple predefined areas of a road network, each imagelet depicting at least one feature present in a corresponding area of the multiple predefined areas, wherein each imagelet is stored with reference location information identifying a reference location of a prior sensor device determined when the prior sensor device captured prior sensor data for the imagelet;
obtaining current sensor data from one or more sensor devices of the vehicle, the current sensor data including (i) image data captured by one or more cameras of the vehicle, (ii) data that identifies a current location of the vehicle, and (iii) data that identifies current environmental conditions;
selecting a set of imagelets from the collection of imagelets based at least in part on the current location of the vehicle;
for at least one imagelet in the selected set of imagelets, upon detecting a disparity between the current environmental conditions and prior environmental conditions that were present when the prior sensor data for the at least one imagelet was captured, applying a transformation to at least one of the image data or the at least one imagelet to account for the disparity;
matching, to the image data, one of the features depicted in the at least one imagelet of the selected set of imagelets; and
updating the current location for a control system of the vehicle based at least in part on the reference location of the prior sensor device that captured the prior sensor data for the at least one imagelet.

2. The method of claim 1 , wherein accessing the collection of imagelets includes accessing a collection of submaps for the geographic region, each submap corresponding to an area of the geographic region, wherein each of the submaps of the collection include one or more of the imagelets of the collection of imagelets.

3. The method of claim 2 , wherein each submap includes at least one data layer that provides a pointcloud of imagelets captured by a sensor device that previously traversed the area of that submap.

4. The method of claim 2 , wherein selecting the set of one or more imagelets includes selecting a subset of imagelets provided with one or more pointclouds of imagelets in an individual submap of the collection of submaps.

5. The method of claim 1 , wherein the current environmental conditions and the prior environmental conditions include lighting conditions.

6. The method of claim 1 , wherein the current environmental conditions and the prior environmental conditions include variance in a time of day, a day of season, or weather condition.

7. The method of claim 1 , wherein applying the transformation includes determining a model for the transformation and applying the transformation based on the model, wherein the model is trained on sets of prior sensor data previously collected from one or more vehicles that obtained the sets of prior sensor data at the current location of the vehicle.

8. The method of claim 1 , wherein applying the transformation includes determining a model for the transformation and applying the transformation based on the model, wherein the model is trained on sets of prior sensor data previously collected from one or more vehicles that obtained the sets of prior sensor data on a road or sub-region that contains the current location of the vehicle.

9. The method of claim 1 , wherein applying the transformation includes determining a model for the transformation and applying the transformation based on the model, wherein the model is trained on sets of prior sensor data previously collected from one or more vehicles that obtained the sets of prior sensor data on a different road than the road that includes the current location of the vehicle.

10. The method of claim 9 , wherein the different road is intersecting or adjacent to the road that includes the current location of the vehicle.

11. The method of claim 1 , further comprising determining an orientation of the vehicle relative to a direction of travel as the vehicle progresses through a given area.

12. The method of claim 1 , wherein updating the current location of the vehicle includes determining a lane or lateral position of the vehicle on a road segment.

13. A computer system comprising:
a memory to store a set of instructions;
one or more processors to use the set of instructions to:
access a collection of imagelets for a geographic region that encompasses multiple predefined areas of a road network, each imagelet depicting at least one feature present in a corresponding area of the multiple predefined areas, wherein each imagelet is stored with reference location information identifying a reference location of a prior sensor device determined when the prior sensor device captured prior sensor data for the imagelet;
obtain current sensor data from one or more sensor devices of a vehicle, the current sensor data including (i) image data captured by one or more cameras of the vehicle, (ii) data that identifies a current location of the vehicle, and (iii) data that identifies current environmental conditions;
select a set of imagelets from the collection of imagelets based at least in part on the current location of the vehicle;
for at least one imagelet in the selected set of imagelets, upon detecting a disparity between the current environmental conditions and prior environmental conditions that were present when the prior sensor data for the at least one imagelet was captured, apply a transformation to at least one of the image data or the at least one imagelet to account for the disparity;
match, to the image data, one of the features depicted in the at least one imagelet of the selected set of imagelets; and
update the current location for a control system of the vehicle based at least in part on the reference location of the prior sensor device that captured the prior sensor data for the at least one imagelet.

14. The computer system of claim 13 , wherein the vehicle is an autonomous vehicle, and the computer system is provided on the autonomous vehicle.

15. The computer system of claim 13 , wherein the vehicle is an autonomous vehicle, and the computer system communicates with the autonomous vehicle over a network.

Assignees

• Uatc Llc

Inventors

• Browning Brett
• Hansen Peter
• Eade Ethan
• Prasser David
• Larose David
• Zlot Robert

Classifications

• G05D1/0276

  using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058) · CPC title

• G01C21/28

  with correlation of data from several navigational instruments · CPC title

• G06K9/6202

  Physics · mapped topic

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• G06T2207/10012

  Stereo images · CPC title