Deep direct localization from ground imagery and location readings

Patent metadata

Field	Value
Publication number	US-10825201-B2
Application number	US-201815900060-A
Country	US
Kind code	B2
Filing date	Feb 20, 2018
Priority date	Feb 20, 2018
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

In one embodiment, a method includes training an image localization model for determining an adjusted location reading for a vehicle based on its location reading and the associated image. The method further includes updating a location associated with the vehicle using the adjusted location reading. The training of the image localization model includes generating feature representations of a number of training images, and learning a relationship between the feature representations and differentials between a number of raw location readings and their respective reference location readings.

First claim

What is claimed is:

1. A method comprising, by a computing system:
receiving an image and a location reading associated with a vehicle, the image and the location reading being captured substantially contemporaneously by a computing device associated with the vehicle;
determining an adjusted location reading associated with the vehicle by processing the image and the location reading using a trained image localization model, wherein the trained image localization model is a machine-learning model configured to output a differential between the location reading and the adjusted location reading, wherein the machine-learning model is trained by:
generating feature representations of a plurality of training images, wherein the plurality of training images are respectively associated with a plurality of raw location readings and a plurality of reference location readings; and
learning a relationship between (1) the feature representations of the plurality of training images and (2) differentials between the plurality of raw location readings and the plurality of reference location readings; and

updating a location associated with the vehicle using the adjusted location reading.

2. The method of claim 1 , wherein determining the adjusted location reading associated with the vehicle by processing the image and the location reading using the trained image localization model comprises:
generating a feature representation for the image by processing the image using the trained image localization model;
inputting the generated feature representation and the location reading to the trained image localization model;
outputting a set of two differential values, corresponding to latitude and longitude respectively; and
calculating the adjusted location reading by combining the location reading with the set of two differential values.

3. The method of claim 1 , wherein generating feature representations of the plurality of training images comprises processing the plurality of training images using a convolutional neural network.

4. The method of claim 1 , wherein learning the relationship between (1) the feature representations of the plurality of training images and (2) the differentials between the plurality of raw location readings and the plurality of reference location readings, is based on applying a loss function to both the feature representations and the differentials.

5. The method of claim 1 , wherein the image localization model is based on a machine-learning architecture comprising a convolutional neural network and one or more long short-term memory units.

6. The method of claim 5 , wherein the one or more long short-term memory units comprise two long short-term memory units, wherein the two long short-term memory units correspond to a latitude and a longitude of a location reading, respectively.

7. The method of claim 1 , further comprising obtaining the plurality of training images using one or more sensors associated with one or more vehicles, wherein the plurality of training images comprise ground scenes captured by an optical camera sensor.

8. The method of claim 1 , wherein the location readings comprise GPS coordinates.

9. The method of claim 1 , wherein the plurality of reference location readings were obtained by:
determining a pre-defined distance;
identifying a plurality of ground control points within the pre-defined distance from the plurality of training images;
determining a plurality of location readings associated with the plurality of ground control points; and
storing the plurality of reference location readings, wherein the plurality of reference location readings comprise the plurality of determined location readings.

10. The method of claim 1 , further comprising calculating an estimated time of arrival of the vehicle to arrive at a destination location, based on the updated location associated with the vehicle.

11. The method of claim 1 , further comprising determining a route for the vehicle to arrive at a destination location, based on the updated location associated with the vehicle.

12. The method of claim 1 , further comprising matching the vehicle with a ride requestor based on the updated location associated with the vehicle.

13. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
receiving an image and a location reading associated with a vehicle, the image and the location reading being captured substantially contemporaneously by a computing device associated with the vehicle;
determining an adjusted location reading associated with the vehicle by processing the image and the location reading using a trained image localization model, wherein the trained image localization model is a machine-learning model configured to output a differential between the location reading and the adjusted location reading, wherein the machine-learning model is trained by:
generating feature representations of a plurality of training images, wherein the plurality of training images are respectively associated with a plurality of raw location readings and a plurality of reference location readings; and
learning a relationship between (1) the feature representations of the plurality of training images and (2) differentials between the plurality of raw location readings and the plurality of reference location readings; and

updating a location associated with the vehicle using the adjusted location reading.

14. The system of claim 13 , wherein the instructions operable when executed by one or more of the processors to cause the system to perform operations comprising determining the adjusted location reading associated with the vehicle by processing the image and the location reading using the trained image localization model comprise instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
generating a feature representation for the image by processing the image using the trained image localization model;
inputting the generated feature representation and the location reading to the trained image localization model;
outputting a set of two differential values, corresponding to latitude and longitude respectively; and
calculating the adjusted location reading by combining the location reading with the set of two differential values.

15. The system of claim 13 , wherein the image localization model is based on a machine-learning architecture comprising a convolutional neural network and one or more long short-term memory units.

16. The system of claim 13 , wherein the one or more computer-readable non-transitory storage media comprise instructions further operable when executed by one or more of the processors to cause the system to perform operations comprising obtaining the plurality of training images using one or more sensors of one or more vehicles, wherein the plurality of training images comprise ground scenes captured by an optical camera sensor.

17. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:
receiving an image and a location reading associated with a vehicle, the image and the location reading being captured substantially contemporaneously by a computing dev

Assignees

• Lyft Inc

Inventors

• Sarukkai Ramesh Rangarajan
• Sun Shaohui

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/97 Primary

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title