Image based localization system

Patent metadata

Field	Value
Publication number	US-11354820-B2
Application number	US-201916573592-A
Country	US
Kind code	B2
Filing date	Sep 17, 2019
Priority date	Nov 17, 2018
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.

First claim

What is claimed is:

1. A computer-implemented method for determining a localized state of an autonomous vehicle, the method comprising:
receiving, by a computing system comprising one or more computing devices, a query image collected by the autonomous vehicle and depicting a surrounding environment of the autonomous vehicle;
inputting, by the computing system, the query image into a machine-learned image embedding model to receive a query embedding as an output of the machine-learned image embedding model;
accessing, by the computing system, a database of pre-computed image embeddings, the pre-computed image embeddings previously computed for a plurality of images by the machine-learned image embedding model;
obtaining, by the computing system, a plurality of candidate embeddings from the database of pre-computed image embeddings based at least in part on vehicle location data associated with the autonomous vehicle and image location data associated with each pre-computed image embedding in the database of pre-computed image embeddings;
comparing, by the computing system, the query embedding to the plurality of candidate embeddings to identify a subset of candidate embeddings that have embeddings that satisfy a similarity threshold; and
determining, by the computing system, the localized state of the autonomous vehicle based at least in part on the image location data associated with each pre-computed image embedding in the subset of candidate embeddings.

2. The computer-implemented method of claim 1 , wherein determining the localized state of the autonomous vehicle based at least in part on the image location data associated with each pre-computed image embedding in the subset of candidate embeddings further comprises:
inputting, by the computing system, the query image into a machine-learned feature extraction model to obtain a query feature representation for the query image;
obtaining, by the computing system, a respective feature representation for a respective image associated with each candidate embedding in the subset of candidate embeddings;
for each candidate embedding in the subset of candidate embeddings, inputting, by the computing system, the query feature representation and the respective feature representation for the respective image associated with the candidate embedding into a machine-learned regression model to obtain a respective relative displacement between the query image and the image associated with the candidate embedding;
determining, by the computing system, the localized state of the autonomous vehicle based at least in part on a set of relative displacements that comprises the respective relative displacement between the query image and the respective image associated with each of the candidate embeddings in the subset of candidate embeddings.

3. The computer-implemented method of claim 2 , wherein the respective feature representation for the respective image associated with each candidate embedding in the subset of candidate embeddings is previously computed by the machine-learned feature extraction model and obtaining, by the computing system, each respective feature representation comprises obtaining, by the computing system, the respective feature representation from a database of feature representations.

4. The computer-implemented method of claim 2 , wherein determining the localized state of the autonomous vehicle based at least in part on the set of relative displacements comprises aggregating the set of relative displacements to obtain the localized state.

5. The computer-implemented method of claim 4 , wherein aggregating the set of relative displacements comprises determining one or more median location coordinates and a median heading angle associated with the set of relative displacements.

6. The computer-implemented method of claim 2 , wherein the machine-learned regression model and the machine-learned feature extraction model have been jointly trained end-to-end on a set of training data that comprises a plurality of pairs of training images, each pair of training images having a known ground truth displacement between the pair of training images.

7. The computer-implemented method of claim 1 , wherein the vehicle location data associated with the autonomous vehicle and the image location data associated with each of the pre-computed image embeddings comprise geolocation coordinates.

8. The computer-implemented method of claim 1 , wherein the machine-learned image embedding model is previously trained using a triplet training scheme, the triplet training scheme utilizing a plurality of image triplets, each image triplet in the plurality of image triplets comprising an anchor image, a positive image, and a negative image, wherein:
the anchor image is associated with a respective geolocation that is closer to a respective geolocation associated with the positive image than a respective geolocation associated with the negative image; and
the positive image is associated with a respective heading angle within a respective heading angle associated with the anchor image by a heading threshold.

9. The computer-implemented method of claim 1 , further comprising:
controlling, by the computing system, motion of the autonomous vehicle based at least in part on the localized state of the autonomous vehicle.

10. A computing system comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:
receiving a query image collected by an autonomous vehicle and depicting a surrounding environment of the autonomous vehicle;
inputting the query image into a machine-learned image embedding model to receive a query embedding as an output of the machine-learned image embedding model;
accessing a database of pre-computed image embeddings, the pre-computed image embeddings previously computed for a plurality of images by the machine-learned image embedding model;
obtaining a plurality of candidate embeddings from the database of pre-computed image embeddings based at least in part on vehicle location data associated with the autonomous vehicle and image location data associated with each pre-computed image embedding in the database of pre-computed image embeddings;
comparing the query embedding to the plurality of candidate embeddings to identify a subset of candidate embeddings that satisfy a threshold; and
determining a localized state of the autonomous vehicle based at least in part on the image location data associated with each pre-computed image embedding in the subset of candidate embeddings.

11. The computing system of claim 10 , wherein determining the localized state of the autonomous vehicle further comprises:
inputting the query image into a machine-learned feature extraction model to obtain a query feature representation for the query image;
for each candidate embedding in the subset of candidate embeddings,
obtaining a respective feature representation for a respective image associated with the candidate embedding; and
inputting the query feature representation and the respective feature representation into a machine-learned regression model to obtain a respective relative displacement between the query image and the respective image associated with the candidate embedding; and

determining the localized state of the autonomous vehicle based at least in part on a set of relative displacements that comprise the respective relative displacement between the query image and the respective image associated with each of the candidate embeddings in the subset o

Assignees

• Uatc Llc

Inventors

• Urtasun Raquel
• Covarrubias Julieta Martinez
• Wang Shenlong
• Fan Hongbo

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06T7/75 Primary

  involving models · CPC title