Image based localization system

Patent metadata

Field	Value
Publication number	US-12400358-B2
Application number	US-202217833414-A
Country	US
Kind code	B2
Filing date	Jun 6, 2022
Priority date	Nov 17, 2018
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.

First claim

What is claimed is:

1. A computer-implemented method for determining a location of a vehicle, the method comprising:
receiving image data associated with an environment of the vehicle;
processing the image data with a machine-learned image embedding model to generate a query embedding for the image data;
identifying one or more identified image embeddings of a plurality of image embeddings associated with the environment of the vehicle based on a plurality of similarity scores between the plurality of image embeddings and the query embedding, wherein one or more similarity scores of the plurality of similarity scores satisfy a threshold similarity score, the one or more similarity scores respective to the one or more identified image embeddings; and
determining the location of the vehicle based on a comparison between the query embedding and the one or more identified image embeddings of the plurality of image embeddings associated with the environment of the vehicle.

2. The computer-implemented method of claim 1 , wherein the plurality of image embeddings are previously computed for a plurality of images of the environment by the machine-learned image embedding model.

3. The computer-implemented method of claim 1 , wherein the one or more identified image embeddings associated with the environment of the vehicle are obtained from a feature representation database remotely located from the vehicle.

4. The computer-implemented method of claim 1 , further comprising:
obtaining the one or more identified image embeddings associated with the environment of the vehicle based on vehicle location data associated with the vehicle.

5. The computer-implemented method of claim 4 , wherein the vehicle location data comprises coarse geolocation coordinates.

6. The computer-implemented method of claim 5 , wherein the coarse geolocation coordinates comprise global positioning system coordinates.

7. The computer-implemented method of claim 4 , wherein the one or more identified image embeddings are associated with image location data, and wherein the one or more identified image embeddings associated with the environment of the vehicle are obtained based on a comparison between the vehicle location data and the image location data.

8. The computer-implemented method of claim 7 , wherein the location of the vehicle is determined based on the image location data.

9. The computer-implemented method of claim 1 , wherein the image data comprises a query image depicting at least a portion of a surrounding environment of the vehicle.

10. The computer-implemented method of claim 9 , wherein determining the location of the vehicle based on the comparison between the query embedding and the one or more identified image embeddings associated with the environment of the vehicle comprises:
determining a relative displacement between the query image and an image associated with at least one of the one or more identified image embeddings; and
determining the location of the vehicle based on the relative displacement.

11. The computer-implemented method of claim 1 , wherein the location of the vehicle is indicative of one or more current geolocation coordinates and a heading angle of the vehicle.

12. The computer-implemented method of claim 1 , wherein the image data is camera data, LIDAR data, or RADAR data.

13. A computing system comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:
receiving image data associated with an environment of a vehicle;
processing the image data with a machine-learned image embedding model to generate a query embedding for the image data;
identifying one or more identified image embeddings of a plurality of image embeddings associated with the environment of the vehicle based on a plurality of similarity scores between the plurality of image embeddings and the query embedding, wherein one or more similarity scores of the plurality of similarity scores satisfy a threshold similarity score, the one or more similarity scores respective to the one or more identified image embeddings; and
determining a location of the vehicle based on a comparison between the query embedding and the one or more identified image embeddings of the plurality of image embeddings associated with the environment of the vehicle.

14. The computing system of claim 13 , wherein the plurality of image embeddings are previously computed for a plurality of images of the environment by the machine-learned image embedding model.

15. The computing system of claim 13 , further comprising:
obtaining the one or more identified image embeddings associated with the environment of the vehicle based on vehicle location data associated with the vehicle.

16. The computing system of claim 13 , wherein the image data comprises a query image depicting at least a portion of a surrounding environment of the vehicle.

17. The computing system of claim 16 , wherein the computing system is located onboard the vehicle, wherein the computing system comprises one or more cameras, and wherein the query image is collected by the one or more cameras.

18. The computing system of claim 13 , wherein the vehicle comprises an autonomous truck.

19. The computing system of claim 18 , wherein the operations further comprise:
controlling a motion of the autonomous truck based on the location of the vehicle.

20. One or more non-transitory, computer-readable media storing instructions that are executable by one or more processors to cause the one or more processors to perform operations, the operations comprising:
receiving image data associated with an environment of a vehicle;
processing the image data with a machine-learned image embedding model to generate a query embedding for the image data;
identifying one or more identified image embeddings of a plurality of image embeddings associated with the environment of the vehicle based on a plurality of similarity scores between the plurality of image embeddings and the query embedding, wherein one or more similarity scores of the plurality of similarity scores satisfy a threshold similarity score, the one or more similarity scores respective to the one or more identified image embeddings; and
determining a location of the vehicle based on a comparison between the query embedding and the one or more identified image embeddings of the plurality of image embeddings associated with the environment of the vehicle.

Assignees

• Aurora Operations Inc

Inventors

• Urtasun Raquel
• Martinez Covarrubias Julieta
• Wang Shenlong
• Fan Hongbo

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06F18/21355

  nonlinear criteria, e.g. embedding a manifold in a Euclidean space · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title