Multi-sync ensemble model for device localization

Patent metadata

Field	Value
Publication number	US-10820172-B2
Application number	US-201916455630-A
Country	US
Kind code	B2
Filing date	Jun 27, 2019
Priority date	Jun 27, 2018
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

A system and method determine the location of a device. The device collects sensor data using one or more sensors. Based on the sensor data, one or more localization models are selected from a plurality of localization models. The selected models are applied to generate one or more potential locations. The current location of the device is determined based on the one or more potential locations.

First claim

The invention claimed is:

1. A method for determining a location of a device, the method comprising:
collecting image data captured by a camera on the device and global coordinates detected by a global positioning system (GPS) receiver on the device;
determining whether the device is currently indoors or outdoors based on an illumination level of the image data;
selecting one or more localization models from a plurality of localization models based on the determination of whether the device is currently indoors or outdoors;
applying the selected one or more localization models to the image data and the global coordinates to generate one or more potential locations of the device; and
determining the location of the device based on the one or more potential locations.

2. The method of claim 1 , wherein the plurality of localization models includes at least one of: a point cloud based model, a plane matching model, a line matching model, a geographic information system (GIS) model, a building recognition model, an object recognition model, a semantic matching model, a cube matching model, a cylinder matching model, a horizon matching model, a light source matching model, and a landscape recognition model.

3. The method of claim 1 , wherein applying the selected one or more localization models comprises applying a plurality of localization models to generate a plurality of potential locations, wherein each applied localization model generates one of the plurality of potential locations.

4. The method of claim 1 , wherein determining the location of the device comprises calculating an average location of the plurality of potential locations.

5. The method of claim 4 , wherein the average is a weighted average.

6. The method of claim 1 , wherein selecting the one or more localization models comprises:
assigning a score to each localization model of the one or more localization models, the score assigned to each localization model indicating a corresponding likelihood of that localization model generating an accurate location; and
selecting the one or more localization models based on the scores for the one or more localization models.

7. The method of claim 6 , wherein the score assigned to each localization model is based on historic performance of the models in similar environments.

8. The method of claim 6 , wherein the score assigned to each localization model is based on the image data and the global coordinates.

9. A device comprising:
one or more processors; and
one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
collecting image data captured by a camera on the device and global coordinates detected by a global positioning system (GPS) receiver on the device;
determining whether the device is currently indoors or outdoors based on an illumination level of the image data;
selecting one or more localization models from a plurality of localization models based on the determination of whether the device is currently indoors or outdoors;
applying the selected one or more localization models to the image data and the global coordinates to generate one or more potential locations of the device; and
determining a location of the device based on the one or more potential locations.

10. The device of claim 9 , wherein the plurality of localization models includes at least one of: a point cloud based model, a plane matching model, a line matching model, a geographic information system (GIS) model, a building recognition model, an object recognition model, a semantic matching model, a cube matching model, a cylinder matching model, a horizon matching model, a light source matching model, and a landscape recognition model.

11. The device of claim 9 , wherein applying the selected one or more localization models comprises applying a plurality of localization models to generate a plurality of potential locations, wherein each applied localization model generates one of the plurality of potential locations.

12. The device of claim 9 , wherein determining the location of the device comprises calculating an average location of the plurality of potential locations.

13. The device of claim 12 , wherein the average is a weighted average.

14. The device of claim 9 , wherein selecting the one or more localization models comprises:
assigning a score to each localization model of the one or more localization models, the score assigned to each localization model indicating a corresponding likelihood of that localization model generating an accurate location; and
selecting the one or more localization models based on the scores for the one or more localization models.

15. The device of claim 14 , wherein the score assigned to each localization model is based on historic performance of the models in similar environments.

16. The device of claim 14 , wherein the score assigned to each localization model is based on the image data and the global coordinates.

17. A non-transitory computer-readable storage medium storing instructions for determining a location of a device, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising:
collecting image data captured by a camera on the device and global coordinates detected by a global positioning system (GPS) receiver on the device;
determining whether the device is currently indoors or outdoors based on an illumination level of the image data;
selecting one or more localization models from a plurality of localization models based on the determination of whether the device is currently indoors or outdoors;
applying the selected one or more localization models to the image data and the global coordinates to generate one or more potential locations of the device; and
determining a location of the device based on the one or more potential locations.

18. The storage medium of claim 17 , wherein the plurality of localization models includes at least one of: a point cloud based model, a plane matching model, a line matching model, a geographic information system (GIS) model, a building recognition model, an object recognition model, a semantic matching model, a cube matching model, a cylinder matching model, a horizon matching model, a light source matching model, and a landscape recognition model.

19. The storage medium of claim 17 , wherein determining the location of the device comprises calculating an average location of the plurality of potential locations.

20. The storage medium of claim 19 , wherein the average is a weighted average.

Assignees

• Niantic Inc

Inventors

• Hu Si Ying Diana
• Ashok Anubhav
• Turner Peter

Classifications

• H04W4/38 Primary

  for collecting sensor information · CPC title

• H04W4/029

  Location-based management or tracking services · CPC title

• H04W4/025

  using location based information parameters · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title