Use of image sensors to query real world for geo-reference information

Patent metadata

Field	Value
Publication number	US-12033351-B2
Application number	US-202318197364-A
Country	US
Kind code	B2
Filing date	May 15, 2023
Priority date	Nov 6, 2019
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

The present disclosure provides systems and methods that makes use of one or more image sensors of a device to provide users with information relating to nearby points of interest. The image sensors may be used to detect features and/or objects in the field of view of the image sensors. Pose data, including a location and orientation of the device is then determined based on the one or more detected features and/or objects. A plurality of points of interest that are within a geographical area that is dependent on the pose data are then determined. The determination may, for instance, be made by querying a mapping database for points of interest that are known to be located within a particular distance of the location of the user. The device then provides information to the user indicating one or more of the plurality of points of interest.

First claim

The invention claimed is:

1. A system comprising:
one or more processors configured to:
detect, based on data derived from use of one or more sensors of a user device, one or more features captured by the one or more sensors;
determine, based on at least the one or more detected features, pose data including a location and orientation of the user device;
determine, based on the pose data, a plurality of points of interest within a particular geographical area that is determined based on the pose data; and
provide, via the user device, information indicating one or more of the plurality of points of interest, wherein the plurality of points of interest are determined by querying a mapping database for points of interest within the particular geographical area and within a particular range from the location.

2. The system of claim 1 , wherein the particular geographical area is within a predefined distance of the location of the user device included in the pose data.

3. The system of claim 1 , wherein the particular geographical area is within a predefined angular range that is based on the orientation of the user device included in the pose data.

4. The system of claim 3 , wherein:
the predefined angular range is wider than a field of view of the one or more sensors.

5. The system of claim 1 , wherein the particular geographical area comprises a first geographical area that is defined by a first distance from the location of the user device and a first angular range and a second geographical area that is defined by a second distance from the location of the user device and a second angular range that is based on the orientation of the user device, wherein the first angular range is wider than the second angular range and the first distance is less than the second distance.

6. The system of claim 1 , wherein the one or more sensors includes at least one of an image sensor, an accelerometer, a visual positioning system, or a location sensor.

7. The system of claim 1 , wherein the points of interest are filtered such that points of interest outside an angular range that is centered on the orientation of the user device are excluded.

8. The system of claim 1 , wherein:
the pose data is determined using a database of feature data generated based on a collection of street level images which depicts a geographic area that includes a sub-area in which the features captured by the one or more sensors of the user device are located.

9. A method comprising:
deriving, using one or more processors, data from one or more sensors of a device;
detecting, using the one or more processors and based on the derived data, one or more features in a particular geographical area;
determining, using the one or more processors and based on at least the one or more detected features, pose data including a location and orientation of the device;
determining, using the one or more processors and based on the pose data, a plurality of points of interest within the particular geographical area; and
providing, using the one or more processors, information indicating one or more of the plurality of points of interest in response to detecting features, wherein the plurality of points of interest are determined by querying a mapping database for points of interest within the particular geographical area and within a particular range from the location.

10. The method of claim 9 , wherein detecting the one or more features in the particular geographical area occurs without a prompt from a user at a time of the detecting.

11. The method of claim 9 , further comprising determining, using the one or more processors, the plurality of points of interest based on a field of view of the one or more sensors.

12. The method of claim 9 , further comprising computing, using the one or more processors, a relevance weight for each of the plurality of points of interest.

13. The method of claim 12 , further comprising providing, using the one or more processors and based on the computed relevance weights, a ranked list of the plurality of points of interest.

14. The method of claim 12 , wherein computing a relevance weight further includes using a distance group factor, a type factor, and a place rating factor.

15. The method of claim 14 , wherein the distance group factor is a value determined based on a predefined distance threshold between the device and each of the plurality of points of interest.

16. The method of claim 14 , wherein the type factor is a value based on a type of point of interest.

17. The method of claim 14 , wherein the place rating factor is a value based on at least one publicly available rating for each of the plurality of points of interest.

18. The method of claim 9 , wherein determining the one or more sensors includes on at least one of a an image sensor, a location sensor, a visual positioning system, or an accelerometer.

19. A non-transitory computer-readable medium storing instructions, which when executed by one or more processors, cause the one or more processors to:
derive data from one or more sensors of a user device;
detect one or more features in a particular geographical area based on the derived data;
determine pose data including a location and orientation of the user device based on at least the one or more detected features;
determine a plurality of points of interest within the particular geographical area based on the pose data; and
provide information indicating one or more of the plurality of points of interest in response to detecting features, wherein the plurality of points of interest are determined by querying a mapping database for points of interest within the particular geographical area and within a particular range from the location.

20. The non-transitory computer-readable medium of claim 19 , wherein detecting the one or more features in the particular geographical area occurs without a prompt from a user at a time of the detecting.

Assignees

• Google Llc

Inventors

• Hincapie Juan David
• Le Andre

Classifications

• G06V20/30

  in albums, collections or shared content, e.g. social network photos or video · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06F16/9032

  Query formulation · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title