Hybrid use of location sensor data and visual query to return local listings for visual query

Patent metadata

Field	Value
Publication number	US-9852156-B2
Application number	US-85556310-A
Country	US
Kind code	B2
Filing date	Aug 12, 2010
Priority date	Dec 3, 2009
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

A visual query is received from a client system, along with location information for the client system, and processed by a server system. The server system sends the visual query and the location information to a visual query search system, and receives from the visual query search system enhanced location information based on the visual query and the location information. The server system then sends a search query, including the enhanced location information, to a location-based search system. The search system receives and provides to the client one or more search results to the client system.

First claim

What is claimed is:

1. A computer-implemented method of processing a visual query comprising:
at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors:
obtaining a visual query from a client system, the visual query including an image that depicts an object;
obtaining location information from the client system, the location information specifying a location of the client system and meeting a first level of accuracy based on the location information specifying the location of the client system within a first predefined distance of an actual location of the client system;
determining, based on the visual query and the location of the client system, enhanced location information that specifies the location of the client system at a second level of accuracy that more accurately specifies the location of the client system than the first level of accuracy based on the enhanced location information specifying the location of the client system within a second predefined distance of the actual location of the client system that is less than the first predefined distance, the determining the enhanced location comprising:
comparing the visual query with a plurality of images of objects located within a predefined range of the location of the client system;
identifying, from the plurality of images and based on the comparison, a matching image that depicts the object depicted in the image;
obtaining, from stored location information for the matching image, a location of the object depicted in the matching image; and
determining the enhanced location of the client system using the stored location information for the matching image, the stored location information specifying a location of the object depicted in the matching image;

obtaining one or more search results in accordance with the enhanced location information; and
sending at least one of the search results to the client system.

2. The method of claim 1 , wherein the one or more search results are obtained in accordance with the enhanced location information and the visual query and wherein at least one search result in the one or more search results includes a result in the direction of a pose of the client system that is determined based on the visual query and the location information.

3. The method of claim 1 , wherein the location information includes a pose associated with the client system or a user associated with the client system and wherein the client system has an orientation and the visual query has an asymmetrical aspect ratio, the method further comprising:
constructing a viewing frustum based on the pose and based on the orientation of the client system determined based on one or more of: the client system sensors and the asymmetrical aspect ratio of the visual query, wherein the viewing frustum has a length that is based on the second level of accuracy, and
sending at least one search result within the viewing frustum to the client system.

4. The method of claim 3 , including obtaining a plurality of initial search results and filtering the initial search results to exclude search results outside the viewing frustum.

5. The method of claim 1 , further comprising:
identifying an accuracy value for the enhanced location information;
favoring search results near the location of the client device specified by the enhanced location information when the enhanced location has an accuracy value at or above a threshold; and
favoring search results with a high prominence value when the location of the client device specified by the enhanced location has an accuracy value below the threshold; and
sending at least one favored search result to the client system.

6. The method of claim 1 , further comprising:
creating an interactive results document comprising a bounding box outlining a respective sub-portion of the visual query and including at least one user selectable link to at least one of the search results, wherein the bounding box is created by projecting earth coordinates of a search result onto screen coordinates of the visual query; and
sending the interactive results document to the client system.

7. The method of claim 1 , wherein the enhanced location information comprises a first enhanced location, the method further comprising:
obtaining second visual query from the client system, the second visual query including a second image;
obtaining second location information from the client system;
requesting second enhanced location information for the client system based on the second visual query and the second location information;
in response to receiving the second enhanced location information and the second enhanced location information having greater accuracy than the second location information received from the client system,
obtaining one or more search results in accordance with the second enhanced location information;

in response to the request for the second enhanced location information being unsuccessful,
obtaining one or more search results in accordance with the second location information; and

sending at least one of the search results to the client system.

8. The method of claim 1 , wherein each of the one or more search results has an associated positional accuracy; and
the method further comprising:
selecting one or more of the search results having highest associated positional accuracy, and returning the selected search results to the client system.

9. The method of claim 1 , wherein each of the one or more search results comprises a respective local listing having an associated position and positional accuracy, the method further comprising:
selecting one or more first search results, each comprising a local listing having an associated position that A) satisfies a first positional closeness requirement with respect to the enhanced location information for the client system, and B) satisfies an accuracy requirement that the local listing's associated position has positional accuracy that is equal to or greater than a threshold; and
sending the one or more first search results to the client system.

10. The method of claim 9 , further comprising:
selecting one or more second search results in accordance with a requirement that each identified second search result satisfies a second positional closeness requirement with respect to at least one of the first search results; and
sending the one or more first search results and the one or more second search results to the client system.

11. The method of claim 1 , wherein each of the one or more search results comprises a respective local listing having an associated position and positional accuracy, the method further comprising:
selecting search results to send to the client system in accordance with the associated position and positional accuracy of each of the search results, the selecting including excluding from the selected search results those search results that A) have positional accuracy less than a threshold, and B) do not satisfy a positional closeness requirement with respect to at least one of the selected search results that has positional accuracy equal to or greater than the threshold and that satisfies a first positional closeness requirement with respect to the enhanced location for the client system.

12. The method of claim 1 , including sending to the client system a street view image matching the visual query.

13. The method of claim 1 , wherein the one or more search results are obtained in accordance with the enhanced location information and the visual query.

14.

Assignees

• Petrou David
• Flynn John
• Adam Hartwig
• Neven Hartmut
• Google Inc

Inventors

• Petrou David
• Flynn John
• Adam Hartwig
• Neven Hartmut

Classifications

• G06F17/30247 Primary

  Physics · mapped topic

• G06F17/30389

  Physics · mapped topic

• H04W4/02 Primary

  Services making use of location information · CPC title

• G06F17/30528

  Physics · mapped topic

• H04W64/00

  Locating users or terminals {or network equipment} for network management purposes, e.g. mobility management · CPC title