Client-server based dynamic search

Patent metadata

Field	Value
Publication number	US-9495389-B2
Application number	US-201313843607-A
Country	US
Kind code	B2
Filing date	Mar 15, 2013
Priority date	Mar 15, 2013
Publication date	Nov 15, 2016
Grant date	Nov 15, 2016

Abstract

Method, mobile device, computer program product and apparatus for performing a search are disclosed. The method of performing a search comprises receiving one or more images of an environment in view of a mobile device, generating a simultaneous localization and mapping of the environment using the one or more images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment, sending a set of the plurality of map points as a search query to a server, receiving a query response from the server, and identifying an object in the environment based at least in part on the query response.

First claim

We claim:

1. A method of performing a search, comprising:
receiving a plurality of images of an environment in view of a mobile device;
generating a simultaneous localization and mapping of the environment using the plurality of images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment;
sending a set of the plurality of map points as a search query to a server at a first time instance;
receiving a query response from the server at a second time instance; and
identifying an object in the environment based at least in part on the query response, wherein the mobile device has a first pose relative to the object at the first time instance and the mobile device has a second pose relative to the object at the second time instance.

2. The method of claim 1 , wherein the plurality of map points comprises:
views of the plurality of surfaces captured by the mobile device from different angles.

3. The method of claim 2 , wherein each of a map point in the plurality of map points comprises a three dimensional location, one or more descriptors of the three dimensional location, and a corresponding surface normal of the three dimensional location.

4. The method of claim 2 , wherein each of a map point in the plurality of map points comprises a representation from a direction orthogonal to a normal direction of the map point.

5. The method of claim 1 , further comprising:
determining the second pose of the mobile device with respect to the object in the environment based at least in part on the query response received at the second time instance.

6. The method of claim 5 , wherein determining the second pose of the mobile device with respect to the object in the environment comprises:
obtaining from the query response a pose of the object with respect to an origin of the environment;
tracking a pose of the mobile device with respect to the origin of the environment; and
determining the second pose of the mobile device with respect to the object using the pose of the object with respect to the origin of the environment and the pose of the mobile device with respect to the origin of the environment.

7. The method of claim 5 , further comprising:
causing an augmentation to be rendered in response to the second pose of the mobile device with respect to the object in the environment.

8. The method of claim 1 further comprising:
tracking movement of the mobile device with respect to an origin of the environment; and
determining the mobile device has moved with respect to the origin of the environment.

9. The method of claim 8 , wherein determining the mobile device has moved comprises at least one of:
determining the mobile device has rotated more than a predetermined angle; and
determining the mobile device has translated more than a predetermined distance.

10. The method of claim 8 , further comprising:
identifying incremental map points captured by the mobile device in response to the movement of the mobile device;
generating an updated simultaneous localization and mapping of the environment with the incremental map points captured by the mobile device; and
sending the incremental map points captured by the mobile device in an updated search query to the server.

11. The method of claim 10 , wherein the updated simultaneous localization and mapping of the environment includes views of the environment larger than a field of view of the environment from the mobile device; and wherein the origin of the environment is out of the field of view of the mobile device.

12. A mobile device, comprising:
a camera configured to receive a plurality of images of an environment in view of the mobile device;
a control unit configured to generate a simultaneous localization and mapping of the environment using the plurality of images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment;
a network interface configured to send a set of the plurality of map points as a search query to a server at a first time instance; the network interface is further configured to receive a query response from the server at a second time instance; and
the control unit is further configured to identify an object in the environment based at least in part on the query response, wherein the mobile device has a first pose relative to the object at the first time instance and the mobile device has a second pose relative to the object at the second time instance.

13. The mobile device of claim 12 , wherein the plurality of map points comprises:
views of the plurality of surfaces captured by the mobile device from different angles.

14. The mobile device of claim 13 , wherein each of a map point in the plurality of map points comprises a three dimensional location, one or more descriptors of the three dimensional location, and a corresponding surface normal of the three dimensional location.

15. The mobile device of claim 13 , wherein each of a map point in the plurality of map points comprises a representation from a direction orthogonal to a normal direction of the map point.

16. The mobile device of claim 12 , further comprising:
logic configured to determine the second pose of the mobile device with respect to the object in the environment based at least in part on the query response received at the second time instance.

17. The mobile device of claim 16 , wherein logic configured to determine the second pose of the mobile device with respect to the object in the environment comprises:
logic configured to obtain from the query response a pose of the object with respect to an origin of the environment;
logic configured to track a pose of the mobile device with respect to the origin of the environment; and
logic configured to determine the second pose of the mobile device with respect to the object using the pose of the object with respect to the origin of the environment and the pose of the mobile device with respect to the origin of the environment.

18. The mobile device of claim 16 , further comprising:
logic configured to cause an augmentation to be rendered in response to the second pose of the mobile device with respect to the object in the environment.

19. The mobile device of claim 12 further comprising:
logic configured to track movement of the mobile device with respect to an origin of the environment; and
logic configured to determine the mobile device has moved with respect to the origin of the environment.

20. The mobile device of claim 19 , wherein logic configured to determine the mobile device has moved comprises at least one of:
logic configured to determine the mobile device has rotated more than a predetermined angle; and
logic configured to determine the mobile device has translated more than a predetermined distance.

21. The mobile device of claim 19 , further comprising:
logic configured to identify incremental map points captured by the mobile device in response to the movement of the mobile device;
logic configured to generate an updated simultaneous localization and mapping of the environment with the incremental map points captured by the mobile device; and
logic configured to send the incremental map points captured by the mobile device in an updated search query to the server.

Assignees

• Qualcomm Inc

Inventors

• Pan Qi
• Wagner Daniel
• Szalavari Zsolt

Classifications

• G06F17/30277 Primary

  Physics · mapped topic

• G06F17/3087

  Physics · mapped topic

• G06F16/9537 Primary

  Spatial or temporal dependent retrieval, e.g. spatiotemporal queries · CPC title

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title