Collaborative and edge-enhanced augmented reality systems

Patent metadata

Field	Value
Publication number	US-11196842-B2
Application number	US-201916584814-A
Country	US
Kind code	B2
Filing date	Sep 26, 2019
Priority date	Sep 26, 2019
Publication date	Dec 7, 2021
Grant date	Dec 7, 2021

Abstract

In one example, a processing system of a mobile computing device including at least one processor may capture visual information of a first location, transmit a recognition request to a first server, the recognition request comprising the visual information, obtain a first recognition result from the first server comprising first annotation content associated with a first item in the visual information of the first location, the first recognition result further comprising visual information of the first item, present at least a portion of first annotation content via the mobile computing device, store the first annotation content and the visual information of the first item in a local cache, detect a first additional mobile computing device via a non-cellular wireless communication modality, and share the first annotation content and the visual information of the first item with the first additional mobile computing device via the non-cellular wireless communication modality.

First claim

What is claimed is:

1. A method comprising:
capturing, by a processing system of a mobile computing device, visual information of a first location;
transmitting, by the processing system, a recognition request to a first server, the recognition request comprising the visual information of the first location;
obtaining, by the processing system, a first recognition result from the first server, the first recognition result comprising first annotation content associated with a first item in the visual information of the first location, the first recognition result further comprising a model of the first item that is derived from visual information of the first item, wherein the first item is detected in the visual information of the first location by the first server based upon the model of the first item;
presenting, by the processing system, at least a portion of the first annotation content via the mobile computing device;
storing, by the processing system, the first annotation content and the model of the first item in a local cache of the mobile computing device;
detecting, by the processing system, a first additional mobile computing device via a non-cellular wireless communication modality; and
sharing, by the processing system, the first annotation content and the model of the first item with the first additional mobile computing device via the non-cellular wireless communication modality.

2. The method of claim 1 , further comprising:
verifying, by the processing system, an availability of the first server to process the recognition request.

3. The method of claim 1 , wherein the first server stores the model of the first item and the first annotation content.

4. The method of claim 3 , wherein the first server matches at least a portion of the visual information of the first location to at least a portion of the model of the first item that is stored at the first server.

5. The method of claim 3 , wherein the first server stores the model of the first item with the first annotation content in response to a previous recognition request from a second additional mobile computing device.

6. The method of claim 5 , wherein when the previous recognition request results in a cache miss at the first server, the first server obtains the model of the first item and the first annotation content from a second server.

7. The method of claim 6 , wherein the second server accesses a first database containing entries comprising models of a first plurality of items and associated first annotation contents, wherein the first server accesses a second database containing entries comprising models of a second plurality of items and associated second annotation contents, wherein the second plurality of items comprises a subset of the first plurality of items.

8. The method of claim 7 , wherein the second plurality of items comprises items for which the first server has provided recognition results to one or more mobile computing devices.

9. The method of claim 1 , wherein the at least the portion of the first annotation content is presented as at least one of:
a visual overlay over the visual information of the first location via the mobile computing device;
a haptic output via the mobile computing device; or
an audio output via the mobile computing device.

10. The method of claim 1 , wherein the first item is detected in the visual information of the first location by the first server based upon the model of the first item via a feature matching detection algorithm.

11. The method of claim 10 , wherein the feature matching detection algorithm comprises a scale-invariant feature transform with a gaussian mixture model algorithm.

12. The method of claim 1 , further comprising:
obtaining, by the processing system, a model of a second item and second annotation content associated with the second item from the first additional mobile computing device or from a second additional mobile computing device via the non-cellular wireless communication modality; and
storing, by the processing system, the model of the second item and the second annotation content in the local cache of the mobile computing device.

13. The method of claim 12 , further comprising:
capturing, by the processing system, visual information of a second location;
determining, by the processing system, that the first server is unavailable;
applying, by the processing system, a feature matching detection algorithm to the visual information of the second location in accordance with the local cache;
detecting, by the processing system, the second item in the visual information of the second location via the feature matching algorithm in accordance with the model of the second item that is stored in the local cache; and
presenting, by the processing system, at least a portion of the second annotation content via the mobile computing device, in response to the detecting the second item in the visual information of the second location.

14. An apparatus comprising:
a processing system including at least one processor; and
a computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:
capturing visual information of a first location;
transmitting a recognition request to a first server, the recognition request comprising the visual information of the first location;
obtaining a first recognition result from the first server, the first recognition result comprising first annotation content associated with a first item in the visual information of the first location, the first recognition result further comprising a model of the first item that is derived from visual information of the first item, wherein the first item is detected in the visual information of the first location by the first server based upon the model of the first item;
presenting at least a portion of the first annotation content via the mobile computing device;
storing the first annotation content and the model of the first item in a local cache of the mobile computing device;
detecting a first additional mobile computing device via a non-cellular wireless communication modality; and
sharing the first annotation content and the model of the first item with the first additional mobile computing device via the non-cellular wireless communication modality.

15. A method comprising:
obtaining, by a processing system of a first server including at least one processor, a first recognition request from a first mobile computing device, the first recognition request comprising visual information of a first location;
applying, by the processing system, a feature matching detection algorithm to the visual information of the first location in accordance with a cache of the processing system, wherein the cache contains:
entries comprising a plurality of models of a plurality of items derived from visual information of the plurality of items; and
associated annotation contents;

detecting, by the processing system, a first item of the plurality of items in the visual information of the first location via the feature matching detection algorithm in accordance with a model of the first item contained in the cache, the model of the first item comprising one of the plurality of models; and
providing, by the processing system, a first recognition result associated with the first item to the first mobile computing device, the first recognition result comprising the model of the first item and first annotation content that is associated with the first item.

16. T

Assignees

• At & T Ip I Lp
• Univ Hong Kong Science & Tech

Inventors

• Han Bo
• Zhang Wenxiao
• Hui Pan
• Gopalakrishnan Vijay
• Ip Cheuk Yiu

Classifications

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• H04L67/568

  Storing data temporarily at an intermediate stage, e.g. caching · CPC title

• G06V20/20

  in augmented reality scenes · CPC title