Systems and methods for context based information delivery using augmented reality

Patent metadata

Field	Value
Publication number	US-9767615-B2
Application number	US-201514692290-A
Country	US
Kind code	B2
Filing date	Apr 21, 2015
Priority date	Apr 23, 2014
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

Generally discussed herein are systems, apparatuses, and methods for providing contextually relevant augmented reality (AR) to a wearable device. In one or more embodiments, a method can include extracting features of one or more objects in a location in a field of view of a camera of a wearable device, retrieving a three dimensional model of the location based on the extracted features, assessing a situation of a user associated with the wearable device, modifying the three dimensional model based on the assessed user situation, and presenting, using the wearable device, at least a portion of the modified three dimensional model.

First claim

What is claimed is:

1. An augmented reality system comprising:
processing circuitry networked to one or more servers, the one or more servers including three dimensional models of objects and features of the objects indexed and stored thereon, the processing circuitry to provide functionality of a plurality of modules comprising:
a feature extraction module to extract a wire frame of a first object in a field of view of a camera of a wearable device of a user and provide the extracted wire frame to the server, wherein the server is to compare the extracted wire frame to wire frames of the three dimensional models of objects near a location of the user and return a three dimensional model of an object of the three dimensional models of objects that includes wire frame that best matches the extracted wire frame;
a situation assessment module to assess a situation of the user;
a modification module to augment the three dimensional model with one or more augmentation objects based on the assessed user situation; and
an image rendering module to render an image on the wearable device including the modified three dimensional model,
wherein the one or more servers include generally featureless regions on or near objects indexed and stored thereon and the modification module is further to determine a location on or around the object on which to situate an augmented reality object that does not obscure the view of the object to the user includes the modification module to look up the generally featureless region in the server, wherein the generally featureless regions includes less than a threshold number of edges as determined by at least one of a maximally stable extrema regions, Viola-Jones, and speeded up robust features (SURF) operation.

2. The system of claim 1 , wherein the plurality of modules further comprise:
a bandwidth assessment module to assess available compute bandwidth at the wearable device, a smart phone networked to the wearable device, and the server and delegate one or more operations to modify the three dimensional model to the smart phone, the wearable device, or the server based on the assessed available compute bandwidth.

3. The system of claim 1 , wherein situation assessment module to assess the situation of the user includes determining if the user is 1) determining their orientation relative to the location; 2) browsing the location; 3) searching for a specific area of interest; or 4) looking for guidance to find the area of interest.

4. The system of claim 3 , wherein the situation assessment module further assesses a buying history and past behavior of the user in determining the user situation.

5. The system of claim 3 , wherein the modification module to modify the three dimensional model based on the user situation includes the modification module to modify the three dimensional model to include different augmented reality objects and different amounts of augmented reality objects for each user situation.

6. The system of claim 1 , wherein the wearable device includes smart glasses or a smart visor.

7. The system of claim 1 , wherein the modification module is to modify the three dimensional model to include an augmented reality object that indicates where a person the user is interested in finding is located.

8. The system of claim 1 , wherein the image rendering module is to modify the three dimensional model to include a view of a camera output, wherein the camera includes a field of view of an area that is occluded to the user.

9. The system of claim 1 , wherein the plurality of modules further comprise a location module to determine a geographic location of the user, wherein the modification module is further to rotate the three dimensional model of the object to match the orientation of the user's head relative to the object and add an augmented reality object to the rotated three dimensional model of the object, wherein the orientation of the user's is relative to a cardinal direction determined based on video data from the camera of the wearable device and the location from the location module.

10. A method for augmented reality, the method comprising:
extracting, using processing circuitry, at least one wire frame of a location in a field of view of a camera of a wearable device based on the line-of-sight and position of the camera of the wearable device;
comparing the extracted at least one wire frame to wire frames of three dimensional models of objects near the location;
retrieving, from a server, a three dimensional model of the location based on the comparison, the retrieved three dimensional model including a wire frame that best matches a wire frame of the extracted at least one wire frame;

assessing a situation of a user associated with the wearable device;
determining one or more objects in the field of view of the camera that the user is interested in based on the assessed user situation;
modifying the three dimensional model based on the assessed user situation and the determined one or more objects;
determining a location on or around the object on which an augmented reality object is placed that does not obscure the view of the object to the user by looking up a generally featureless region of the location on or near the object indexed and stored in the server on which to situate an augmented reality object of the augmented reality objects, wherein the generally featureless region includes less than a threshold number of edges as determined by at least one of a maximally stable extrema regions, Viola-Jones, and speeded up robust features (SURF) operation; and

presenting, using the wearable device, at least a portion of the modified three dimensional model.

11. The method of claim 10 , further comprising:
assessing available compute bandwidth at the wearable device, a smart phone networked to the wearable device, and a server networked to the wearable device; and
delegating computer program computation required to modify the three dimensional model to the smart phone, the wearable device, and the server as a function of the available compute bandwidth.

12. The method of claim 10 , wherein assessing the situation of the user includes determining if the user is 1) determining their orientation relative to the location; 2) browsing the location; 3) searching for a specific area of interest; or 4) looking for guidance to find the area of interest.

13. The method of claim 12 , wherein modifying the three dimensional model based on the user situation includes modifying the three dimensional model to include different augmented reality objects for each user situation.

14. A non-transitory computer readable storage device including instructions stored thereon, the instructions which, when executed by a machine configure the machine to:
extract at least one wire frame of a location in a field of view of a camera of a wearable device;
compare the extracted at least one wire frame to wire frames of three dimensional models of objects near the location;
retrieve, from a server, a three dimensional model of the location based on the extracted at least one wire frame, the retrieved three dimensional model including a wire frame that best matches a wire frame of the extracted at least one wire frame;
assess a situation of a user associated with the wearable device;
modify the three dimensional model based on the assessed user situation;
determine a location on or around the object on which an augmented reality object is placed that does not obscure the view of the object to the user by identifying a generally featureless region of the location on or near the object indexed and stored in

Assignees

• Raytheon Co

Inventors

• Young Darrell L
• Gondran Gregory R
• Ribardo Jr Thomas G
• Bigham Mark A

Classifications

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G09G2370/02

  Networking aspects · CPC title

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title