User interface for augmented reality enabled devices

Patent metadata

Field	Value
Publication number	US-9317972-B2
Application number	US-201213718923-A
Country	US
Kind code	B2
Filing date	Dec 18, 2012
Priority date	Dec 18, 2012
Publication date	Apr 19, 2016
Grant date	Apr 19, 2016

Abstract

Method and apparatus for displaying augmented reality contents are disclosed. The method may include controlling a camera to scan an environment in view of a user, identifying a set of surfaces in the environment for displaying user interface windows according to characteristics of the environment, prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment, and displaying the set of augmented reality contents on the set of surfaces in a display. Characteristics of the environment comprise at least aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and/or background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

First claim

We claim:

1. A method of displaying augmented reality contents, comprising:
receiving, by a camera, a scan of an environment in view of a user;
identifying a set of surfaces in the environment for displaying user interface windows;
assigning a surface priority level to each surface in the set of surfaces, wherein the surface priority level for each surface is selected from a plurality of surface priority levels;
assigning a surface trackability level to each surface in the set of surfaces, wherein the surface trackability level for each surface is selected from a plurality of surface trackability levels;
assigning a content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment, wherein the content priority level for each content item is selected from a plurality of content priority levels; and
displaying each content item in the set of augmented reality contents for display on a surface in the set of surfaces in a display based on the surface priority level, the surface trackability level, and the content priority level.

2. The method of claim 1 , wherein the environment comprises real world objects, and wherein the set of surfaces includes surfaces in top, bottom, left, right, and front view of the user; and wherein the display is a head mounted display.

3. The method of claim 1 , wherein assigning the surface priority level to each surface in the set of surfaces comprises:
assigning the surface priority level based on: areas of the set of surfaces, orientations of the set of surfaces, and locations of the set of surfaces relative to field of view of the user, wherein the plurality of surface priority levels include high, medium and low surface priority levels.

4. The method of claim 1 , wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:
assigning the content priority level based on at least one of: aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

5. The method of claim 1 , wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:
assigning the content priority level based on a history of prior use patterns of the user in the environment.

6. The method of claim 5 , wherein assigning the content priority level based on a history of prior use patterns of the user in the environment comprises at least one of:
selecting a first set of augmented reality contents for display in response to a determination of the user being at work;
selecting a second set of augmented reality contents for display in response to a determination of the user being at home;
selecting a third set of augmented reality contents for display in response to a determination of the user being in a business meeting; and
selecting a fourth set of augmented reality contents for display in response to a determination of the user being in a social event.

7. The method of claim 6 , wherein
the first set of augmented reality contents includes application programs for communication and documentation the user is accustomed to use at work; and
the second set of augmented reality contents includes application programs for communication and entertainment the user is accustomed to use at home.

8. The method of claim 7 , wherein
the first set of augmented reality contents for communication and documentation includes email, web browser, and office productivity applications; and
the second set of augmented reality contents for communication and entertainment includes social networking websites, movie and video game applications.

9. The method of claim 1 , wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:
assigning the content priority level based on at least one of: scaling factors of the set of augmented reality contents to be displayed, and logical relationships of the set of augmented reality contents to be displayed.

10. The method of claim 9 , wherein the logical relationships of the set of augmented reality contents to be displayed comprises:
a document and a corresponding video presentation of the document.

11. The method of claim 9 , wherein the logical relationships of the set of augmented reality contents to be displayed comprises:
a document and corresponding presentation slides of the document.

12. The method of claim 1 , wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:
assigning the content priority level based on predefined user preferences.

13. The method of claim 12 , wherein the predefined user preferences comprise at least one of:
importance of an augmented reality content to the user;
amount of the augmented reality content to be displayed; and
viewing orientation of the augmented reality content with respect to the set of surfaces.

14. The method of claim 1 , further comprising:
identifying a set of markers in the environment;
associating the set of markers with the set of surfaces; and
tracking the set of surfaces using the set of markers.

15. A computer program product comprising a non-transitory medium storing computer programs for execution by one or more computer systems, the computer program product comprising:
code for receiving, by a camera, a scan of an environment in view of a user;
code for identifying a set of surfaces in the environment for displaying user interface windows;
code for assigning a surface priority level to each surface in the set of surfaces, wherein the surface priority level for each surface is selected from a plurality of surface priority levels;
code for assigning a surface trackability level to each surface in the set of surfaces, wherein the surface trackability level for each surface is selected from a plurality of surface trackability levels;
code for assigning a content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment, wherein the content priority level for each content item is selected from a plurality of content priority levels; and
code for displaying each content item in the set of augmented reality contents for display on a surface in the set of surfaces in a display based on the surface priority level, the surface trackability level, and the content priority level.

16. The computer program product of claim 15 , wherein the environment comprises real world objects, and wherein the set of surfaces includes surfaces in top, bottom, left, right, and front view of the user; and wherein the display is a head mounted display.

17. The computer program product of claim 15 , wherein code for assigning the surface priority level to each surface in the set of surfaces comprises:
code for assigning the surface priority level based on: areas of the set of surfaces, orientations of the set of surfaces, and locations of the set of surfaces relative to field of view of the user, wherein the plurality of surface priority levels include high, medium and low surface priorit

Assignees

• Qualcomm Inc

Inventors

• Forutanpour Babak
• Ganesh Shriram
• Baker Daniel S
• Yamakawa Devender A

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title