Context adaptive user interface for augmented reality display

Patent metadata

Field	Value
Publication number	US-10019962-B2
Application number	US-201113212172-A
Country	US
Kind code	B2
Filing date	Aug 17, 2011
Priority date	Aug 17, 2011
Publication date	Jul 10, 2018
Grant date	Jul 10, 2018

Abstract

A user interface includes a virtual object having an appearance in context with a real environment of a user using a see-through, near-eye augmented reality display device system. A virtual type of object and at least one real world object are selected based on compatibility criteria for forming a physical connection like attachment, supporting or integration of the virtual object with the at least one real object. Other appearance characteristics, e.g. color, size or shape, of the virtual object are selected for satisfying compatibility criteria with the selected at least one real object. Additionally, a virtual object type and appearance characteristics of the virtual object may be selected based on a social context of the user, a personal context of the user or both.

First claim

What is claimed is:

1. A system for providing a user interface with a virtual object appearing in context with a real environment of a user using a see through, augmented reality display device system comprising:
a computer system communicatively coupled via a network to the see-through near-eye, augmented reality display device system for receiving image data of a real environment of a user, the computer system comprising one or more processors and a memory accessible by the one or more processors;
the one or more processors being configured to identify one or more appearance characteristics including surface characteristics of one or more real world objects in the real environment;
the one or more processor being configured to identify one or more types of realistic physical connections available for providing a realistic visual connectivity relationship based on any of the appearance characteristics identified for the one or more real world objects;
a set of graphical design rules being stored in the memory, the set of graphical design rules defining compatibility criteria between appearance characteristics including as between respective surface characteristics;
the one or more processors being configured to select a type of object for the virtual object and to select an anchor real world objects set from the one or more real world objects based on compatibility criteria of the set of graphical design rules for forming a realistic physical connection using one of the one or more of the identified types of realistic physical connection available;
the one or more processors being configured to select at least one virtual connecting member for realistic visual interposition between a member of the selected anchor real world objects set and the virtual object so as to provide said realistic visual connectivity relationship using the selected at least one virtual connecting member as being visually interposed between a member of the selected anchor real world objects set and the virtual object; and
the one or more processors being configured to generate display data for an image of the virtual object having the realistic physical connection with a selected member of the anchor real world objects set with the selected at least one virtual connecting member being visually interposed there between, the generated display data being usable by the see-through, near-eye, augmented reality display device system.

2. The system of claim 1 further comprising the computer system for receiving data from one or more applications executing in the see-through, near-eye, augmented reality display system for identifying content being represented by the virtual object for display.

3. The system of claim 2 further comprising the one or more processors having access to applications executing on the see-through, near-eye, augmented reality display device system of the user for determining at least one of a social context for the user or a personal context for the user.

4. The system of claim 2 wherein the one or more processors have access to one or more stored images of real world objects; and
the one or more processors for identifying one or more appearance characteristics of one or more real world objects in the real environment based on the stored images.

5. The system of claim 1 wherein the set of graphical design rules for defining compatibility criteria between appearance characteristics comprise definitions of at least one of the following:
complimentary colors;
a texture associated with a type of material;
a geometric orientation of a surface of a type of object; and
a type of physical connection available for the surface of the type of object.

6. The system of claim 1 wherein the set of graphical design rules link one or more animation effects with a type of appearance characteristic.

7. Processor-readable storage implemented as hardware storage having plural and respectively addressable storage locations for simultaneously storing respective addressable data items including instructions encoded therein for causing one or more processors to execute a method causing display of virtual imagery that appears to be realistically and in context integrated with a real world object of a real environment viewable by a user using a see-through, augmented reality display system, the executed method comprising:
first identifying a real world object present in the viewable real environment and to which predetermined virtual imagery can be visually coupled by way of a visually interposed virtual connecting member such that the predetermined virtual imagery appears to be realistically and in context connectively integrated with the identified real world object;
second identifying a virtual connecting member usable for realistic connective disposition visually between the predetermined virtual imagery and the identified real world object so as to provide said realistic and in context, visually connective integration of the virtual imagery with the identified real world object;
generating display data for displaying the virtual connecting member as being visually connectedly interposed between the predetermined virtual imagery and the identified real world object; and
using the generated display data to cause display by the see-through, augmented reality display system of the predetermined virtual imagery as being realistically and in context, visually connectively integrated with the identified real world object.

8. The processor-readable storage media of claim 7 wherein:
said first identifying includes identifying one or more real world appearance characteristics including those of respective surfaces of one or more real world objects present in the real environment;
said second identifying includes identifying one or more types of realistic physical connections that may be used during display to provide visually realistic virtual connectivities between the predetermined virtual imagery and at least one of the real world objects based on at least one of the identified real world surface appearance characteristics of the one or more real world objects;
the method further comprising:
based on the identified types of realistic physical connections, selecting a virtual object type from one or more virtual object type candidates capable of realistically being realistically portrayed as having at least one of the identified types of realistic physical connections; and
selecting from an anchor candidates group of the one or more real world objects, a set whose members can each serve as a realistic physical connection providing anchor to be visually connected by way of the identified virtual connecting member to the predetermined virtual imagery based on the selected virtual object type, the members of the selected set of the one or more real objects being capable of forming a realistic physical connection with at least one of the virtual objects of the selected virtual object type and vice versa.

9. The processor-readable storage media of claim 8 wherein the selecting of the virtual object type and the selecting of the anchor real world objects set comprises:
selecting the anchor real world objects set based on positions of the real world objects of the anchor real world object set with respect to a user field of view of the see-through, augmented reality display device system.

10. The processor-readable storage media of claim 8 wherein the method further comprises:
identifying a social context of the user based on user profile data; and
selecting the virtual object type based on the social context of the user.

11. The processor-readable storage media of claim 10 wherein the method further comprises:

Assignees

• Liu James C
• Andrews Anton O
• Vaught Benjamin I
• Maitlen Craig R
• Novak Christopher M
• Small Sheridan Martin
• Microsoft Technology Licensing Llc

Inventors

• Liu James C
• Andrews Anton O
• Vaught Benjamin I
• Maitlen Craig R
• Novak Christopher M
• Small Sheridan Martin

Classifications

• H04N13/344

  with head-mounted left-right displays · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G09G3/3413

  Details of control of colour illumination sources · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title