Anchoring virtual images to real world surfaces in augmented reality systems

Patent metadata

Field	Value
Publication number	US-9384594-B2
Application number	US-201213434317-A
Country	US
Kind code	B2
Filing date	Mar 29, 2012
Priority date	Mar 29, 2011
Publication date	Jul 5, 2016
Grant date	Jul 5, 2016

Abstract

A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include capturing an image and spatial data with a body mounted camera and sensor array, receiving an input indicating a first anchor surface, calculating parameters with respect to the body mounted camera and displaying a virtual object such that the virtual object appears anchored to the selected first anchor surface. Further operations may include receiving a second input indicating a second anchor surface within the captured image that is different from the first anchor surface, calculating parameters with respect to the second anchor surface and displaying the virtual object such that the virtual object appears anchored to the selected second anchor surface and moved from the first anchor surface.

First claim

What is claimed is:

1. A method for rendering virtual images in an augmented reality system, comprising:
capturing an image with a body mounted camera;
capturing spatial data with a body mounted sensor array;
recognizing an object within the captured image;
receiving a first user input selecting a first anchor surface in the captured image that corresponds to a first surface located in the image;
calculating parameters including distance and orientation with respect to the body mounted camera that correspond to the first anchor surface;
displaying a virtual object so the virtual object appears anchored to the selected first anchor surface;
receiving a second input selecting a second anchor surface within the captured image that is different from the first anchor surface, wherein the second anchor surface corresponds to a second surface located in the image and the first and second surfaces are different;
calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the second anchor surface; and
displaying the virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface.

2. The method of claim 1 , further comprising continuously updating the display of the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface as the user turns his/her head and moves with respect to the selected first or second anchor surface.

3. The method of claim 1 , wherein displaying the virtual object comprises
displaying the virtual object on a head mounted display, the method further comprising outputting video images as if appearing on the head mounted display anchored to the selected first or second anchor surface.

4. The method of claim 1 , wherein displaying the generated virtual object so the virtual object appears to the user to be anchored to the selected second anchor surface and moved from the first anchor surface comprises:
generating the virtual object as a rectangular shaped virtual object that resembles a flat screen display; and
displaying the generated virtual object on a semitransparent display with the virtual object superimposed on a real world surface visible through the semitransparent display.

5. The method of claim 1 , wherein the first or second anchor surface comprises a wall in a room, and wherein the virtual object remains anchored on the wall in the room as the user moves about in the room.

6. The method of claim 1 , wherein displaying the generated virtual object so the virtual object appears anchored to the selected first or second anchor surface comprises generating the virtual object so that it appears to be contacting the first or second anchor surface and so that the first or second anchor surface appear to be a rigid point of support for the anchored virtual object.

7. The method of claim 1 , wherein:
displaying the generated virtual object so the virtual object appears anchored to the selected anchor surface comprises generating the virtual object so that it appears to be spaced apart from the first or second anchor surface but hovering in a fixed position and spaced relative to the first or second anchor surface.

8. The method of claim 3 , further comprising transmitting data comprising the captured spatial data, the generated virtual object and the image to a second head mounted display.

9. The method of claim 1 , further comprising:
receiving audio from the user utilizing a microphone;
applying a detection algorithm to the received audio to detect a predetermined audio signal; and
implementing the predefined user input when the predetermined audio signal is detected.

10. The method of claim 9 , wherein applying a detection algorithm to the received audio to detect a predetermined audio signal comprises applying a detection algorithm to the received audio to detect a predetermined audio signal spoken by the user corresponding to a predefined user input.

11. The method of claim 9 , wherein the predefined user input corresponding to the detected predetermined audio signal is one of the first input, the second input or a confirmation of one of the first and second inputs.

12. The method of claim 1 , further comprising:
capturing the image with the body mounted camera that is mounted and disposed together with a head mounted display; and
displaying the virtual object on the head mounted display.

13. The method of claim 12 , wherein the body mounted camera and head mounted display comprise a body mounted device, the method further comprising:
establishing a wireless data link between the body mounted device and a second body mounted device;
receiving audio data from the second body mounted device via the wireless data link; and
using the received audio data to output audio to the user.

14. The method of claim 13 , further comprising:
receiving video data via the wireless data link; and
using the received video data to output video images on the head mounted display received from the second body mounted device,
wherein the video displays an anchored virtual object.

15. The method of claim 14 , wherein receiving video data via the wireless data link comprises receiving one or more of a three dimensional model data for a virtual object, three dimensional data for an anchoring surface, three dimensional data for an anchoring point, three dimensional depth data, and three dimensional distance data.

16. The method of claim 1 , further comprising:
receiving a third input selecting a standby location for the virtual object, wherein the standby location is different from the first and the second anchor surfaces;
calculating parameters including distance and orientation with respect to the body mounted camera that corresponds to the standby location; and
displaying the virtual object at the standby location.

17. The method of claim 16 , wherein the standby location is displayed as free floating on a display, wherein the virtual object appears on the display as movable and unconnected to surfaces in the image as the user turns his/her head and moves.

18. The method of claim 1 , wherein:
capturing an image with a body mounted camera comprises capturing video data by a full-color video camera; and
capturing spatial data with a body mounted sensor array comprises capturing spatial data utilizing a distance sensor to determine distances to objects in the image.

19. The method of claim 18 , wherein capturing spatial data with a body mounted sensor array comprises capturing distance information of objects in the image using an infrared laser and an infrared sensor.

20. The method of claim 18 , further comprising building a three dimensional map based on the captured video data and the calculated spatial data.

21. The method of claim 1 , further comprising:
tracking movements of the user utilizing the body mounted camera;
applying a detection algorithm to the tracked user movements to detect a predetermined gesture; and
executing a command corresponding to the predetermined gesture when the predetermined gesture is detected.

22. The method of claim 21 , wherein the detected predetermined gesture is selected from the group consisting of an “OK” sign, a fist, an open hand, pointing with one finger, pointing with two fingers, pointing with three or four fingers, an outstretched hand, a hand rotation, a wave, a movement of one or more fingers, a m

Assignees

• Maciocci Giuliano
• Everitt Andrew J
• Mabbutt Paul
• Berry David T
• Qualcomm Inc

Inventors

• Maciocci Giuliano
• Everitt Andrew J
• Mabbutt Paul
• Berry David T

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• H04N9/3173

  wherein the projection device is specially adapted for enhanced portability · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title