Holographic object feedback

Patent metadata

Field	Value
Publication number	US-9367136-B2
Application number	US-201313862210-A
Country	US
Kind code	B2
Filing date	Apr 12, 2013
Priority date	Apr 12, 2013
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

Methods for providing real-time feedback to an end user of a mobile device as they are interacting with or manipulating one or more virtual objects within an augmented reality environment are described. The real-time feedback may comprise visual feedback, audio feedback, and/or haptic feedback. In some embodiments, a mobile device, such as a head-mounted display device (HMD), may determine an object classification associated with a virtual object within an augmented reality environment, detect an object manipulation gesture performed by an end user of the mobile device, detect an interaction with the virtual object based on the object manipulation gesture, determine a magnitude of a virtual force associated with the interaction, and provide real-time feedback to the end user of the mobile device based on the interaction, the magnitude of the virtual force applied to the virtual object, and the object classification associated with the virtual object.

First claim

What is claimed is:

1. An electronic device for providing feedback associated with virtual object interactions, comprising:
one or more processors, the one or more processors acquire a first virtual object and determine an object classification associated with the first virtual object, the one or more processors detect an object manipulation gesture performed by an end user of the electronic device and detect an interaction with the first virtual object based on the detection of the object manipulation gesture, the one or more processors determine a magnitude of a virtual force associated with the interaction, the one or more processors determine a feedback response to the interaction based on the magnitude of the virtual force and the object classification, the one or more processors output the feedback response, the one or more processors detect that a palm of the end user's hand is facing away from the electronic device, the one or more processors render one or more images corresponding with the first virtual object such that a first portion of the first virtual object is rendered and a second portion of the first virtual object that would be occluded by the end user's hand is not rendered in response to detecting that the palm of the end user's hand is facing away from the mobile device, the one or more processors cause a position of the first virtual object within the augmented reality environment to be snapped to a center of the end user's hand prior to rendering the one or more images in response to detecting the interaction with the first virtual object, the object manipulation gesture corresponds with a grabbing gesture; and
a see-through display in communication with the one or more processors, the see-through display displays the one or more images of the first virtual object such that the first virtual object is perceived to exist within the augmented reality environment.

2. The electronic device of claim 1 , wherein:
the one or more processors determine a material property associated with the first virtual object, the one or more processors determine the feedback response based on the magnitude of the virtual force and the material property.

3. The electronic device of claim 1 , wherein:
the one or more processors determine a region within the augmented reality environment associated with the first virtual object, the one or more processors detect the interaction with the first virtual object by detecting the object manipulation gesture being performed within the region, the region extends beyond the first virtual object by a particular distance.

4. The electronic device of claim 1 , wherein:
the feedback response includes a haptic response, the haptic response comprises a vibrating of the electronic device.

5. The electronic device of claim 1 , wherein:
the one or more processors shift a position of the first virtual object within the augmented reality environment such that the first portion of the first virtual object appears between a thumb and index finger of the end user's hand.

6. The electronic device of claim 1 , wherein:
the feedback response includes a haptic response, the haptic response comprises a vibrating of a second electronic device different from the electronic device at a particular magnitude that is proportional to the magnitude of the virtual force.

7. A method for providing feedback associated with virtual object interactions, comprising:
acquiring one or more virtual objects associated with an augmented reality environment, the one or more virtual objects include a first virtual object;
determining an object classification associated with the first virtual object;
detecting an interaction with the first virtual object by an end user of a mobile device;
determining a magnitude of a virtual force associated with the interaction;
determining a feedback response in response to detecting the interaction based on the magnitude of the virtual force and the object classification;
outputting the feedback response;
detecting that a palm of the end user's hand is facing away from the mobile device;
rendering one or more images corresponding with the first virtual object such that a first portion of the first virtual object is rendered and a second portion of the first virtual object that would be occluded by the end user's hand is not rendered in response to detecting that the palm of the end user's hand is facing away from the mobile device, the rendering one or more images corresponding with the first virtual object includes snapping a position of the first virtual object to a center of the end user's hand prior to rendering the one or more images in response to detecting the interaction with the first virtual object, the interaction corresponds with a grabbing gesture; and
displaying the one or more images on the mobile device.

8. The method of claim 7 , further comprising:
determining a material property associated with the first virtual object, the determining a feedback response to the interaction includes determining the feedback response based on the magnitude of the virtual force and the material property.

9. The method of claim 7 , further comprising:
determining a region within the augmented reality environment associated with the first virtual object, the detecting an interaction with the first virtual object includes detecting the object manipulation gesture being performed within the region.

10. The method of claim 9 , wherein:
the region extends beyond the first virtual object by a particular distance.

11. The method of claim 7 , wherein:
the rendering one or more images includes shifting a position of the first virtual object within the augmented reality environment such that the first portion of the first virtual object appears between a thumb and index finger of the end user's hand.

12. The method of claim 7 , wherein:
the feedback response includes a haptic response, the haptic response comprises a vibrating of an electronic device at a particular magnitude that is proportional to the magnitude of the virtual force.

13. The method of claim 7 , wherein:
the object classification is associated with metallic virtual objects.

14. The method of claim 7 , wherein:
the mobile device comprises an HMD;
the acquiring one or more virtual objects is performed by the HMD;
the detecting an interaction with the first virtual object is performed by the HMD; and
the determining a feedback response to the interaction is performed by the HMD.

15. One or more physical hardware storage devices containing processor readable code for programming one or more processors to perform a method for providing feedback associated with virtual object interactions comprising the steps of:
acquiring a first virtual object associated with an augmented reality environment;
determining an object classification associated with the first virtual object;
detecting an object manipulation gesture within the augmented reality environment performed by an end user of a mobile device;
detecting an interaction with the first virtual object in response to detecting the object manipulation gesture;
determining a magnitude of a virtual force associated with the interaction;
determining a feedback response to the interaction based on the magnitude of the virtual force and the object classification;
outputting the feedback response, the feedback response includes at least one of a visual response, an audio response, or a haptic response;
detecting that a palm of the end user's hand is facing away from the mobile device;
rendering one or more images corresponding with the firs

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Latta Stephen G
• Poulos Adam G
• Brown Cameron G
• Mcculloch Daniel J
• Kaplan Matthew
• Navratil Arnulfo Zepeda
• Paulovich Jon
• Tsunoda Kudo

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title