Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments

Patent metadata

Field	Value
Publication number	US-9696795-B2
Application number	US-201514626898-A
Country	US
Kind code	B2
Filing date	Feb 19, 2015
Priority date	Feb 13, 2015
Publication date	Jul 4, 2017
Grant date	Jul 4, 2017

Abstract

The technology disclosed relates to a method of realistic rotation of a virtual object for an interaction between a control object in a three-dimensional (3D) sensory space and the virtual object in a virtual space that the control object interacts with. In particular, it relates to detecting free-form gestures of a control object in a three-dimensional (3D) sensory space and generating for display a 3D solid control object model for the control object during the free-form gestures, including sub-components of the control object and in response to detecting a two e sub-component free-form gesture of the control object in the 3D sensory space in virtual contact with the virtual object, depicting, in the generated display, the virtual contact and resulting rotation of the virtual object by the 3D solid control object model.

First claim

What is claimed is:

1. A method of manipulating virtual objects using real motions of one or more hands in a three-dimensional (3D) sensory space, the method including:
capturing an image of the hands in the a three-dimensional (3D) sensory space and sensing a location of the hands;
incorporating the image of the hands into a virtual reality scene; and
outlining in a contrasting color a modeled position of the location of the hands and incorporating the outline into the virtual reality scene.

2. The method of claim 1 , further including changing an appearance of the outline of the hands in the virtual reality scene upon detection of a discrepancy between the image of the hands and the outline.

3. The method of claim 1 , further including changing an appearance of the image of the hands in the virtual reality scene upon detection of a discrepancy between the image of the hands and the outline.

4. A system implementing the method of claim 1 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of the hands in the a three-dimensional (3D) sensory space and sense a location of the hands;
incorporate the image of the hands into a virtual reality scene; and
outline in a contrasting color a modeled position of the location of the hands and incorporate the outline into the virtual reality scene.

5. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 1 , the method including:
capturing an image of the hands in the a three-dimensional (3D) sensory space and sensing a location of the hands;
incorporating the image of the hands into a virtual reality scene; and
outlining in a contrasting color a modeled position of the location of the hands and incorporating the outline into the virtual reality scene.

6. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:
capturing an image of at least one hand in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene; and
sensing a pinch action between a thumb and first finger of the first hand and rendering a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.

7. The method of claim 6 , further including:
sensing a pinch action between a thumb and first finger of a second hand and rendering a second virtual pinch force image positioned between the thumb and the first finger of the second hand while the pinch action continues;
sensing a movement of at least one of the first and second hands that increases a separation distance between the first and second hands, while sensing continuing pinching actions of both the first and second hands; and
rendering a new virtual object between the first and second pinch force images, responsive to the increased separation distance, wherein at least a size of the new virtual object is responsive to the separation distance.

8. The method of claim 7 , wherein an orientation of the new virtual object is responsive to positions of the first and second hands.

9. A system implementing the method of claim 6 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of at least one hand in a three-dimensional (3D) sensory space and sense a location of a first hand;
incorporate the image of the first hand into a virtual reality scene; and
sense a pinch action between a thumb and first finger of the first hand and render a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.

10. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 6 , the method including:
capturing an image of at least one hand in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene; and
sensing a pinch action between a thumb and first finger of the first hand and rendering a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.

11. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene that includes a grabable virtual object;
sensing a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determining whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determining further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a gesture rate at which the first hand closes on the virtual object;
a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and
linear velocity of the palm of the first hand relative to the virtual object; and

responsive to determining that a gesture of the first hand is intended to grab the virtual object, linking motion of the first hand to manipulation of the virtual object.

12. The method of claim 11 , further including taking into account when determining whether the gesture is intended to interact with the grabable virtual object a maximum separation between any of the fingertips of the first hand and the proximate surface.

13. The method of claim 11 , further including taking into account when determining whether the gesture is intended to interact with the grabable virtual object a rotational velocity of the palm of the first hand in the 3D sensory space.

14. A system implementing the method of claim 11 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of a first hand;
incorporate the image of the first hand into a virtual reality scene that includes a grab able virtual object;
sense a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determine whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determination further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a gesture rate at which the first hand closes on the virtual object;
a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and
linear velocity of the palm of the first hand relative to the virtual object; and

responsive to the determination that a gesture of the first hand is intended to grab the virtual object, link motion of the firs

Assignees

• Leap Motion Inc

Inventors

• Marcolina Alex
• Holz David S
• Durdik Paul A

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T15/08

  Volume rendering · CPC title