Systems and method of interacting with a virtual object

Patent metadata

Field	Value
Publication number	US-9767613-B1
Application number	US-201514626683-A
Country	US
Kind code	B1
Filing date	Feb 19, 2015
Priority date	Jan 23, 2015
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

The technology disclosed relates to a method of interacting with a virtual object. In particular, it relates to referencing a virtual object in an augmented reality space, identifying a physical location of a device in at least one image of the augmented reality space, generating for display a control coincident with a surface of the device, sensing interactions between at least one control object and the control coincident with the surface of the device, and generating data signaling manipulations of the control coincident with the surface of the device.

First claim

What is claimed is:

1. A method of interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the method including:
referencing the virtual object for display by the head mounted device in an augmented reality space;
identifying a physical location of a device in at least one image of the augmented reality space;
generating for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;
sensing interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;
generating data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device; and
emphasizing, in the augmented reality space and by the head mounted device, points of interaction between extremities of the device and the virtual control generated for display by the head mounted device.

2. The method of claim 1 , wherein the control object is a hand.

3. The method of claim 2 , wherein the control object is a portion of the hand.

4. The method of claim 1 , wherein the control object is a tool.

5. The method of claim 1 , wherein the control object is a combination of a hand and a tool.

6. The method of claim 1 , further including:
generating data representing the virtual control across an interface of the device.

7. The method of claim 1 , wherein the device is a smartphone.

8. The method of claim 1 , further including:
sensing interactions between the control object and the virtual control coincident with and superimposed upon the surface of the device when the control object touches the virtual control.

9. The method of claim 1 , further including:
sensing interactions between the control object and the virtual control generated for display by the head mounted device the interactions being sensed when the control object approaches the virtual control generated for display by the head mounted device within an initial hover proximity threshold.

10. The method of claim 1 , wherein the virtual object generated for display by the head mounted device in the augmented reality space is responsive to the manipulations of the virtual control generated for display by the head mounted device in the augmented reality space.

11. The method of claim 1 , further including:
sensing interactions between the control object and the virtual control generated for display by the head mounted device, the interactions being sensed using at least one phase conjugate component that creates a contrast between portions of the control object and a rest of the augmented reality space.

12. The method of claim 11 , wherein the at least one phase conjugate component is a retro-reflector.

13. The method of claim 1 , further including:
sensing gestures in a three-dimensional (3D) sensory space away from the surface and proximate to the virtual object using a motion sensory control device; and
generating data signaling manipulations of the virtual object in the augmented reality space responsive to the gestures.

14. The method of claim 1 , further including:
generating data virtually emphasizing at least part of the control object in the augmented reality space.

15. The method of claim 1 , further including:
generating data virtually representing the control object in the augmented reality space.

16. The method of claim 1 , further comprising:
identifying, by the head mounted device, a real world item other than the device;
determining, based on the identified real world item, a specific virtual control that corresponds to the identified real world item; and
implementing the specific virtual control as the virtual control coincident with and superimposed upon the surface of the device.

17. The method of claim 1 , further comprising:
identifying, by the head mounted device, a physical environment in which the head mounted device resides;
determining, based on the identified physical environment, a specific virtual control that corresponds to the identified physical environment; and
implementing the specific virtual control as the virtual control coincident with and superimposed upon the surface of the device.

18. A system of interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the system including:
a processor and a computer readable storage medium storing computer instructions configured to cause the processor to:
reference the virtual object for display by the head mounted device in an augmented reality space;
identify a physical location of a device in at least one image of the augmented reality space;
generate for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;
sense interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;
generate data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device; and
emphasize, in the augmented reality space and by the head mounted device, points of interaction between extremities of the device and the virtual control generated for display by the head mounted device.

19. A non-transitory machine readable medium, storing one or more instructions for interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the instructions when executed by one or more processors cause the one or more processors to perform the following:
reference a virtual object for display by the head mounted device in an augmented reality space;
identify a physical location of a device in at least one image of the augmented reality space;
generate for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;
sense interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;
generate data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device; and
emphasize, in the augmented reality space and by the head mounted device, points of interaction between extremities of the device and the virtual control generated for display by the head mounted d

Assignees

• Leap Motion Inc

Inventors

• Bedikian Raffi
• He Hongyuan (Jimmy)
• Holz David S

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G09G5/14

  Display of multiple viewports · CPC title

• G06F2203/04808

  Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen · CPC title