User interface for integrated gestural interaction and multi-user collaboration in immersive virtual reality environments

Patent metadata

Field	Value
Publication number	US-10921949-B2
Application number	US-201916510535-A
Country	US
Kind code	B2
Filing date	Jul 12, 2019
Priority date	Dec 18, 2014
Publication date	Feb 16, 2021
Grant date	Feb 16, 2021

Abstract

The technology disclosed relates to user interfaces for controlling augmented reality environments. Real and virtual objects can be seamlessly integrated to form an augmented reality by tracking motion of one or more real objects within view of a wearable sensor system using a combination a RGB (red, green, and blue) and IR (infrared) pixels of one or more cameras. It also relates to enabling multi-user collaboration and interaction in an immersive virtual environment. In particular, it relates to capturing different sceneries of a shared real world space from the perspective of multiple users. The technology disclosed further relates to sharing content between wearable sensor systems. In particular, it relates to capturing images and video streams from the perspective of a first user of a wearable sensor system and sending an augmented version of the captured images and video stream to a second user of the wearable sensor system.

First claim

What is claimed is:

1. A method of tracking motion of a wearable sensor system, the method including:
detecting, from a video stream, (i) a hand of a user, (ii) multiple electronic devices included in a scene of a real world space, and (iii) one or more features of the detected hand;
determining at least one point on the hand to which a virtual device including a contextual menu can be affixed, the virtual device providing a virtual interface for interacting with the multiple electronic devices included in the scene of the real world space;
generating, for display, a presentation including (i) a visual rendering of the hand and (ii) at least one instance of the virtual device affixed to the visual rendering of the hand, wherein the contextual menu included in the virtual device includes menu items facilitating a control interface for changing operational modes of the multiple electronic devices included in the scene of the real world space; and
responsive to a selection of one of the menu items of the contextual menu to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device and (ii) determining that a user gesture indicates a request to interact with the particular electronic device of the multiple electronic devices included in the scene of the real world space.

2. The method of claim 1 , wherein the presentation is generated for display across an interface of the wearable sensor system.

3. The method of claim 2 , further including automatically sizing the virtual device to fit the presentation based on determined motion information of the wearable sensor system with respect to at least one feature of the detected one or more features of the hand.

4. The method of claim 3 , wherein the automatic sizing of the virtual device further includes adjusting a size of the virtual device in relation to a size of the hand.

5. The method of claim 1 , wherein the one or more features of the hand also includes one or more features of a body portion engaged with the wearable sensor system.

6. The method of claim 1 , wherein the contextual menu included in the virtual device corresponds to a unique control feature that controls a specific operation of the particular electronic device.

7. The method of claim 1 , wherein the virtual device includes an ornamentation.

8. The method of claim 1 , wherein the virtual device includes a scroll bar.

9. The method of claim 1 , wherein the virtual device includes a knob.

10. The method of claim 1 , wherein the virtual device includes a slider.

11. The method of claim 1 , wherein the virtual device includes a button.

12. The method of claim 1 , wherein the virtual device includes a virtual object indicating a command input.

13. A method of providing an augmented reality environment, the method including:
detecting, from a video stream, a representation of a user body portion and multiple electronic devices within a field of view;
determining at least one point on the representation of the user body portion to which a virtual device including a contextual command input can be affixed;
integrating the virtual device including the contextual command input onto the representation of the user body portion;
generating, for display, a presentation including (i) a visual rendering of the user body portion and (ii) at least one instance of the virtual device affixed to the visual rendering of the user body portion, wherein the contextual command input included in the virtual device facilitates a control interface for changing operational modes of the multiple electronic devices; and
responsive to a selection of one item of the contextual command input to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual command input included in the virtual device to include items specifically for the particular electronic device and (ii) determining that a user gesture indicates a request to interact with the particular electronic device of the multiple electronic devices included in the field of view.

14. The method of claim 13 , wherein the integrating includes:
adjusting a size of the virtual device to correspond to a size of the representation of the user body portion.

15. The method of claim 14 , wherein the adjusting includes:
sizing the virtual device to maintain a fixed relationship between the size of the virtual device and the size of the representation of the user body portion.

16. The method of claim 13 , wherein the determining of the at least one point includes:
determining, based at least upon an identity of the user body portion, the at least one point to which the virtual device can be affixed.

17. The method of claim 13 , wherein one or more cameras capture a position of the user body portion in at least one of an RGB mode with ambient illumination and an infrared (IR) mode when an IR source is activated.

18. A system including a memory and one or more processors, the memory storing computer program instructions for providing an augmented reality environment, the computer program instructions, when executed on the one or more processors, causing the one or more processors to implement the method of claim 13 .

19. A non-transitory computer readable storage medium impressed with computer program instructions for providing an augmented reality environment, the computer program instructions, when executed on one or more processors, causing the one or more processors to implement the method of claim 13 .

20. A system, comprising:
a processing system configured to (i) detect, from information sensed from a real world space, at least one of a position and a motion of a hand of a user, (ii) determine at least one point on the hand to which a virtual device including a contextual menu can be affixed for interacting with multiple electronic devices included in a scene of the real world space, (iii) detect multiple electronic devices included in a scene of the real world space and (iv) detect, from the sensed information, a finger gesture made freely in 3D space; and
a subsystem configured to generate, for display, a presentation including (i) a visual rendering of the hand and (ii) at least one instance of the virtual device affixed to the visual rendering of the hand, wherein the contextual menu included in the virtual device includes menu items facilitating a control interface for changing operational modes of the multiple electronic devices included in the scene of the real world space,
wherein, the processing system and the subsystem, responsive to a selection of one of the menu items of the contextual menu to select an operation of a particular electronic device of the multiple electronic devices, (i) updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device and (ii) determining that a user gesture indicates a request to interact with the particular electronic device of the multiple electronic devices included in the scene of the real world space.

21. A non-transitory computer readable storage medium impressed with computer program instructions to track motion of a wearable sensor system, the computer program instructions, when executed on a processor, implement a method comprising:
detecting, from a video stream, (i) a hand of a user, (ii) multiple electronic devices in

Assignees

• Ultrahaptics Ip Two Ltd

Inventors

• Holz David S
• Fox Barrett
• Hay Kyle A
• Hare Gabriel A
• Yu Wilbur Yung Sheng
• Edelhart Dave
• Medich Jody
• Plemmons Daniel

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04847

  Interaction techniques to control parameter settings, e.g. interaction with sliders or dials · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title