User interface for integrated gestural interaction and multi-user collaboration in immersive virtual reality environments

Patent metadata

Field	Value
Publication number	US-11599237-B2
Application number	US-202117175439-A
Country	US
Kind code	B2
Filing date	Feb 12, 2021
Priority date	Dec 18, 2014
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

The technology disclosed relates to user interfaces for controlling augmented reality environments. Real and virtual objects can be seamlessly integrated to form an augmented reality by tracking motion of one or more real objects within view of a wearable sensor system using a combination a RGB (red, green, and blue) and IR (infrared) pixels of one or more cameras. It also relates to enabling multi-user collaboration and interaction in an immersive virtual environment. In particular, it relates to capturing different sceneries of a shared real world space from the perspective of multiple users. The technology disclosed further relates to sharing content between wearable sensor systems. In particular, it relates to capturing images and video streams from the perspective of a first user of a wearable sensor system and sending an augmented version of the captured images and video stream to a second user of the wearable sensor system.

First claim

What is claimed is:

1. A method of tracking motion of a wearable sensor system, the method including:
generating, for display, a presentation including (i) a visual rendering of at least a portion of a detected appendage and (ii) at least one instance of a virtual device including a contextual menu providing a virtual interface for interacting with multiple electronic devices included in a scene of a real world space, the at least one instance of the virtual device being affixed to the visual rendering of the portion of the appendage, wherein the contextual menu included in the virtual device includes menu items facilitating a control interface for changing operational modes of at least one electronic device of the multiple electronic devices; and
responsive to a selection of one of the menu items of the contextual menu to select an operation of a particular electronic device of the multiple electronic devices, updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device.

2. The method of claim 1 , further comprising determining that a user gesture indicates a request to interact with the particular electronic device of the multiple electronic devices included in the real world space.

3. The method of claim 1 , wherein the presentation is generated for display across an interface of the wearable sensor system and wherein the method further includes automatically sizing the virtual device to fit the presentation based on determined motion information of the wearable sensor system with respect to at least one detected feature of the appendage.

4. The method of claim 3 , wherein the automatic sizing of the virtual device further includes adjusting a size of the virtual device in relation to a size of the appendage.

5. The method of claim 1 , wherein the contextual menu included in the virtual device corresponds to a unique control feature that controls a specific operation of the particular electronic device.

6. The method of claim 1 , wherein the virtual device includes an ornamentation.

7. The method of claim 1 , wherein the virtual device includes a scroll bar.

8. The method of claim 1 , wherein the virtual device includes a knob.

9. The method of claim 1 , wherein the virtual device includes a slider.

10. The method of claim 1 , wherein the virtual device includes a button.

11. The method of claim 1 , wherein the virtual device includes a virtual object indicating a command input.

12. A method of providing an augmented reality environment, the method including:
integrating a virtual device including a contextual command input onto a representation of a detected user body portion, the contextual command input providing an interaction with multiple electronic devices in a real world space;
generating, for display, a presentation including (i) a visual rendering of the user body portion and (ii) at least one instance of the virtual device affixed to the visual rendering of the user body portion, wherein the contextual command input included in the virtual device facilitates a control interface for changing operational modes of at least one electronic device of the multiple electronic devices; and
responsive to a selection of one item of the contextual command input to select an operation of a particular electronic device of the multiple electronic devices, updating the contextual command input included in the virtual device to include items specifically for the particular electronic device.

13. The method of claim 12 , wherein the integrating includes:
determining that a user gesture indicates a request to interact with the particular electronic device of the multiple electronic devices included in the real world space; and
adjusting a size of the virtual device to correspond to a size of the representation of the user body portion.

14. The method of claim 13 , wherein the adjusting includes:
sizing the virtual device to maintain a fixed relationship between the size of the virtual device and the size of the representation of the user body portion.

15. A system including a memory and one or more processors, the memory storing computer program instructions for providing an augmented reality environment, the computer program instructions, when executed on the one or more processors, causing the one or more processors to implement the method of claim 13 .

16. A non-transitory computer readable storage medium impressed with computer program instructions for providing an augmented reality environment, the computer program instructions, when executed on one or more processors, causing the one or more processors to implement the method of claim 13 .

17. The method of claim 12 , further comprising:
determining, based at least upon an identity of the user body portion, at least one point to which the virtual device can be affixed.

18. The method of claim 12 , wherein one or more cameras capture a position of the user body portion in at least one of an RGB mode with ambient illumination and an infrared (IR) mode when an IR source is activated.

19. A system, comprising:
a processing system configured to (i) detect, from information sensed from a real world space, at least one of a position and a motion of an appendage of a user, (ii) determine at least one point on the appendage to which a virtual device including a contextual menu can be affixed for interacting with multiple electronic devices included in a scene of the real world space, and (iii) detect, from the sensed information, a finger gesture made freely in 3D space; and
a processing system configured to generate, for display, a presentation including (i) a visual rendering of at least a portion of a detected appendage and (ii) at least one instance of a virtual device including a contextual menu providing a virtual interface for interacting with multiple electronic devices included in a scene of a real world space, the at least one instance of the virtual device being affixed to the visual rendering of the portion of the appendage, wherein the contextual menu included in the virtual device includes menu items facilitating a control interface for changing operational modes of at least one electronic device of the multiple electronic devices,
wherein, the processing system, responsive to a selection of one of the menu items of the contextual menu to select an operation of a particular electronic device of the multiple electronic devices, updating the contextual menu included in the virtual device to include menu items specifically for the particular electronic device.

20. A non-transitory computer readable storage medium impressed with computer program instructions to track motion of a wearable sensor system, the computer program instructions, when executed on a processor, implement a method comprising:
generating for display, a presentation including (i) a visual rendering of the at least a portion of a detected appendage and (ii) at least one instance of a virtual device including a contextual menu providing a virtual interface for interacting with multiple electronic devices included in a scene of a real world space, the at least one instance of the virtual device being affixed to the visual rendering of the portion of the appendage, wherein the contextual menu included in the virtual device includes menu items facilitating a control interface for changing operational modes of at least one electronic device of the multiple electronic devices; and
responsive to a selection of one of the menu items

Assignees

• Ultrahaptics Ip Two Ltd

Inventors

• Holz David S
• Fox Barrett
• Hay Kyle A
• Hare Gabriel A
• Yu Wilbur Yung Sheng
• Edelhart Dave
• Medich Jody
• Plemmons Daniel

Classifications

• G06F3/04812 Primary

  Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title