Systems and methods of rerendering image hands to create a realistic grab experience in virtual reality/augmented reality environments

Patent metadata

Field	Value
Publication number	US-11244513-B2
Application number	US-202016815533-A
Country	US
Kind code	B2
Filing date	Mar 11, 2020
Priority date	Sep 8, 2015
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

The technology disclosed relates to a method of realistic rendering of a real object as a virtual object in a virtual space using an offset in the position of the hand in a three-dimensional (3D) sensory space. An offset between expected positions of the eye(s) of a wearer of a head mounted device and a sensor attached to the head mounted device for sensing a position of at least one hand in a three-dimensional (3D) sensory space is determined. A position of the hand in the three-dimensional (3D) sensory space can be sensed using a sensor. The sensed position of the hand can be transformed by the offset into a re-rendered position of the hand as would appear to the wearer of the head mounted device if the wearer were looking at the actual hand. The re-rendered hand can be depicted to the wearer of the head mounted device.

First claim

What is claimed is:

1. A system including:
a processor; and
a memory storing computer instructions that, when executed by the processor, perform operations comprising:
calculating a transformation matrix using an offset identified between an eye of a user and an actual position of a gesture sensor, such that the transformation matrix, when applied to a sensed position of a control object, decreases a rendered size of the control object in accordance with the identified offset;
applying the transformation matrix to the sensed position of the control object to transform the sensed position of the control object into a re-rendered position of the control object, the re-rendered position of the control object placing, in a three-dimensional (3D) sensory space, the control object at the offset with respect to the sensed position of the control object, such that the user perceives the re-rendered position of the control object, situated in the 3D sensory space, as expected by the user when the user is looking at the actual control object in the 3D sensory space; and
providing for display, to the user, a re-rendered control object image using the re-rendered position of the control object.

2. The system of claim 1 , further configured to perform:
sensing additional positions of the control object and determining a motion or gesture from the sensed additional positions; and
applying the offset to the additional positions of the control object to transform the sensed additional positions to re-rendered positions of the control object as seen by the user, thereby providing the re-rendered control object image having the motion or gesture of the control object.

3. The system of claim 2 , further configured to perform:
sensing an indication of quality of tracking for the gesture sensor, the indication of quality of tracking indicating whether the gesture sensor is tracking a position or motion of the control object; and
adding to the re-rendered control object image a visual indication corresponding to the indication of quality of tracking, thereby providing the user with a visual cue of how well the gesture sensor is tracking the control object.

4. The system of claim 3 , wherein the indication of quality of tracking includes an indication that the gesture sensor is able to recognize the control object in captured images.

5. The system of claim 4 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered control object image when the gesture sensor is able to recognize the control object in captured images.

6. The system of claim 4 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered control object image when the gesture sensor is unable to recognize the control object in captured images.

7. A method comprising:
calculating a transformation matrix using an offset identified between an eye of a user and an actual position of a gesture sensor, such that the transformation matrix, when applied to a sensed position of a control object, decreases a rendered size of the control object in accordance with the identified offset;
applying the transformation matrix to the sensed position of the control object to transform the sensed position of the control object into a re-rendered position of the control object, the re-rendered position of the control object placing, in a three-dimensional (3D) sensory space, the control object at the offset with respect to the sensed position of the control object, such that the user perceives the re-rendered position of the control object, situated in the 3D sensory space, as expected by the user when the user is looking at the actual control object in the 3D sensory space; and
providing for display, to the user, a re-rendered control object image using the re-rendered position of the control object.

8. The method of claim 7 , further including:
sensing additional positions of the control object and determining a motion or gesture from the sensed additional positions; and
applying the offset to the additional positions of the control object to transform the sensed additional positions to re-rendered positions of the control object as seen by the user, thereby providing the re-rendered control object image having the motion or gesture of the control object.

9. The method of claim 8 , further including:
sensing an indication of quality of tracking for the gesture sensor, the indication of quality of tracking indicating whether the gesture sensor is tracking a position or motion of the control object; and
adding to the re-rendered control object image a visual indication corresponding to the indication of quality of tracking, thereby providing the user with a visual cue of how well the gesture sensor is tracking the control object.

10. The method of claim 9 , wherein the indication of quality of tracking includes an indication that the gesture sensor is able to recognize the control object in captured images.

11. The method of claim 10 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered control object image when the gesture sensor is able to recognize the control object in captured images.

12. The method of claim 10 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered control object image when the gesture sensor is unable to recognize the control object in captured images.

13. A non-transitory computer-readable recording medium having instructions recorded thereon, the instructions, when executed by a processor, perform a method comprising:
calculating a transformation matrix using an offset identified between an eye of a user and an actual position of a gesture sensor, such that the transformation matrix, when applied to a sensed position of a control object, decreases a rendered size of the control object in accordance with the identified offset;
applying the transformation matrix to the sensed position of the control object to transform the sensed position of the control object into a re-rendered position of the control object, the re-rendered position of the control object placing, in a three-dimensional (3D) sensory space, the control object at the offset with respect to the sensed position of the control object, such that the user perceives the re-rendered position of the control object, situated in the 3D sensory space, as expected by the user when the user is looking at the actual control object in the 3D sensory space; and
providing for display, to the user, a re-rendered control object image using the re-rendered position of the control object.

14. The non-transitory computer-readable recording medium of claim 13 , wherein the method further comprises:
sensing additional positions of the control object and determining a motion or gesture from the sensed additional positions; and
applying the offset to the additional positions of the control object to transform the sensed additional positions to re-rendered positions of the control object as seen by the user, thereby providing the re-rendered control object image having the motion or gesture of the control object.

15. The non-transitory computer-readable recording medium of claim 14 , wherein the method further comprises:
sensing an indication of quality of tracking for the gesture sensor, the indication of quality of tracking indicating whether the g

Assignees

• Ultrahaptics Ip Two Ltd

Inventors

• Marcolina Alex
• Holz David

Classifications

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title