Optical hand tracking in virtual reality systems

Patent metadata

Field	Value
Publication number	US-2017102775-A1
Application number	US-201615288453-A
Country	US
Kind code	A1
Filing date	Oct 7, 2016
Priority date	Oct 8, 2015
Publication date	Apr 13, 2017
Grant date	—

Abstract

A system tracks movement of the VR input device relative to a portion of a user's skin, track movement of the VR input device relative to a physical surface external to the VR input device, or both. The system includes an illumination source integrated with a tracking glove coupled to a virtual reality console, and the illumination source is configured to illuminate a portion of skin on a finger of a user. The system includes an optical sensor integrated with the glove, and the optical sensor is configured to capture a plurality of images of the illuminated portion of skin. The system includes a controller configured to identify differences between one or more of the plurality of images, and to determine estimated position data based in part on the identified differences.

First claim

What is claimed is:

1 . A tracking device comprising:
an illumination source integrated with a tracking glove coupled to a virtual reality console, the illumination source configured to illuminate a portion of skin on a finger of a user; an optical sensor integrated with the-glove and configured to capture a plurality of images of the illuminated portion of skin; and a controller configured to:
identify differences between some of the plurality of images, the differences corresponding to movement of the tracking glove with respect to the illuminated portion of skin, and
determine estimated position data for the tracking glove based in part on the identified differences.

2 . The tracking device of claim 1 , wherein the controller is further configured to:
determine that the tracking glove has changed its configuration with respect to the user's hand based on the estimated positions, the configuration selected from a group consisting of: detachment of the tracking glove from the user's hand, slip of the tracking glove from the user's hand, or both detachment of the tracking glove from the user's hand and slip of the tracking glove from the user's hand.

3 . The tracking device of claim 1 , further comprising:
an additional illumination source configured to illuminate a portion of surface external to an exterior of the tracking glove; an additional optical sensor configured to capture a plurality of images of the illuminated portion of the surface; and the controller further configured to:
identify differences between some of the plurality of images, the differences corresponding to movement of the tracking glove with respect to the surface, and
determine estimated position data based in part on the identified differences.

4 . The tracking device of claim 3 , the controller is further configured to
determine one or more actions performed by the user associated with content presented to the user based in part on the movement of the tracking glove.

5 . The tracking device of claim 3 , wherein the additional optical sensor is located on an area selected from a group consisting of: a portion of the tracking glove near a fingertip of the user, multiple portions of the tracking glove near multiple fingertips of the user, a portion of the glove near a finger pad of the user, multiple portions of the glove near multiple finger pads of the user, or some combination thereof.

6 . The tracking device of claim 3 , wherein the one or more identified actions performed by the user are selected from a group consisting of: repositioning a cursor within content presented to the user, selecting content presented to the user, navigating through content presented to the user, typing action performed by the user, or some combination thereof.

7 . The tracking device of claim 1 , wherein the controller is further configured to:
identify differences between some of the plurality of images, the differences corresponding to movement of the illuminated portion of skin, and determine estimated position data for the finger based in part on the identified differences.

8 . The tracking device of claim 7 , wherein the movement of the illuminated portion of skin comprises at least bending of a particular joint of the user's finger.

9 . The tracking device of claim 1 , wherein the controller is further configured to:
identify common features of different images of the plurality of images; identify differences between the common features in the different images of the plurality of images; and determine estimated position data based in part on the identified differences between the common features

10 . The tracking device of claim 1 , wherein the controller is further configured to:
select a set of images from the plurality of images based on the identified differences; generate a corresponding set of displacement vectors based on the selected set of images; and generate a total displacement vector based on the generated set of displacement vectors, the total displacement vector indicating a change in position of the tracking glove relative to the user.

11 . The tracking device of claim 1 , wherein the portion of skin is selected from a group consisting of: a portion of skin around a joint of the user's finger, a portion of skin around fingertip of the user's finger, a portion of palmar skin of the user's hand, a portion of dorsal skin of the user's hand, or some combination thereof.

12 . The tracking device of claim 1 , wherein a resolution of the imaging device is at most 16×16 pixels.

13 . The tracking device of claim 1 , wherein the imaging device is configured to capture video with a frame rate of at least 1 kilo-Hertz.

14 . A system comprising:
a tracking glove that tracks movement of the tracking glove with respect to a user's skin, comprising:
an illumination source integrated with the tracking glove coupled to a virtual reality console, the illumination source configured to illuminate a portion of skin on a finger of the user, and
an optical sensor integrated with the-glove and configured to capture a plurality of images of the illuminated portion of skin; and
a controller configured to:
identify differences between some of the plurality of images, the differences corresponding to movement of the tracking glove with respect to the illuminated portion of skin,
determine estimated position data for the tracking glove based in part on the identified differences, and
generate content based in part on the determined estimated position data; and

an electronic display configured to display the generated content to the user.

15 . The system of claim 14 , wherein the controller is further configured to:
determine that the tracking glove has changed its configuration with respect to the user's hand based on the determined estimated positions, the configuration selected from a group consisting of: detachment of the tracking glove from the user's hand, slip of the tracking glove from the user's hand, and both detachment of the tracking glove from the user's hand and slip of the tracking glove from the user's hand.

16 . The system of claim 15 , wherein responsive to determining that the glove has slipped from the user' hand, the controller is further configured to:
determine an offset between generated content associated with the movement of the tracking glove and a true pose of the user hand for compensating changes of the glove configuration.

17 . The system of claim 15 , wherein responsive to determining that the tracking glove has detached from the user' hand, the controller is further configured to:
generate content indicating the tracking glove has lost contact with the user's hand for presentation.

18 . The system of claim 14 , further comprising:
an additional illumination source configured to illuminate a portion of surface external to an exterior of the tracking glove; an additional optical sensor configured to capture a plurality of images of the illuminated portion of the surface; and the controller further configured to:
identify differences between some of the plurality of images, the differences corresponding to movement of the tracking glove with respect to the surface,
determine estimated position data for the finger based in part on the identified differences, and
determine one or more actions performed by the user associated with content presented to the user based in part on the movement of the tracking glove.

Assignees

• Oculus Vr Llc

Inventors

• Keller Sean Jason
• Trutna Tristan Thomas
• Perek David R
• Cleary Iii Bruce A
• Scally Brian Michael

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/005

  Input arrangements through a video camera · CPC title