Controller tracking for multiple degrees of freedom

Patent metadata

Field	Value
Publication number	US-10852847-B2
Application number	US-201715660216-A
Country	US
Kind code	B2
Filing date	Jul 26, 2017
Priority date	Jul 26, 2017
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

A method for controller tracking with multiple degrees of freedom includes generating depth data at an electronic device based on a local environment proximate the electronic device. A set of positional data is generated for at least one spatial feature associated with a controller based on a pose of the electronic device, as determined using the depth data, relative to the at least one spatial feature associated with the controller. A set of rotational data is received that represents three degrees-of-freedom (3DoF) orientation of the controller within the local environment, and a six degrees-of-freedom (6DoF) position of the controller within the local environment is tracked based on the set of positional data and the set of rotational data.

First claim

What is claimed is:

1. A method, comprising:
generating depth data based on images of a reflection of a modulated light pattern captured at an electronic device of a local environment proximate the electronic device, the depth data comprising data corresponding to distances of spatial features of objects in the local environment from the electronic device;
determining a pose of the electronic device based on the depth data;
generating, based on the pose of the electronic device relative to at least one spatial feature comprising anatomical features or gestures associated with a controller, a set of positional data representing three degrees-of-freedom (3DoF) translation of the controller within the local environment;
receiving a set of rotational data representing 3DoF orientation of the controller within the local environment;
tracking a six degrees-of-freedom (6DoF) position of the controller within the local environment based on the set of positional data and the set of rotational data;
receiving a set of touchpad information from the controller, wherein the set of touchpad information includes a contact position of a thumb of a user's hand on a touchpad of the controller;
determining an offset of the contact position from a center axis of the touchpad; and
refining the 6DoF position of the controller based on a corrected pose of the electronic device relative to the thumb position, wherein the corrected pose accounts for the offset of the contact position.

2. The method of claim 1 , further comprising:
projecting, at the electronic device, the modulated light pattern into the local environment.

3. The method of claim 1 , further comprising:
identifying a position of the user's hand, wherein the at least one spatial feature includes a thumb position of the user's hand; and
determining the 6DoF position of the controller based on the pose of the electronic device relative to the thumb position.

4. The method of claim 3 , further comprising:
identifying a grasp pattern of the user's hand, wherein the at least one spatial feature further includes an identified relationship between the thumb position and the grasp pattern; and
determining the 6DoF position of the controller based on the pose of the electronic device relative to the identified relationship between the thumb position and the grasp pattern.

5. The method of claim 4 , further comprising:
determining, based on the identified relationship between the thumb position and the grasp pattern, an inferred thumb position when a thumb of the user's hand is not visible to the electronic device.

6. The method of claim 1 , further comprising:
identifying a position of a visible marking on a surface of the controller; and
determining the 6DoF position of the controller based on the pose of the electronic device relative to the visible marking.

7. A system for performing controller tracking, comprising:
a depth sensor to capture, at an electronic device, images of a reflection of a modulated light pattern and to generate depth data from a local environment proximate the electronic device based on the images, the depth data comprising data corresponding to distances of spatial features of objects in the local environment from the electronic device;
a hand-held controller including a touchpad configured to detect a contact position of a user's thumb; and
a processor to:
determine a pose of the electronic device based on the depth data;
generate a set of positional data representing three degrees-of-freedom (3DoF) translation of the hand-held controller within the local environment based on the pose of the electronic device relative to at least one spatial feature associated with the hand-held controller;
receive a set of rotational data representing a 3DoF orientation of the hand-held controller within the local environment;
track a 6DoF position of the hand-held controller within the local environment based on the set of positional data and the set of rotational data;
determine an offset of the contact position from a center axis of the touchpad; and
determine the 6DoF position of the hand-held controller based on a corrected pose of the electronic device that accounts for the offset of the contact position.

8. The system of claim 7 , wherein the electronic device comprises a projector to project the modulated light pattern into the local environment.

9. The system of claim 7 , wherein the hand-held controller includes an inertial measurement unit (IMU) configured to capture 3DoF rotational orientation data.

10. The system of claim 7 , wherein:
the processor is to determine a position of a visible marking on a surface of the controller, and further wherein the processor is to track the 6DoF position of the controller based at least on the pose of the electronic device relative to the visible marking.

11. A non-transitory computer readable medium having stored therein instructions, which when executed by a processor, perform operations comprising:
generating depth data based on images of a reflection of a modulated light pattern captured at an electronic device of a local environment proximate the electronic device, the depth data comprising data corresponding to distances of spatial features of objects in the local environment from the electronic device;
determining a pose of the electronic device based on the depth data;
generating, based on the pose of the electronic device relative to at least one spatial feature associated with a controller, a set of positional data representing three degrees-of-freedom (3DoF) translation of the controller within the local environment;
receiving a set of rotational data representing 3DoF orientation of the controller within the local environment;
tracking a six degrees-of-freedom (6DoF) position of the controller within the local environment based on the set of positional data and the set of rotational data;
receiving a set of touchpad information from the controller, wherein the set of touchpad information includes a contact position of a thumb of a user's hand on a touchpad of the controller; and
determining the 6DoF position of the controller based on a corrected pose of the electronic device relative to the thumb contact position, wherein the corrected pose accounts for an offset of the contact position from a center axis of the touchpad.

12. The non-transitory computer readable medium of claim 11 , the instructions, which when executed by a processor, further perform operations comprising:
identifying a position of the user's hand, wherein the at least one spatial feature includes a thumb position of the user's hand; and
determining the 6DoF position of the controller based on the pose of the electronic device relative to the thumb position.

13. The non-transitory computer readable medium of claim 12 , the instructions, which when executed by a processor, further perform operations comprising:
identifying a grasp pattern of the user's hand, wherein the at least one spatial feature further includes an identified relationship between the thumb position and the grasp pattern; and
determining the 6DoF position of the controller based on the pose of the electronic device relative to the identified relationship between the thumb position and the grasp pattern.

14. The non-transitory computer readable medium of claim 13 , the instructions, which when executed by a processor, further perform operations comprising:
determining, based on the identified relationship between the thumb position and the grasp pattern, an inferred thumb position when a thumb of the user's hand is not visible to the electronic

Assignees

• Google Llc

Inventors

• Hesch Joel
• Chen Shiqi
• Lee Johnny
• Garg Rahul

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/30204

  Marker · CPC title