Passive optical and inertial tracking in slim form-factor

Patent metadata

Field	Value
Publication number	US-10146334-B2
Application number	US-201615274334-A
Country	US
Kind code	B2
Filing date	Sep 23, 2016
Priority date	Jun 9, 2016
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

Apparatus and systems directed to a wireless hand-held inertial controller with passive optical and inertial tracking in a slim form-factor, for use with a head mounted virtual or augmented reality display device (HMD), that operates with six degrees of freedom by fusing (i) data related to the position of the controller derived from a forward-facing optical sensor located in the HMD with (ii) data relating to the orientation of the controller derived from an inertial measurement unit located in the controller.

First claim

What is claimed is:

1. In a virtual or augmented reality computing environment that includes a head mounted display (HMD) device and a main processor, a wireless hand-held controller for providing one or more user inputs to the main processor, wherein the main processor determines an orientation of the wireless hand-held controller based on detected position data and orientation data, the hand-held controller comprising:
a stylus having a first end, a second end and an elongate middle portion between the first end and the second end, the stylus also including a first optically reflective marker at or proximate the first end and a second optically reflective marker at or proximate the second end, the first and second optically reflective markers being adapted to reflect light, comprising at least IR (infrared) light emitted from the HMD device, toward an optical sensor for the HMD device which is used to generate position data; and
the elongate middle portion enclosing a microcontroller, an inertial measurement unit (IMU), a communications radio and one or more buttons configured and positioned for selective actuation with a finger or thumb of a user, the inertial measurement unit tracking orientation of the hand-held controller in three dimensional space relative to a predetermined frame of reference and providing orientation data and acceleration data to the microcontroller, and the communications radio providing wireless communications between the microcontroller and the main processor so as to provide the orientation data and user inputs derived from selective activation by the user of the one or more buttons to the main processor.

2. The hand-held controller of claim 1 , wherein the first optically reflective marker and the second optically reflective marker each comprise an infrared marker.

3. The hand-held controller of claim 2 further comprising a material covering the first and second optically reflective markers that is transmissive to light in the IR spectrum but opaque to light in the visible spectrum.

4. The hand-held controller of claim 3 , wherein the hand-held controller is shaped in form of an object used in one or more video games.

5. The hand-held controller of claim 4 , wherein the IMU comprises one or more of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer.

6. The hand-held controller of claim 5 , wherein the orientation data comprises Θ pitch , Θ yaw and Θ roll coordinates.

7. A system comprising:
a head mounted display (HMD) device having a processor;
a wireless hand-held inertial controller configured to communicate with the processor to selectively provide one or more user inputs, the hand-held inertial controller comprising:
a stylus having a first end, a second end and an elongate middle portion between the first end and the second end, the stylus also including a first optically reflective marker at or proximate the first end and a second optically reflective marker at or proximate the second end, the first and second optically reflective markers being adapted to reflect lights, comprising at least IR (infrared) light emitted from the HMD device, toward an optical sensor located on the HMD device; and
the elongate middle portion enclosing a microcontroller, an inertial measurement unit (IMU), a communications radio and one or more buttons configured and positioned for selective actuation with a finger or thumb of a user, the inertial measurement unit tracking orientation of the hand-held inertial controller in three dimensional space relative to a predetermined frame of reference and providing orientation and acceleration data to the microcontroller, and the communications radio providing wireless communications so as to provide orientation data and user inputs to the processor; and

an optical sensor located on the HMD device for determining the position of each of the first and second optically reflective markers relative to the HMD device by detecting light reflected by at least one of the first and second optically reflective markers, and for providing position data to the processor, and
wherein the processor uses the orientation data and the position data to track the hand-held inertial controller within three dimensional space with six degrees of freedom.

8. The system of claim 7 , wherein the HMD device comprises a virtual reality display.

9. The system of claim 7 , wherein the HMD device comprises a three dimensional, augmented reality display.

10. The system of claim 9 , wherein the first optically reflective marker and the second optically reflective marker each comprise an infrared marker.

11. The system of claim 10 further comprising a material covering the first and second optically reflective markers that is transmissive to light in the IR spectrum but opaque to light in the visible spectrum.

12. The system of claim 10 , wherein the hand-held controller is shaped in form of an object used in one or more video games.

13. The system of claim 9 , wherein the IMU comprises one or more of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer.

14. The system of claim 13 , wherein the orientation data comprises θ pitch , θ yaw and θ roll coordinates.

15. The system of claim 9 , wherein the optical sensor comprises a depth camera.

16. The system of claim 15 , wherein the depth camera is an infrared depth camera.

17. The system of claim 16 , wherein the depth camera is mounted on the HMD device in a forward orientation relative to the HMD device.

18. The system of claim 9 , wherein the position data comprises x, y and z coordinates.

19. The system of claim 9 , wherein the orientation data and the position data of the hand-held inertial controller is transformed by the processor into x, y, z, θ pitch , Θ yaw and θ roll coordinates within a real world frame of reference.

20. The system of claim 9 , wherein the communications radio is a Bluetooth radio and the wireless communications are Bluetooth transmissions.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Balan Alexandru Octavian
• Dulu Constantin
• Edmonds Christopher Douglas
• Finocchio Mark James

Classifications

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• H04N13/257

  Colour aspects · CPC title

• H04N13/383

  for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• A63F13/24

  Constructional details thereof, e.g. game controllers with detachable joystick handles · CPC title