Low-power semi-passive relative six-degree-of- freedom tracking

Patent metadata

Field	Value
Publication number	US-11340460-B2
Application number	US-202016949027-A
Country	US
Kind code	B2
Filing date	Oct 9, 2020
Priority date	May 18, 2020
Publication date	May 24, 2022
Grant date	May 24, 2022

Abstract

A system for tracking a position and an orientation of an external device such as a smart watch or a handheld controller relative to a head mounted display (HMD) device is provided. A combination of active markers and passive markers on the external device may be detected by a point tracking camera of the HMD to collect position data for the external device relative to the HMD. Acceleration and/or orientation data of the external device relative to the HMD may be collected from an inertial measurement unit(s) (IMU(s)) of the external device and/or the HMD. The fusion of the position data with the data collected by the IMU(s) may allow for tracking of the external device relative to the HMD.

First claim

What is claimed is:

1. A system, including:
a head mounted display (HMD) configured to display virtual content, the HMD including:
a display device;
a camera;
an illumination device; and
a controller controlling operation of the HMD; and

an external device paired with the HMD, the external device including:
an interface device;
a first active marker and a second active marker detectable by the camera of the HMD;
at least one passive marker detectable by the camera of the HMD; and
a controller controlling operation of the external device,

wherein, during a block of operation time in which the external device is within a field of view of the camera of the HMD:
during at least one period within the block of operation time in which the first active marker is on, the second active marker is off and the illumination device is off;
during at least one period within the block of operation time in which the second active marker is on, the first active marker is off and the illumination device is off; and
during at least one period during the block of operation time in which the illumination device is on, the first active marker is off and the second active marker is off; and

wherein the controller of the HMD is configured to control the display device such that virtual content displayed by the HMD is aligned with the external device in response to detection of at least one of the first active marker, the second active marker, or the at least one passive marker within a field of view of the camera of the HMD.

2. The system of claim 1 , wherein the at least one passive marker includes a retroreflective marker that is detectable by the camera of the HMD in response to illumination of the retroreflective marker by the illumination device of the HMD.

3. The system of claim 2 , wherein
the first active marker is located at a first position on the external device;
the second active marker is located at a second position on the external device; and
the at least one passive marker includes a retroreflective marker at a third position on the external device.

4. The system of claim 3 , wherein
the first active marker includes a light source that selectively emits light detected by the camera of the HMD; and
the second active marker includes a light source that selectively emits light detected by the camera of the HMD.

5. The system of claim 4 , wherein at least one of
an intensity of the light emitted by the light source of the first active marker is different than an intensity of the light emitted by the light source of the second active marker; or
a pattern of the light emitted by the light source of the first active marker is different than a pattern of the light emitted by the light source of the second active marker.

6. The system of claim 3 , wherein, when the external device is within the field of view of the camera of the HMD,
the first active marker and the second active marker are detectable by the camera; and
the retroreflective marker is detectable by the camera in response to illumination of the field of view of the camera by the illumination device.

7. The system of claim 6 , wherein the controller of the external device is configured to control operation of the first active marker and the second active marker, and the controller of the external device is configured to control operation of the illumination device.

8. The system of claim 1 , wherein the controller of the HMD is configured to:
detect position data of the external device based on detection of the first active marker, the second active marker, and the at least one passive marker;
combine the position data with at least one of acceleration data or orientation data received from the external device; and
determine a six-degree-of-freedom (6DOF) position of the external device relative to the HMD based on the combined position data and at least one of acceleration data or orientation data.

9. The system of claim 8 , wherein the controller of the HMD is configured to control operation of the display device to display the virtual content in an augmented reality environment, at a position corresponding to the determined 6DOF position of the external device.

10. The system of claim 1 , wherein the controller of the HMD is configured to control the display device such that the virtual content displayed by the HMD is aligned with an interface surface of the external device, and maintains alignment with a position and an orientation of the interface device of the external device while the external device is detected within the field of view of the camera of the HMD.

11. The system of claim 1 , wherein the interface device includes at least one of an input device or an output device, and wherein the controller of the HMD is configured to control the display device to display virtual content in response to an input detected at the interface device of the external device.

12. A computer-implemented method, comprising:
detecting, by a camera of a head mounted display (HMD), at least one active marker and at least one passive marker, on an external device within a field of view of the camera, the external device being paired with the HMD, including, during a block of operation time in which the external device is within the field of view of the camera of the HMD:
detecting a first active marker of the external device during at least one period in which the first active marker is on, a second active marker of the external device is off and an illumination device of the HMD is off;
detecting the second active marker during at least one period in which the second active marker is on, the first active marker is off and the illumination device is off; and
detecting the at least one passive marker during at least one period in which the illumination device is on, the first active marker is off and the second active marker is off;

detecting, by a processor of the HMD, position data of the external device based on the detection of the at least one active marker and the at least one passive marker;
combining, by the processor, the position data with acceleration data and orientation data received from the external device;
determining, by the processor, a six-degree-of-freedom (6DOF) position of the external device relative to the HMD based on the combined position data, acceleration data and orientation data; and
triggering for display, by a display device of the HMD, virtual content at a position corresponding to the 6DOF position of the external device.

13. The computer-implemented method of claim 12 , wherein
detecting the at least one active marker includes:
detecting the first active marker at a first position on the external device; and
detecting the second active marker at a second position on the external device; and

detecting the at least one passive marker includes:
detecting a retroreflective marker at a third position on the external device.

14. The computer-implemented method of claim 13 , wherein
detecting the first active marker includes detecting light selectively emitted by a light source of the first active marker;
detecting the second active marker includes detecting light selectively emitted by a light source of the second active marker; and
detecting the retroreflective marker includes detecting the retroreflective marker in response to illumination of the field of view of the camera by an illumination device of the HMD.

15. The computer-implemented method of claim 14 , wherein at least one of:
an intensity of the light emitted by the light source of the first active marker is different than an

Assignees

• Google Llc

Inventors

• Muldoon Ian
• Kimber Clayton
• Swerdlow Jeremy
• Ravindran Srivatsan

Classifications

• G06F3/0325 Primary

  using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title