System for tracking a handheld device in an augmented and/or virtual reality environment

Patent metadata

Field	Value
Publication number	US-2016378204-A1
Application number	US-201615187218-A
Country	US
Kind code	A1
Filing date	Jun 20, 2016
Priority date	Jun 24, 2015
Publication date	Dec 29, 2016
Grant date	—

Abstract

A system for tracking a first electronic device, such as a handheld electronic device, in a virtual reality environment generated by a second electronic device, such as a head mounted display may include the fusion of data collected by sensors of the electronic device with data collected by sensors of the head mounted display, together with data collected by a front facing camera of the electronic device related to the front face of the head mounted display.

First claim

What is claimed is:

1 . A method, comprising:
generating and displaying a virtual environment on a display of a first electronic device operating in an ambient environment; tracking movement of a second electronic device in the ambient environment based on position data of the second electronic device relative to the first electronic device, and orientation data collected by sensors of the second electronic device; and translating the tracked movement of the second electronic environment into a corresponding action in the virtual environment generated by the first electronic device.

2 . The method of claim 1 , wherein tracking movement of the second electronic device includes:
collecting position data of the second electronic device relative to the first electronic device based on data collected by a depth camera of the first electronic device; and receiving, at the first electronic device, acceleration data of the second electronic device detected by an accelerometer of the second electronic device and orientation data of the second electronic device detected by a gyroscope of the first electronic device.

3 . The method of claim 2 , wherein tracking movement of the second electronic device also includes:
combining, by the first electronic device, the position data collected by the depth camera of the first electronic device, with the acceleration data and the orientation data received from the second electronic device to determine a current position, acceleration and orientation of the second electronic device; and comparing a current position, acceleration and orientation of the second electronic device to a previous position, acceleration and orientation of the second electronic device to track movement of the second electronic device.

4 . The method of claim 3 , wherein the first electronic device is a head mounted display device, and the second electronic device is a handheld controller operably coupled to the head mounted display device.

5 . The method of claim 2 , wherein collecting position data related to the second electronic device based on data collected by the depth camera includes collecting the position data related to the second electronic device based on responses to infrared signals generated by the depth camera.

6 . The method of claim 1 , wherein tracking movement of the second electronic device includes:
initializing a position between a front face of the first electronic device and a front face of the second electronic device and capturing an initial image of the front face of the first electronic device; capturing a current image of the front face of the first electronic device with a front facing camera of the second electronic device; comparing the current image of the front face of the first electronic device to the initial image of the front face of the first electronic device; and determining a position and an orientation of the second electronic device relative to the first electronic device based on the comparison.

7 . The method of claim 6 , wherein comparing the current image of the front face of the first electronic device to the initial image of the front face of the first electronic device includes:
comparing an initial contour of the first electronic device detected in the initial image to a current contour of the first electronic device detected in the current image; and determining at least one of a change in position or a change in orientation of the second electronic device relative to the first electronic device based on the comparison.

8 . The method of claim 6 , wherein tracking movement of the second electronic device also includes:
combining, by a processor of the first electronic device, the determined position of the second electronic device relative to the first electronic device with acceleration data of the second electronic device from an accelerometer of the second electronic device and orientation data of the second electronic device from a gyroscope of the second electronic device to determine a current position, acceleration and orientation of the second electronic device; and comparing the current position, acceleration and orientation of the second electronic device to a previous position, acceleration and orientation of the second electronic device to track movement of the second electronic device.

9 . The method of claim 1 , wherein tracking movement of the second electronic device includes:
initializing a position between a front face of the first electronic device and a front face of the second electronic device and capturing an initial image of the front face of the second electronic device; capturing a current image of the front face of the second electronic device with a camera of the first electronic device; comparing the current image of the front face of the second electronic device to the initial image of the front face of the second electronic device; and determining a position and an orientation of the second electronic device relative to the first electronic device based on the comparison.

10 . The method of claim 9 , wherein comparing the current image of the front face of the second electronic device to the initial image of the front face of the second electronic device includes:
comparing an initial contour of the second electronic device detected in the initial image to a current contour of the second electronic device detected in the current image; and determining at least one of a change in position or a change in orientation of the second electronic device relative to the first electronic device based on the comparison.

11 . A system, comprising:
a head mounted electronic device, including:
a housing;
a display and lenses included in the housing;
a depth camera on the housing and configured to collect position data related to a handheld electronic device operably coupled to the head mounted electronic device; and
a processor controlling operation of the second electronic device, wherein the head mounted electronic device is configured to receive acceleration data and orientation data related to movement of the handheld electronic device from the handheld electronic device, and to determine a location and movement of the handheld electronic device relative to the head mounted electronic device based on the position data collected by the depth camera, and the acceleration data and the orientation data received from the handheld electronic device.

12 . The system of claim 11 , wherein the processor is configured to compare a current position, acceleration and orientation of the handheld electronic device to a previous position, acceleration and orientation of the handheld electronic device, and to track movement of the handheld electronic device based on the comparison.

13 . The system of claim 11 , wherein the processor is configured to:
compare an initial image of a front face of the head mounted electronic device, captured by the handheld electronic device, to a current image of the front face of the head mounted electronic device, captured by the handheld electronic device, and determine a position and an orientation of the handheld electronic device relative to the head mounted electronic device based on the comparison.

14 . The system of claim 13 , wherein, in comparing the current image of the front face of the head mounted display device to the initial image of the front face of the head mounted electronic device, the processor is configured to:
compare an initial contour of the head mounted electronic device detected in the initial image to a current contour of the head mounted electronic de

Assignees

• Google Inc

Inventors

• Chen Shiqi
• Garg Rahul
• Plagemann Christian
• Kaeser Dominik Philemon
• Georgel Pierre

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G01C3/08

  Use of electric radiation detectors · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title