System for tracking a handheld device in virtual reality

Patent metadata

Field	Value
Publication number	US-10083544-B2
Application number	US-201615193777-A
Country	US
Kind code	B2
Filing date	Jun 27, 2016
Priority date	Jul 7, 2015
Publication date	Sep 25, 2018
Grant date	Sep 25, 2018

Abstract

A system for tracking a first electronic device, such as a handheld smartphone, in a virtual reality environment generated by a second electronic device, such as a head mounted display may include detection, by a camera included in one of the first electronic device or the second electronic device, of at least one visual marker included on the other of the first electronic device or the second electronic device. Features detected within the field of view corresponding to known features of the visual markers may be used to locate and track movement of the first electronic device relative to the second electronic device, so that movement of the second electronic device may be translated into an interaction in a virtual experience generated by the second electronic device.

First claim

What is claimed is:

1. A method, comprising:
operably coupling a first electronic device with a second electronic device so that the first electronic device and the second electronic device are in communication with each other while operating in an ambient environment, the second electronic device being a head mounted display device used by a user and configured to generate and display a virtual environment while operating in the ambient environment, and the first electronic device being a handheld controller used by the user;
tracking movement of the first electronic device relative to the second electronic device in a first mode, including:
detecting, by a sensor of the first electronic device, at least one visual marker on the second electronic device, as the first electronic device moves independent of the second electronic device in the ambient environment;
determining a location of the second electronic device relative to the first electronic device and tracking movement of the first electronic device in the ambient environment based on features detected in the at least one visual marker on the second electronic device, including:
calculating an inverse transform of the location of the second electronic device and determining a location of the first electronic device relative to the second electronic device; and
comparing a current location of the first electronic device to a previous location of the first electronic device to track movement of the first electronic device; and

translating the detected and tracked movement of the first electronic device in the ambient environment into a corresponding action in the virtual environment.

2. The method of claim 1 , wherein the detecting at least one visual marker and the tracking movement of the first electronic device in the ambient environment includes:
collecting optical data using an optical sensor of the first electronic device, the optical data including images of the second electronic device;
scanning the collected optical data and detecting a plurality of predetermined features in the scanned optical data;
comparing the detected plurality of predetermined features to known images of the at least one visual marker on the second electronic device; and
identifying the at least one visual marker in the collected optical data based on the detection of the plurality of predetermined features.

3. The method of claim 2 , wherein the detected plurality of predetermined features includes an irregular pattern of at least one of a plurality of corners or a plurality of edges, and wherein comparing the detected plurality of predetermined features to known images of the at least one visual marker, and identifying the at least one visual marker in the collected optical data includes:
comparing an orientation of the irregular pattern of the detected plurality of predetermined features to an orientation of a corresponding plurality of features in the known image of the at least one visual marker; and
determining a position and an orientation of the second electronic device relative to the first electronic device based on the comparison.

4. The method of claim 3 , wherein collecting optical data using an optical sensor of the first electronic device includes:
capturing images of the second electronic device using an infrared sensor of the first electronic device with illumination provided by an infrared illuminator, the at least one visual marker including an infrared pattern on the first electronic device that is only visible to the infrared sensor while illuminated by the infrared illuminator.

5. The method of claim 2 , wherein the optical sensor is a front facing camera of the handheld controller and the at least one visual marker includes a first two-dimensional barcode on a first surface of the head mounted display device and a second two-dimensional barcode on a second surface of the head mounted display device.

6. The method of claim 1 , further comprising tracking movement of the first electronic device relative to the second electronic device in a second mode, including:
collecting image data using an optical sensor of the second electronic device, the optical data including images of the first electronic device;
scanning the collected optical data and detecting a plurality of predetermined features in the scanned optical data;
comparing the detected plurality of predetermined features to known images of the at least one visual marker on the second electronic device;
identifying the at least one visual marker in the collected optical data;
determining a location of the first electronic device relative to the second electronic device based on the identified at least one visual marker based on the detection of the plurality of predetermined features; and
comparing a current location of the first electronic device to a previous location of the first electronic device to track movement of the first electronic device.

7. The method of claim 6 , wherein the detected plurality of predetermined features includes an irregular pattern of at least one of a plurality of corners or a plurality of edges, and wherein comparing the detected plurality of predetermined features to known images of the at least one visual marker, and identifying the at least one visual marker in the collected optical data includes:
comparing an orientation of the irregular pattern of the detected plurality of predetermined features to an orientation of a corresponding plurality of features in the known image of the at least one visual marker; and
determining a position and an orientation of the second electronic device relative to the first electronic device based on the comparison.

8. The method of claim 7 , wherein collecting optical data using an optical sensor of the second electronic device includes:
capturing images of the first electronic device using an infrared sensor of the second electronic device with illumination provided by an infrared illuminator, the at least one visual marker including an infrared pattern defined on the first electronic device that is visible to the infrared sensor while illuminated by the infrared illuminator.

9. The method of claim 6 , wherein the optical sensor is a camera on the head mounted display device and the visual marker is a two-dimensional barcode rendered on a display screen of the handheld controller or located on an outer surface of the handheld controller.

10. A system, comprising:
a handheld electronic device configured to be operably coupled with a head-mounted electronic device the head-mounted electronic device including a plurality of visual markers on an exterior surface of a housing of the head-mounted electronic device, the handheld electronic device including an optical sensor; and
a processor,
wherein the handheld electronic device is configured to
determine a location of the head mounted electronic device relative to the handheld electronic device based on images captured by the optical sensor of the handheld electronic device, and based on identification of predetermined features in the captured images, the predetermined features corresponding to the plurality of visual markers, and
calculating an inverse transform of the location of the head mounted electronic device relative to the handheld electronic device, to determine a location of the handheld electronic device relative to the head mounted electronic device,
as the handheld electronic device moves independent of the head-mounted electronic device in an ambient environment.

11. The system of claim 10 , wherein the plurality of visual markers includes:
a first visual marker on a first surface of the head mounted electronic device;

Assignees

• Google Inc
• Google Llc

Inventors

• Chen Shiqi
• Xu Zhaoyang
• Faaborg Alexander James

Classifications

• G06T7/74

  involving reference images or patches · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/005 Primary

  Input arrangements through a video camera · CPC title

• G06T2207/30244

  Camera pose · CPC title