System and method for aligning virtual objects on peripheral devices in low-cost augmented reality/virtual reality slip-in systems

Patent metadata

Field	Value
Publication number	US-11100713-B2
Application number	US-201816104452-A
Country	US
Kind code	B2
Filing date	Aug 17, 2018
Priority date	Aug 17, 2018
Publication date	Aug 24, 2021
Grant date	Aug 24, 2021

Abstract

Techniques for aligning a virtual object with a physical object in an Augmented Reality (AR) or Virtual Reality (VR) application are described. An electronic peripheral includes a first inertial measurement unit (“IMU”). A head mounted display includes a second IMU. An estimated attitude for the electronic peripheral is generated using data from the first IMU. An estimated attitude for the head mounted display is generated using data from the second IMU. An orientation of a virtual object is determined based on the estimated first and second attitudes, such that the virtual object is aligned with an object in a user's physical environment when the virtual object is displayed to the user. The virtual object is displayed on the head mounted display.

First claim

What is claimed is:

1. A system, comprising:
an electronic peripheral for an Augmented Reality (AR) or Virtual Reality (VR) application, the electronic peripheral comprising a first inertial measurement unit (“IMU”), the first IMU comprising:
a first magnetometer;
a first gyroscope; and
a first accelerometer;

a head mounted display, comprising a second IMU, the second IMU comprising:
a second magnetometer;
a second gyroscope; and
a second accelerometer;

a mobile device, comprising a third IMU, the third IMU comprising:
a third magnetometer
a third gyroscope; and
a third accelerometer;

a processor; and
a memory storing a program, which, when executed on the processor, performs an operation, the operation comprising:
receiving a first estimated attitude for the electronic peripheral, the first estimated attitude generated using data from the first IMU;
determining a second estimated attitude for the head mounted display, comprising:
determining an initial second estimated attitude for the head mounted display, at the mobile device, using data from the third gyroscope, and the third accelerometer, in the mobile device; and
after determining the initial second estimated attitude, modifying the determined initial second estimated attitude using data from the second magnetometer in the head mounted display and not from the third magnetometer; and
predicting a future attitude for the head mounted display, for a future time, based on the modified second estimated attitude for the head mounted display determined using data from the second magnetometer and not from the third magnetometer;

determining a first orientation of a virtual object for display on the head mounted display based on the estimated first and second attitudes, such that the virtual object is aligned with the electronic peripheral when the virtual object is displayed to a user; and
displaying the virtual object on the head mounted display using the determined first orientation;
correcting the predicted future attitude for the head mounted display based on measured data from the second IMU;
determining a second orientation of the virtual object for display on the head mounted display based on the corrected predicted future attitude for the head mounted display; and
displaying the virtual object on the head mounted display using the determined second orientation.

2. The system of claim 1 , wherein the first estimated attitude for the electronic peripheral is generated by the electronic peripheral.

3. The system of claim 1 , wherein the mobile device comprises the processor and the memory storing the program.

4. The system of claim 3 , wherein the second estimated attitude is determined by the head mounted display.

5. A computer implemented method of aligning a virtual object with a physical object in an AR or VR application, comprising:
receiving a first estimated attitude for an electronic peripheral for an AR or VR application, the electronic peripheral comprising a first IMU, the first IMU comprising a first magnetometer, a first gyroscope, and a first accelerometer, wherein the first estimated attitude is generated using data from the first IMU;
determining a second estimated attitude for a head mounted display, the head mounted display comprising a second IMU, the second IMU comprising a second magnetometer, a second gyroscope, and a second accelerometer, wherein the determining the second estimated attitude comprises:
determining, at a mobile device, an initial second estimated attitude for the head mounted display using data from a third gyroscope, and a third accelerometer, in the mobile device, wherein the mobile device further comprises a third magnetometer;
after determining the initial second estimated attitude, modifying the determined initial second estimated attitude using data from the second magnetometer in the head mounted display and not from the third magnetometer; and
predicting a future attitude for the head mounted display, for a future time, based on the modified second estimated attitude for the head mounted display determined using data from the second magnetometer display and not from the third magnetometer;

determining a first orientation of a virtual object for display on the head mounted display based on the estimated first and second attitudes, such that the virtual object is aligned with the electronic peripheral when the virtual object is displayed to a user;
displaying the virtual object on the head mounted display using the determined first orientation;
correcting the predicted future attitude for the head mounted display based on measured data from the second IMU;
determining a second orientation of the virtual object for display on the head mounted display based on the corrected predicted future attitude for the head mounted display; and
displaying the virtual object on the head mounted display using the determined second orientation.

6. The method of claim 5 , wherein the first estimated attitude for the electronic peripheral is generated by the electronic peripheral.

7. The method of claim 6 , wherein determining the first orientation of the virtual object further comprises comparing the first estimated attitude for the electronic peripheral and the second estimated attitude for the head mounted display with a plurality of fixed vectors, comprising a first vector relating to gravity and a second vector relating to magnetic north.

8. The method of claim 5 , wherein the second estimated attitude is determined by the head mounted display.

9. A computer program product for aligning a virtual object with a physical object in an AR or VR application, the computer program product comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, wherein the code, when executed by a processor, performs an operation, the operation comprising:
receiving a first estimated attitude for an electronic peripheral for an AR or VR application, the electronic peripheral comprising a first IMU, the first IMU comprising a first magnetometer, a first gyroscope, and a first accelerometer, wherein the first estimated attitude is generated using data from the first IMU;
determining a second estimated attitude for a head mounted display, the head mounted display comprising a second IMU, the second IMU comprising a second magnetometer, a second gyroscope, and a second accelerometer, wherein the determining the second estimated attitude comprises:
determining, at a mobile device, an initial second estimated attitude for the head mounted display using data from a third gyroscope, and a third accelerometer, in the mobile device, wherein the mobile device further comprises a third magnetometer;
after determining the initial second estimated attitude, modifying the determined initial second estimated attitude using data from the second magnetometer in the head mounted display and not from the third magnetometer; and
predicting a future attitude for the head mounted display, for a future time, based on the modified second estimated attitude for the head mounted display determined using data from the second magnetometer and not from the third magnetometer;

determining a first orientation of a virtual object for display on the head mounted display based on the estimated first and second attitudes, such that the virtual object is aligned with the electronic peripheral when the virtual object is displayed to a user;
displaying the virtual object on the head mounted display using the determined first orientation;
correcting the predicted future attitude for the head mounted display based on measured data from the second IMU;
determining a second orientation of the virt

Assignees

• Disney Entpr Inc

Inventors

• Nocon Nathan D
• Nelson Amy E
• Goslin Michael P
• Davis Seth A
• Davis Randall S

Classifications

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title