Dead reckoning positional prediction for augmented reality and virtual reality applications

Patent metadata

Field	Value
Publication number	US-2020082548-A1
Application number	US-201816123394-A
Country	US
Kind code	A1
Filing date	Sep 6, 2018
Priority date	Sep 6, 2018
Publication date	Mar 12, 2020
Grant date	—

Abstract

Techniques for predicting a virtual camera view in an augmented reality (AR) or virtual reality (VR) application. A first change in position of a user device over a first time period is determined based on analyzing a plurality of frames of image data related to an AR or VR application. A dead reckoning calculation is used to predict a second change in position of the user device over a second time period, based on the first change in position and data received from an Inertial Measurement Unit (IMU) associated with the user device. A plurality of frames of image data are generated for display in the AR or VR application, based on the predicted second change in position of the user device.

First claim

What is claimed is:

1 . A method of predicting a virtual camera view in an augmented reality (AR) or virtual reality (VR) application, comprising:
determining a first change in position of a user device over a first time period based on analyzing a plurality of frames of image data related to an AR or VR application; predicting a second change in position of the user device over a second time period, using a dead reckoning calculation and based on the first change in position and data received from an Inertial Measurement Unit (IMU) associated with the user device; and generating a plurality of frames of image data for display in the AR or VR application, based on the predicted second change in position of the user device.

2 . The method of claim 1 , wherein generating the plurality of frames of image data for display in the AR or VR application further comprises:
determining a virtual camera view associated with the AR or VR application, wherein the virtual camera view includes a virtual object and a physical object, wherein a relationship between the virtual object and the physical object in the virtual camera view is based on the predicted second change in position of the user device, and wherein at least one of the plurality of frames of image data is based on the determined virtual camera view.

3 . The method of claim 1 , wherein determining the first change in position of the user device further comprises:
identifying a first position of the user device at a first point in time based on analyzing a first image captured using the user device; identifying a second position of the user device at a second point in time based on analyzing a second image captured using the user device, wherein the second point of time occurs after the first point in time ; and determining the first change in position based on the identified first position of the user device and the identified second position of the user device.

4 . The method of claim 3 , wherein identifying the first position of the user device comprises identifying a first location of a reference object in the captured first image and wherein identifying the second position of the user device comprises identifying a second location of the reference object in the captured second image.

5 . The method of claim 1 , further comprising:
determining a first change in attitude of the user device over the first time period based on analyzing data received from the IMU associated with the user device, wherein the data relates to the first time period, and wherein predicting the second change in position of the user device is further based on the determined first change in attitude of the user device.

6 . The method of claim 1 , wherein generating the plurality of frames of image data for display in the AR or VR application further comprises:
identifying a display frame rate associated with the AR or VR application; determining a first number of frames for generation, based on the identified display frame rate and the second time period; and generating the first number of frames of image data.

7 . The method of claim 1 , further comprising:
determining a yaw correction value associated with the user device during the first time period based on analyzing data from a magnetometer associated with the user device, wherein the data relates to the first time period, and wherein predicting the second change in position of the user device is further based on the determined yaw correction value.

8 . The method of claim 1 , wherein the user device comprises a head mounted display.

9 . The method of claim 7 , wherein the application is an AR application.

10 . A computer program product for predicting a virtual camera view in an AR or VR application, the computer program product comprising:
a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising:
determining a first change in position of a user device over a first time period based on analyzing a plurality of frames of image data related to an AR or VR application;
predicting a second change in position of the user device over a second time period, using a dead reckoning calculation and based on the first change in position and data received from an Inertial Measurement Unit (IMU) associated with the user device; and
generating a plurality of frames of image data for display in the AR or VR application, based on the predicted second change in position of the user device.

11 . The computer program product of claim 10 , wherein generating the plurality of frames of image data for display in the AR or VR application further comprises:
determining a virtual camera view associated with the AR or VR application, wherein the virtual camera view includes a virtual object and a physical object, wherein a relationship between the virtual object and the physical object in the virtual camera view is based on the predicted second change in position of the user device, and wherein at least one of the plurality of frames of image data is based on the determined virtual camera view.

12 . The computer program product of claim 10 , wherein determining the first change in position of the user device further comprises:
identifying a first position of the user device at a first point in time based on analyzing a first image captured using the user device; identifying a second position of the user device at a second point in time based on analyzing a second image captured using the user device, wherein the second point of time occurs after the first point in time ; and determining the first change in position based on the identified first position of the user device and the identified second position of the user device.

13 . The computer program product of claim 12 , wherein identifying the first position of the user device comprises identifying a first location of a reference object in the captured first image and wherein identifying the second position of the user device comprises identifying a second location of the reference object in the captured second image.

14 . The computer program product of claim 10 , the operation further comprising:
determining a first change in attitude of the user device over the first time period based on analyzing data received from the IMU associated with the user device, wherein the data relates to the first time period, and wherein predicting the second change in position of the user device is further based on the determined first change in attitude of the user device.

15 . The computer program product of claim 10 , wherein generating the plurality of frames of image data for display in the AR or VR application further comprises:
identifying a display frame rate associated with the AR or VR application; determining a first number of frames for generation, based on the identified display frame rate and the second time period; and generating the first number of frames of image data.

16 . A system, comprising:
a processor; and a memory storing a program, which, when executed on the processor, performs an operation, the operation comprising:
determining a first change in position of a user device over a first time period based on analyzing a plurality of frames of image data related to an AR or VR application;
predicting a second change in position of the user device over a second time period, using a dead reckoning calculation and based on the first change in position and da

Assignees

• Disney Entpr Inc

Inventors

• Graham Todd Michael
• Nelson Amy E

Classifications

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title