Deep predictor recurrent neural network for head pose prediction

Patent metadata

Field	Value
Publication number	US-11320899-B2
Application number	US-201917253888-A
Country	US
Kind code	B2
Filing date	Jul 22, 2019
Priority date	Jul 23, 2018
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.

First claim

What is claimed is:

1. A wearable system comprising:
a display configured to be disposed before an eye of a wearer of the wearable system, the display configured to display virtual content to the wearer of the wearable system;
a head pose sensor configured to provide head pose data associated with the display;
non-transitory memory configured to store a head pose prediction recurrent neural network;
a hardware processor in communication with the head pose sensor, the display, and the non-transitory memory, the hardware processor programmed to:
receive the head pose data;
input the head pose data to the head pose prediction recurrent neural network;
execute the head pose prediction recurrent neural network to output a predicted head pose at a time horizon, wherein the time horizon indicates a time associated with rendering latency;
calculate which virtual content will be in a field of view of the wearer based on the predicted head pose; and
cause the display to render the calculated virtual content at the time horizon, wherein rendering of the calculated virtual content is configured to be completed when the display is associated with the predicted head pose.

2. The wearable system of claim 1 , wherein the display is configured to present the virtual content at multiple depth planes.

3. The wearable system of claim 1 , wherein the head pose sensor comprises an inertial measurement unit.

4. The wearable system of claim 1 , wherein the head pose prediction recurrent neural network comprises a recurrent neural network (RNN) and a fully connected (FC) layer.

5. The wearable system of claim 1 , wherein the head pose prediction recurrent neural network comprises a plurality of repeating modules.

6. The wearable system of claim 5 , wherein at least one of the plurality of repeating modules comprises a long short term memory cell (LSTM) or a gated recurrent unit (GRU).

7. The wearable system of claim 1 , wherein the head pose prediction recurrent neural network comprises a stack of one or more long short term memory (LSTM) cells or a stack of one or more gated recurrent units (GRUs).

8. The wearable system of claim 7 , wherein the head pose prediction recurrent neural network comprises three LSTM cells or three GRUs.

9. The wearable system of claim 1 , wherein the predicted head pose comprises a 3 degree-of-freedom (DOF) pose.

10. The wearable system of claim 1 , wherein the predicted head pose comprises head orientation, and the hardware processor is further programmed to predict head position at the time horizon using a predictive, non-neural network model.

11. The wearable system of claim 10 , wherein the predictive, non-neural network model comprises a constant acceleration model.

12. A method for rendering virtual content, the method comprising:
under control of a rendering pipeline comprising computer hardware:
receiving head pose data for a head of a user, the head pose data reflecting a pose associated with a display configured to be disposed before an eye of the user;
predicting, using at least a recurrent neural network (RNN) and the head pose data, a predicted head pose at a prediction horizon, wherein the predicted horizon indicates a time associated with rendering latency;
calculate which virtual content will be in a field of view of the user based on the predicted head pose; and
rendering the calculated virtual content at the prediction horizon, wherein rendering of the calculated virtual content is configured to be completed when the display is associated with the predicted head pose.

13. The method of claim 12 , wherein the head pose data comprises a time series of one or more of: (1) data from an inertial measurement unit (IMU), (2) velocity or acceleration data for the head of the user, or both (1) and (2).

14. The method of claim 13 , further comprising filtering data to provide the velocity or acceleration data.

15. The method of claim 14 , wherein the filtering is performed with an extended Kalman filter (EKF).

16. The method of claim 12 , wherein the RNN comprises a stack of one or more long short term memory (LSTM) cells or a stack of one or more gated recurrent units (GRUs).

17. The method of claim 16 , wherein the RNN comprises a stack of three LSTM cells or three GRUs.

18. The method of claim 12 , wherein the RNN is connected to a fully connected (FC) layer configured to output the predicted head pose.

19. The method of claim 12 , wherein the predicted head pose comprises an orientation of the head of the user, and the method further comprises predicting, at the prediction horizon, a position of the head of the user with a constant acceleration model.

Assignees

• Magic Leap Inc

Inventors

• Barak Lior
• Rosenthal Guy
• Perry Adi

Classifications

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• A63F13/5255

  according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character · CPC title

• A63F13/211 Primary

  using inertial sensors, e.g. accelerometers or gyroscopes · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/012 Primary

  Head tracking input arrangements · CPC title