Reprojection for high field rate displays

Patent metadata

Field	Value
Publication number	US-2023377241-A1
Application number	US-202217750266-A
Country	US
Kind code	A1
Filing date	May 20, 2022
Priority date	May 20, 2022
Publication date	Nov 23, 2023
Grant date	—

Abstract

In various examples there is a method performed by a Head Mounted Display, HMD, comprising a high field rate display configured to display fields of rendered frames at a field rate. The method comprises receiving a stream of the rendered frames for display on the high field rate display, the stream of rendered frames having a frame rate. The process applies an early stage reprojection to the rendered frames of the stream of rendered frames at a rate which is lower than the field rate. The process applies a late stage reprojection to fields of the rendered frames at the field rate, wherein the early stage reprojection uses more computational resources than the late reprojection.

First claim

What is claimed is:

1 . A method performed by a Head Mounted Display (HMD) comprising a high field rate display configured to display fields of rendered frames at a field rate, the method comprising:
receiving a stream of the rendered frames for display on the high field rate display, the stream of rendered frames having a frame rate; applying an early stage reprojection to the rendered frames of the stream of rendered frames at a rate which is lower than the field rate; applying a late stage reprojection to fields of the rendered frames at the field rate, wherein the early stage reprojection uses more computational resources than the late stage reprojection.

2 . The method as claimed in claim 1 , wherein the rendered frames are red green blue depth RGBD frames and wherein a depth channel of the RGBD frames is used by the early stage reprojection.

3 . The method as claimed in claim 1 , wherein the early stage reprojection is applied at the frame rate.

4 . The method as claimed in claim 1 , wherein the early stage reprojection comprises, for each frame, applying a transformation to the frame according to a current pose of the HMD, and the late stage reprojection comprises, for each field, applying a transformation to the field according to a current pose of the HMD; and wherein the current pose of the HMD is obtained using information from sensors in the HMD.

5 . The method as claimed in claim 1 further comprising:
rendered frame;
Receiving a pose of the HMD which was used to render the received Using the received pose of the HMD and a predicted pose of the HMD as input to the early stage reprojection.

6 . The method of claim 5 wherein the predicted pose of the HMD is computed using data from an inertial measurement unit, IMU.

7 . The method as claimed in claim 1 , wherein the applying of the late stage reprojection comprises:
Carrying out the late stage reprojection in series for each field of a frame and using, for each field, a pose of the HMD received just in time for the late stage reprojection.

8 . The method of claim 7 wherein the late stage reprojection is carried out at a specified time per field and wherein the late stage reprojection uses a predicted pose of the HMD for each of the fields, the predicted pose computed by the HMD.

9 . The method of claim 8 wherein the predicted pose computed by the HMD is computed using data about a motion path of the HMD.

10 . The method as claimed in claim 1 , wherein the fields of the rendered frames are displayed on the high field rate display following the completion of the late stage reprojection.

11 . The method as claimed in claim 1 , wherein the early stage reprojection comprises at least one of: planar deviation-based image reprojection, per-pixel backwards reprojection, planar deviation-based image reprojection using additional rendered views of the rendered frames, per pixel backwards reprojection using additional rendered view of the rendered frames.

12 . The method as claimed in claim 1 , wherein the late stage reprojection comprises any one or more of: a homography, a planar reprojection, a depth reprojection and a motion reprojection.

13 . The method of claim 1 , wherein the stream of rendered frames is received from a remote entity which has rendered the frames from a complex 3D model.

14 . The method of claim 1 , wherein the steam of rendered frames is received from a companion device connected to the HMD.

15 . A head mounted display (HMD) comprising:
a high field rate display configured to display fields of rendered frames at a field rate; a processing unit configured to:
receive a stream of rendered frames, the stream of rendered frames having a frame rate;
apply an early stage reprojection to individual rendered frames of the stream of rendered frames at a rate which is lower than the field rate;

subsequently apply a late stage reprojection to fields of the rendered frames of the received stream at the field rate, wherein the early stage reprojection uses more computational resources compared to the late stage reprojection.

16 . The HMD of claim 15 wherein the early stage reprojection is applied at the frame rate.

17 . The HMD of claim 15 wherein processing unit is arranged such that the early stage reprojection comprises, for each frame, applying a transformation to the frame according to a current pose of the HMD, and the late stage reprojection comprises, for each field, applying a transformation to the field according to a current pose of the HMD; and wherein the current pose of the HMD is obtained from sensors in the HMD.

18 . The HMD of claim 15 wherein the processing unit is arranged to receive a pose of the HMD which was used to render the received rendered frame, and to use the received pose of the HMD and a current observed pose of the HMD as input to the early stage reprojection.

19 . The HMD of claim 15 wherein the processing unit is arranged to carry out the late stage reprojection in series for each field of a frame and using, for each field, a pose of the HMD received just in time for the late stage reprojection.

20 . A method performed by a Head Mounted Display (HMD) comprising a high field rate display configured to display fields of rendered frames at a field rate, the method comprising:
receiving a stream of the rendered frames for display on the high field rate display, the stream of rendered frames having a frame rate; applying an early stage reprojection to the rendered frames of the stream of rendered frames at a rate which is the frame rate; applying a late stage reprojection to fields of the rendered frames at the field rate, wherein the early stage reprojection is more complex than the late stage reprojection.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Frommhold Dag Birger
• Voss-Wolff Christian
• Michail Ashraf Ayman

Classifications

• G09G2354/00

  Aspects of interface with display user · CPC title

• G09G2310/0235

  Field-sequential colour display · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G09G2340/0435

  Change or adaptation of the frame rate of the video stream · CPC title