Late stage reprojection

Patent metadata

Field	Value
Publication number	US-9514571-B2
Application number	US-201313951351-A
Country	US
Kind code	B2
Filing date	Jul 25, 2013
Priority date	Jul 25, 2013
Publication date	Dec 6, 2016
Grant date	Dec 6, 2016

Abstract

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.

First claim

What is claimed is:

1. A mobile device, comprising:
one or more processors configured to generate a rendered image associated with a first predicted pose of the mobile device and determine a second predicted pose of the mobile device, the second predicted pose is different from the first predicted pose, the second predicted pose corresponds with a point in time during which a second updated image is displayed, the one or more processors configured to determine a pose difference between the first predicted pose and the second predicted pose and generate the second updated image based on the pose difference and at least a portion of the rendered image; and
a display in communication with the one or more processors, the display includes a pixel array, the display configured to store the second updated image within the pixel array and generate an updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array, the display configured to display the updated image.

2. The mobile device of claim 1 , wherein:
the pixel array includes a multiplexor configured to set a first pixel of the pixel array, the multiplexor configured to select one of a second voltage corresponding with a second pixel of the pixel array or a third voltage corresponding with a third pixel of the pixel array.

3. The mobile device of claim 1 , wherein:
the one or more processors configured to identify a virtual object being focused on by an end user of the mobile device and determine a stabilization plane based on a location of the virtual object within an augmented reality environment, the one or more processors configured to generate the rendered image based on the stabilization plane.

4. The mobile device of claim 1 , wherein:
the one or more processors configured to generate the second updated image by applying a homographic transformation to the at least a portion of the rendered image.

5. The mobile device of claim 4 , wherein:
the homographic transformation comprises a multi-plane homography.

6. The mobile device of claim 4 , wherein:
the homographic transformation comprises an affine homography.

7. The mobile device of claim 1 , wherein:
the mobile device comprises an HMD; and
the display configured to generate the updated image based on an acceleration of the mobile device.

8. A method for displaying images, comprising:
generating a rendered image corresponding with a first predicted pose of a mobile device;
determining a second predicted pose of the mobile device, the second predicted pose is different from the first predicted pose and corresponds with a point in time during which at least a portion of an updated image is displayed using a display of the mobile device, the display includes a pixel array;
determining a pose difference between the first predicted pose and the second predicted pose;
generating the updated image based on the pose difference and at least a portion of the rendered image, the generating the updated image includes storing the rendered image within the pixel array and generating the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array; and
displaying the at least a portion of the updated image using the display of the mobile device.

9. The method of claim 8 , wherein:
the second predicted pose corresponds with a middle display time for displaying the at least a portion of the updated image using the display; and
and
the pixel array includes a multiplexor configured to set a first pixel of the pixel array, the multiplexor configured to select one of a second voltage corresponding with a second pixel of the pixel array or a third voltage corresponding with a third pixel of the pixel array.

10. The method of claim 8 , wherein:
the generating a rendered image includes identifying a virtual object being focused on by an end user of the mobile device, determining a stabilization plane based on a location of the virtual object within an augmented reality environment, and generating the rendered image based on the stabilization plane.

11. The method of claim 8 , wherein:
the at least a portion of the updated image corresponds with one or more pixel rows within the updated image.

12. The method of claim 8 , wherein:
the generating the updated image includes shifting the pixel values within the pixel array using the pixel interconnections between the pixels of the pixel array and a plurality of multiplexors embedded within the pixels of the pixel array.

13. The method of claim 8 , wherein:
the mobile device comprises an HMD.

14. One or more storage devices containing processor readable code for programming one or more processors to perform a method for displaying images associated with one or more virtual objects using an HMD comprising the steps of:
determining a first predicted pose associated with the HMD;
generating a rendered image based on the first predicted pose;
determining a second predicted pose associated with the HMD subsequent to the determining a first predicted pose, the second predicted pose corresponds with a middle display time for displaying an updated image using a display of the HMD, the display includes a pixel array;
determining a pose difference between the first predicted pose and the second predicted pose;
generating the updated image based on the pose difference and at least a portion of the rendered image, the generating the updated image includes storing the rendered image within the pixel array and generating the updated image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array; and
displaying the updated image using the display of the HMD.

15. The one or more storage devices of claim 14 , wherein:
the generating a rendered image includes identifying a virtual object being focused on by an end user of the HMD, determining a stabilization plane based on a distance of the virtual object from the HMD, and generating the rendered image based on the stabilization plane.

16. The one or more storage devices of claim 14 , wherein:
the pixel array includes a multiplexor configured to set a first pixel of the pixel array, the multiplexor configured to select one of a second voltage corresponding with a second pixel of the pixel array or a third voltage corresponding with a third pixel of the pixel array.

17. The one or more storage devices of claim 14 , wherein:
the pixel array includes a plurality of multiplexors configured to set a plurality of pixel values within the pixel array.

18. The one or more storage devices of claim 14 , wherein:
the generating the updated image includes applying a pixel offset adjustment to the at least a portion of the rendered image using the pixel interconnections between pixels within the pixel array.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Williams Oliver Michael Christian
• Barham Paul
• Isard Michael
• Wong Tuan
• Woo Kevin
• Klein Georg
• Service Douglas Kevin
• Michail Ashraf Ayman
• Pearson Andrew
• Shetter Martin
• Margolis Jeffrey Neil
• Ackerman Nathan
• Chan Calvin
• Tomlin Arthur C

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T15/205

  Image-based rendering · CPC title

• G02B27/017

  Head mounted · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B27/0172

  characterised by optical features · CPC title