Reprojection OLED display for augmented reality experiences

Patent metadata

Field	Value
Publication number	US-9721395-B2
Application number	US-201615221605-A
Country	US
Kind code	B2
Filing date	Jul 28, 2016
Priority date	Jun 28, 2013
Publication date	Aug 1, 2017
Grant date	Aug 1, 2017

Abstract

Methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image by circuitry within the display.

First claim

What is claimed is:

1. A method, comprising:
acquiring a rendered image;
determining a first transformation to be applied to a first portion of the rendered image based on a first predicted pose associated with a mobile device;
determining a second transformation to be applied to a second portion of the rendered image based on a second predicted pose different from the first predicted pose associated with the mobile device;
generating a first portion of an updated image by applying the first transformation to the first portion of the rendered image using the mobile device;
generating a second portion of the updated image by applying the second transformation to the second portion of the rendered image using the mobile device; and
displaying the updated image using the mobile device.

2. The method of claim 1 , wherein:
the generating the first portion of the updated image includes applying a first homographic transformation to the first portion of the rendered image; and
the generating the second portion of the updated image includes applying a second homographic transformation different from the first homographic transformation to the second portion of the rendered image.

3. The method of claim 2 , wherein:
the first homographic transformation comprises an affine transformation.

4. The method of claim 1 , wherein:
the applying the first transformation to the first portion of the rendered image includes applying a first weighted mapping to pixels within the first portion of the rendered image; and
the applying the second transformation to the second portion of the rendered image includes applying a second weighted mapping different from the first weighted mapping to pixels within the second portion of the rendered image.

5. The method of claim 1 , wherein:
the generating the first portion of the updated image includes applying a first pixel offset adjustment to the first portion of the rendered image; and
the generating the second portion of the updated image includes applying a second pixel offset different from the first pixel offset to the second portion of the rendered image.

6. The method of claim 5 , wherein:
the mobile device includes a display; and
the applying the first pixel offset adjustment includes loading a first image associated with the first portion of the rendered image into a pixel array of the display and shifting the first image while the first image is stored within the pixel array using circuitry integrated within the pixel array.

7. The method of claim 6 , wherein:
the shifting the first image while the first image is stored within the pixel array using circuitry integrated within the pixel array includes shifting the first image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array.

8. The method of claim 5 , wherein:
the mobile device includes a display; and
the applying the first pixel offset adjustment includes loading a first image associated with the first portion of the rendered image into a buffer of the display and shifting the first image within the buffer.

9. The method of claim 1 , wherein:
the generating the second portion of the updated image includes generating the second portion of the updated image while the first portion of the updated image is displayed using the mobile device.

10. The method of claim 1 , wherein:
the first predicted pose corresponds with a first point in time and the second predicted pose corresponds with a second point in time subsequent to the first point in time; and
the mobile device comprises a head-mounted display device.

11. An electronic device, comprising:
one or more processors configured to acquire a rendered image, the one or more processors configured to determine a first transformation to be applied to a first portion of the rendered image based on a first predicted pose associated with the electronic device and determine a second transformation to be applied to a second portion of the rendered image based on a second predicted pose different from the first predicted pose associated with the electronic device, the one or more processors configured to generate a first portion of an updated image by applying the first transformation to the first portion of the rendered image and generate a second portion of the updated image by applying the second transformation to the second portion of the rendered image; and
a see-through display configured to display the updated image.

12. The electronic device of claim 11 , wherein:
the first transformation comprises an affine transformation.

13. The electronic device of claim 11 , wherein:
the see-through display configured to generate the first portion of the updated image by applying a first pixel offset adjustment to the first portion of the rendered image and to generate the second portion of the updated image by applying a second pixel offset different from the first pixel offset to the second portion of the rendered image.

14. The electronic device of claim 13 , wherein:
the see-through display configured to apply the first pixel offset adjustment by loading a first image associated with the first portion of the rendered image into a buffer of the see-through display and shifting the first image within the buffer.

15. The electronic device of claim 13 , wherein:
the see-through display configured to apply the first pixel offset adjustment by loading a first image associated with the first portion of the rendered image into a pixel array of the see-through display and shifting the first image while the first image is stored within the pixel array using circuitry integrated within the pixel array.

16. The electronic device of claim 15 , wherein:
the see-through display configured to shift the first image using pixel interconnections between pixels within the pixel array to shift pixel values within the pixel array.

17. The electronic device of claim 11 , wherein:
the one or more processors configured to generate the second portion of the updated image while the first portion of the updated image is displayed using the see-through display.

18. The electronic device of claim 11 , wherein:
the electronic device comprises a head-mounted display device; and
the see-through display comprises an OLED display.

19. One or more storage devices containing processor readable code for programming one or more processors to perform a method for displaying images using an electronic device comprising the steps of:
generating a rendered image that includes one or more virtual objects;
determining a first homographic transformation to be applied to a first portion of the rendered image based on a first predicted pose of the electronic device;
determining a second homographic transformation to be applied to a second portion of the rendered image based on a second predicted pose different from the first predicted pose of the electronic device;
generating a first portion of an updated image by applying the first homographic transformation to the first portion of the rendered image using the electronic device;
generating a second portion of the updated image by applying the second homographic transformation to the second portion of the rendered image using the electronic device; and
displaying the updated image using the electronic device.

20. The one or more storage devices of claim 19 , wherein:
the electronic device comprises a head-mounted display device;
the generating the first portion of the upda

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chan Calvin
• Margolis Jeffrey Neil
• Pearson Andrew
• Shetter Martin
• Michail Ashraf Ayman
• Corlett Barry

Classifications

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• G09G2340/0435

  Change or adaptation of the frame rate of the video stream · CPC title

• G06T3/0006

  Physics · mapped topic

• G09G3/007

  Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title