Frame projection for augmented reality environments

Patent metadata

Field	Value
Publication number	US-9978180-B2
Application number	US-201615005702-A
Country	US
Kind code	B2
Filing date	Jan 25, 2016
Priority date	Jan 25, 2016
Publication date	May 22, 2018
Grant date	May 22, 2018

Abstract

Motion vector estimation is provided for generating and displaying images at a frame rate that is greater than a rendering frame rate. The displayed images may include late stage graphical adjustments of pre-rendered scenes that incorporate motion vector estimations. A head-mounted display (HMD) device may determine a predicted pose associated with a future position and orientation of the HMD, render a current frame based on the predicted pose, determine a set of motion vectors based on the current frame and a previous frame, generate an updated image based on the set of motion vectors and the current frame, and display the updated image on the HMD. In one embodiment, the HMD may determine an updated pose associated with the HMD subsequent to or concurrent with generating the current frame, and generate the updated image based on the updated pose and the set of motion vectors.

First claim

What is claimed is:

1. A mobile device, comprising:
a display; and
one or more processors configured to communicate with the display, one or more of the processors being configured to access a first frame whose data has been rendered and whose data has been thereafter pose-wise adjusted using first pose information associated with a first pose for the mobile device and to access a second frame whose data has been rendered and thereafter pose-wise adjusted using different second pose information associated with a second pose for the mobile device, the second pose occurring after the first pose, one or more of the processors being configured to transform the first frame based on a difference between the first pose information and the second pose information such that the transformed first frame appears as if the transformed first frame had been generated to correspond with the later occurring second pose, one or more of the processors being configured to determine a set of motion vectors based on inferred image animation between the transformed first frame and the second frame, and one or more of the processors being configured to generate a first field for display by the display of the mobile device based on the second frame and the determined set of motion vectors.

2. The mobile device of claim 1 , wherein:
each motion vector is associated with a block of one or more pixels of the display;
determining the set of motion vectors includes determining a velocity for each block based on displacement associated with the block and a first time associated with the first frame and a second time associated with the second frame; and
the one or more processors are configured to determine a set of offsets for a set of pixels of the display based on the velocity of the block and a target display time of each pixel.

3. The mobile device of claim 2 , wherein:
the first pose information corresponds to a first predicted pose of a camera portion of the mobile device and the second pose information corresponds to a second predicted pose of the camera portion of the mobile device.

4. The mobile device of claim 3 , wherein:
the target display time is associated with third pose information corresponding to a third predicted pose of the camera portion of the mobile device, the third predicted pose being associated with a point in time between the second predicted pose and the first predicted pose.

5. The mobile device of claim 3 , wherein:
the target display time is associated with third pose information corresponding to a third predicted pose of the camera portion of the mobile device, the third predicted pose being associated with a point in time following the second predicted pose and the first predicted pose.

6. The mobile device of claim 3 , wherein the one or more processors are configured to generate the first field by:
determining third pose information corresponding to a third predicted pose subsequent to accessing the second frame;
generating a first set of uv values for the second frame based on a difference between the second pose information and the third pose information; and
modifying the first set of uv values using the offset for each pixel to generate a second set of uv values associated with the target display time for each pixel.

7. The mobile device of claim 6 , wherein generating the first set of uv values for the second frame comprises applying a pixel offset adjustment to at least a portion of the second frame.

8. A method, comprising:
rendering a first image based on a first predicted pose associated with a mobile device and rendering a second image based on a different second predicted pose associated with the mobile device;
generating a projected image from a selected one of the rendered first and second images, the projected image being a transformation of the selected one of the rendered first and second images where the transformation causes the projected image to appear as if it were captured using the pose of the other of the rendered first and second images;
determining a set of motion of vectors based on a determined difference between the projected image and the other of the rendered first and second images;
generating an updated image for displaying on a display of the mobile device based on the determined set of motion vectors and at least one of the projected image and the other of the rendered first and second images; and
displaying the updated image on the display.

9. The method of claim 8 , wherein the generating of the projected image comprises:
determining a pose difference between the first predicted pose and the second predicted pose; and
applying a homographic transformation to at least a portion of the selected one of the rendered first and second image based on the determined pose difference.

10. The method of claim 8 , wherein:
generating the updated image comprises modifying a set of pixel values associated with the at least one of the projected image and the other of the rendered first and second images based on the determined set of motion vectors.

11. The method of claim 10 , wherein:
the first image is part of a first rendered frame and the second image is part of a second rendered frame;
the modifying of the set of pixel values comprises modifying a set of uv values; and
the generating of the updated image comprises generating a first field including the updated image.

12. The method of claim 11 , further comprising:
determining an area of focus or tracking based on eye tracking information associated with a later timed one of the first and second images;
wherein modifying the set of uv values comprises modifying a subset of the set of uv values that are associated with the area of focus.

13. The method of claim 11 , wherein modifying the set of pixel values comprises:
determining at least one target display time for the first field; and
determining a set of offsets based on the set of motion vectors and the at least one target display time.

14. The method of claim 13 , wherein determining the set of offsets comprises:
determining a velocity of each motion vector based on a time difference between when the first rendered frame and the second rendered frame are to be displayed.

15. The method of claim 14 , wherein:
the display is a scanning display;
determining the at least one target display time comprises determining a plurality of target display times associated with the first field, the plurality of target display times include a first target display time associated with a first pixel in a first row and a second target display time associated with a second pixel in a second row, the second target display time is after the first target display time; and
determining the set of offsets includes determining a first offset for the first pixel based on the first target display time and determining a second offset for the second pixel based on the second target display time.

16. The method of claim 13 , wherein:
the display is a color sequential display configured to sequentially display the first field, a second field, and a third field for the second rendered frame;
generating the first field for the display includes projecting the set of motion vectors for a first target display time associated with the first field; and
the method further comprises generating the second field by projecting the set of motion vectors for a second target display time associated with the second field, and generating the third field by projecting the set of motion vectors for a third target display time associated with the third field.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Margolis Jeffrey Neil
• Crisler Matthew

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N19/139

  Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B27/017

  Head mounted · CPC title