Velocity and depth aware reprojection

Patent metadata

Field	Value
Publication number	US-10114454-B2
Application number	US-201715407833-A
Country	US
Kind code	B2
Filing date	Jan 17, 2017
Priority date	Jun 22, 2016
Publication date	Oct 30, 2018
Grant date	Oct 30, 2018

Abstract

In various embodiments, methods and systems for reprojecting images based on a velocity depth late stage reprojection process are provided. A reprojection engine supports reprojecting images based on an optimized late stage reprojection process that is performed based on both depth data and velocity data. Image data and corresponding depth and velocity data of the image data is received. A determination of an adjustment to be made to the image is made. The determination is made based on motion data, the depth data and the velocity data. The motion data corresponds to a device associated with displaying the image data. The velocity data supports determining calculated correction distances for portions of the image data. The image data is adjusted based on the determined adjustment. Adjusting the image data is based on integrating depth-data-based translation and velocity-data-based motion correction, into a single pass implementation, to adjust the image data.

First claim

The invention claimed is:

1. A system for reprojection virtual images based on velocity and depth late stage reprojection, the system comprising:
one or more processors and memory configured for providing computer program instructions to the one or more processors;
a reprojection engine configured to:
access a frame buffer having image data and corresponding color data, depth data, and velocity data of the image data, wherein the image data comprises stationary objects and non-stationary objects;
determine an adjustment to be made to the image data, based on motion data, depth data and velocity data, the motion data corresponding to a device associated with displaying the image data, the velocity data supports determining calculated correction distances for non-stationary objects in the image data; and
adjusting the image data based on the adjustment determined, wherein for a selected pixel from a plurality of pixels in the frame buffer:
translating and correcting an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location;
based on translating and correcting the original ray, determining a location at which the reprojected ray intersects a blurred depth map of the depth data; and
sampling a color of a pixel for display based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

2. The system of claim 1 , wherein the reprojection engine is further configured to determine the location at which the reprojected ray intersects the blurred depth map is based on:
determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,
then determining an initial location on the reprojected ray corresponding to the initial depth value;
then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray; and
then determining an updated location on the reprojected ray corresponding to the updated depth value.

3. The system of claim 2 , wherein the reprojection engine is further configured to determine the location at which the reprojected ray intersects the blurred depth map by iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray.

4. The system of claim 2 , wherein the reprojection engine is further configured to cause display via a display associated with the device the image data after translating and correcting the original ray to the reprojected ray.

5. The system of claim 1 , wherein correcting the original ray for the selected pixel is based on calculating a corrected distance that indicates a distance moved over a change in time based on the velocity data.

6. The system of claim 1 , wherein correcting the original ray for the selected pixel is based on:
identifying a leading silhouette for the pixel within the velocity data in the frame buffer;
extending the leading silhouette along a direction of a motion vector associated with the pixel by at least one step;
sampling velocity data for the pixel; and
multiplying the velocity data by an inter-frame time step such that a pre-rendered position is transformed by a calculated corrected distance in an opposite direction.

7. The system of claim 1 , wherein adjusting the image data is based on integrating depth data-based translation and velocity-data-based motion correction into a single pass implementation.

8. A computer-implemented method for reprojecting virtual images based on velocity and depth late stage reprojection, the method comprising:
accessing image data and corresponding depth data and velocity data of the image data;
determining an adjustment to be made to the image data, based on motion data, depth data and velocity data, the motion data corresponding to a device associated with displaying the image data, the velocity data supports determining calculated correction distances for portions of the image data; and
adjusting the image data based on the adjustment determined, wherein adjusting the image data is based on integrating depth data-based translation and velocity-data-based motion correction.

9. The method of claim 8 , wherein adjusting the image data is based on:
translating and correcting an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location;
based on translating and correcting the original ray, determining a location at which the reprojected ray intersects a blurred depth map of the depth data; and
sampling a color of a pixel for display based upon a color in a frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

10. The method of claim 9 , wherein determining the location at which the reprojected ray intersects the blurred depth map comprises:
determining an initial depth value on the blurred depth map corresponding to the original frame buffer location;
then determining an initial location on the reprojected ray corresponding to the initial depth value;
then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray; and
then determining an updated location on the reprojected ray corresponding to the updated depth value.

11. The method of claim 10 , wherein determining the location at which the reprojected ray intersects the blurred depth map comprises iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray.

12. The method of claim 9 , wherein correcting the original ray for the selected pixel is based on calculating a corrected distance that indicates a distance moved over a change in time based on the velocity data.

13. The method of claim 9 , wherein correcting the original ray for the selected pixel is based on:
identifying a leading silhouette for the pixel within the velocity data in the frame buffer;
extending the leading silhouette along a direction of a motion vector associated with the pixel by at least one step;
sampling velocity data for a pixel; and
multiplying the velocity data by an inter-frame time step such that a pre-rendered position is transformed by the calculated corrected distance in an opposite direction.

14. The method of claim 8 , further comprising displaying via a display associated with the device the image data after adjusting the image data.

15. One or more computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, causes the one or more processors to perform a method for reprojecting images based on velocity and depth late stage reprojection, the method comprising:
referencing a frame buffer having image data and corresponding color data, depth data, and velocity data of the image data, wherein the image data comprises stationary objects and non-stationary objects;
determining an adjustment to be made to the image data, based on motion data, depth data and velocity data, the motion data corresponding to a device associated with displaying the image data, the velocity data supports determining calculated correction distances for non-stationary objects in the image data; and
adjusting the image data based on the adjustment determined, wherein adjusting the ima

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Boulton Michael George
• Michail Ashraf Ayman
• Schneider Gerhard Albert
• You Yang

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• H04N13/117 Primary

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N13/279

  the virtual viewpoint locations being selected by the viewers or determined by tracking · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title