Depth-aware reprojection

Patent metadata

Field	Value
Publication number	US-10129523-B2
Application number	US-201615190085-A
Country	US
Kind code	B2
Filing date	Jun 22, 2016
Priority date	Jun 22, 2016
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

Examples are disclosed that relate to depth-aware late-stage reprojection. One example provides a computing system configured to receive and store image data, receive a depth map for the image data, processing the depth map to obtain a blurred depth map, and based upon motion data, determine a translation to be made to the image data. Further, for each pixel, the computing system is configured to translate an original ray extending from an original virtual camera location to an original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location, determine a location at which the reprojected ray intersects the blurred depth map, and sample a color of a pixel for display based upon a color corresponding to the location at which the reprojected ray intersects the blurred depth map.

First claim

The invention claimed is:

1. A computing system comprising a display, a sensor subsystem, and a reprojection subsystem configured to:
receive image data to be displayed via the display;
receive a depth map for the image data;
store the image data in a frame buffer;
process the depth map to obtain a blurred depth map;
based upon motion data received from the sensor subsystem, determine a translation to be made to the image data for display; and
for each pixel of a plurality of pixels in the frame buffer, based upon the translation determined,
translate an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location,
determine a location at which the reprojected ray intersects the blurred depth map, and
sample a color of a pixel for display based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

2. The computing system of claim 1 , wherein the computing system comprises a head-mounted display device.

3. The computing system of claim 1 , wherein the reprojection subsystem is further configured to display via the display the image data after translating the original ray to the reprojected ray.

4. The computing system of claim 1 , wherein the reprojection subsystem is further configured to make a rotation to the image data based upon motion data received from the sensor subsystem.

5. The computing system of claim 1 , wherein the reprojection subsystem is configured to process the depth map to obtain the blurred depth map by omitting background values and weights beyond threshold values during a depth map blur convolution.

6. The computing system of claim 1 , wherein the reprojection subsystem is further configured to determine the location at which the reprojected ray intersects the blurred depth map by
determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,
then determining an initial location on the reprojected ray corresponding to the initial depth value,
then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray, and
then determining an updated location on the reprojected ray corresponding to the updated depth value.

7. The computing system of claim 6 , wherein the reprojection subsystem is further configured to determine the location at which the reprojected ray intersects the blurred depth map by iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray.

8. The computing system of claim 6 , wherein the reprojection subsystem is further configured to iteratively repeat the determining of the updated depth value and the determining of the updated location on the reprojected ray a fixed number of times for each pixel in the frame buffer.

9. The computing system of claim 1 , wherein the reprojection subsystem is further configured to reduce a resolution of the depth map.

10. On a computing system comprising a display and a sensor subsystem, a method comprising:
receiving image data to be displayed via the display;
receiving a depth map for the image data;
storing the image data in a frame buffer;
processing the depth map to obtain a blurred depth map;
based upon motion data received from the sensor subsystem, determining a translation to be made to the image data prior to display; and
for each pixel of a plurality of pixels in the frame buffer, based upon the translation determined,
translating an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location,
determining a location at which the reprojected ray intersects the blurred depth map, and
sampling a color of a pixel in the frame buffer for display at the reprojected frame buffer location based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

11. The method of claim 10 , further comprising displaying via the display the image data after translating the original ray to the reprojected ray for each pixel in the frame buffer.

12. The method of claim 10 , wherein the plurality of pixels comprises each pixel in the frame buffer.

13. The method of claim 10 , wherein determining the location at which the reprojected ray intersects the blurred depth map comprises
determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,
then determining an initial location on the reprojected ray corresponding to the initial depth value,
then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray, and
then determining an updated location on the reprojected ray corresponding to the updated depth value.

14. The method of claim 13 , wherein determining the location at which the reprojected ray intersects the blurred depth map comprises iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray.

15. The method of claim 14 , further comprising iteratively repeating the determining of the updated depth value and the determining of the updated location on the reprojected ray a same number of times for each pixel in the frame buffer.

16. The method of claim 10 , further comprising reducing a resolution of the depth map.

17. A head-mounted display device comprising a display, a sensor subsystem, and a reprojection subsystem configured to:
receive image data to be displayed via the display;
receive depth data for the image data;
store the image data in a frame buffer;
process the depth map to obtain a blurred depth map;
based upon motion data received from the sensor subsystem, determine a translation to be made to the image data prior to display; and
for each pixel of a plurality of pixels in the frame buffer, based upon the translation determined,
translate an original ray extending from an original virtual camera location to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location to a reprojected frame buffer location,
determine a location at which the reprojected ray intersects the blurred depth map using successive approximation, and
sample a color of a pixel in the frame buffer for display at the reprojected frame buffer location based upon a color in the frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

18. The head-mounted display device of claim 17 , wherein the reprojection subsystem is configured to determine the location at which the reprojected ray intersects the blurred depth map using successive approximation by
determining an initial depth value on the blurred depth map corresponding to the original frame buffer location,
then determining an initial location on the reprojected ray corresponding to the initial depth value,
then determining an updated depth value on the blurred depth map corresponding to the initial location on the reprojected ray, and
then determining an updated location on the reprojected ray corresponding to the updated depth value.

19. The head-mounted display device of claim

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Michail Ashraf Ayman
• Klein Georg
• Pearson Andrew Martin
• Mathe Zsolt
• Grossman Mark S
• Xu Ning

Classifications

• G06T15/06

  Ray-tracing · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• H04N13/324

  Colour aspects · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title