Discontinuity-aware reprojection

Patent metadata

Field	Value
Publication number	US-10237531-B2
Application number	US-201715407858-A
Country	US
Kind code	B2
Filing date	Jan 17, 2017
Priority date	Jun 22, 2016
Publication date	Mar 19, 2019
Grant date	Mar 19, 2019

Abstract

In various embodiments, methods and systems reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection are provided. A reconstruction point, that indicates camera pose information, is accessed. The reconstruction point is associated with a plurality of sample points of a three-dimensional (3D) virtual scene. One or more closest sample points, relative to the reconstruction point, are identified, from the plurality of sample points. Each of the one or more closest sample points is associated with a cube map of color data and depth data. A relative convergence score is determined for each of the one or more closest sample points based on performing a depth-aware cube map late stage reprojection operation in relation to the reconstruction point. A subset of the one or more closest sample points is identified based on the relative convergence score. A reconstructed 3D virtual image is generated using the subset.

First claim

The invention claimed is:

1. A system for reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection, the system comprising:
one or more processors and memory configured for providing computer program instructions to the one or more processors;
a reprojection engine configured to:
access a reconstruction point that indicates camera pose information associated with a 3D virtual scene, the 3D virtual scene is associated with a plurality of sample points each having a cube map of color data and depth data;
identify one or more closest sample points relative to the reconstruction point, the one or more closest sample points is identified from the plurality of sample points;
determine a relative convergence score generated for each of the one or more closest sample points based on performing a depth-aware cube map late stage projection operation for each of the one or more closest sample points in relation to the reconstruction point, wherein the relative convergence score is an indicator of a capacity of a sample point to reconstruct at least portions of the 3D virtual scene from the reconstruction point;
identifying a subset of the one or more closest sample points based on eliminating one or more of the one or more closest sample points based on the relative convergence score; and
generating a reconstructed 3D virtual scene using the subset of the one or more closest sample points.

2. The system of claim 1 , wherein the reprojection engine is further configured to perform the depth-aware cube map late stage projection operation based on:
adjusting an original ray extending from an original virtual camera location associated with a selected sample point to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location associated with the reconstruction point to a reprojected frame buffer location; and
determining a location at which the reprojected ray intersects the depth data comprising a blurred depth map.

3. The system of claim 2 , wherein the reprojection engine is further configured to determine the location at which the reprojected ray intersects the blurred depth map by iteratively repeating determining of updated depth values and determining of updated locations on the reprojected ray; and
sample a color of a pixel for display based upon a color in a frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

4. The system of claim 2 , wherein the reprojection engine is further configured to generate the original frame buffer and the reprojected frame buffer based on the cube map color data and depth data.

5. The system of claim 1 , wherein the reprojection engine is further configured to generate the reconstructed 3D virtual scene using a first stereoscopic view and a second stereoscopic view of a stereoscopic projection of the 3D virtual scene from the reconstruction point, using the subset of the plurality of sample points.

6. The system of claim 1 , wherein selecting locations for plurality of sample points in the 3D virtual scene is based on a sample point selection scheme that includes selecting more sample points in locations of the 3D virtual scene with more content detail and selecting less sample points in locations of the 3D virtual scene with less content detail.

7. The system of claim 6 , wherein a distance between a selected sample point and a closest sample point is greater than an intra-ocular distance for stereoscopic reprojection of the reconstructed 3D virtual scene.

8. A computer-implemented method for reprojecting three-dimensional (3D) virtual scenes using discontinuity depth late stage reprojection, the method comprising:
accessing a reconstruction point that indicates camera pose information associated with a 3D virtual scene, the 3D virtual scene is associated with a plurality of sample points each having a cube map of color data and depth data;
identifying a first selected sample point from the plurality of sample points, wherein the first selected sample point is a closest sample point relative to the reconstruction point;
determining a dis-occlusion associated with generating a reconstructed 3D virtual scene from the reconstruction point, wherein the dis-occlusion is determined based on performing a depth-aware cube map late stage projection operation for first selected sample point in relation to reference point;
identifying a second selected sample point, from the plurality of sample points, for resolving the dis-occlusion, wherein the second selected sample point is a second closest sample point relative to the reconstruction point; and
generating a corresponding first portion and second portion of the reconstructed 3D virtual scene using the first selected sample point and the second selected sample point based on performing the depth-aware cube map late stage projection operation on the first selected sample point and the second selected sample point in relation to the reference point.

9. The method of claim 8 , wherein performing a depth-aware cube map late stage projection operation on a selected sample point is based on:
adjusting an original ray extending from an original virtual camera location associated with a selected sample point to a corresponding original frame buffer location to a reprojected ray extending from a translated camera location associated with the reconstruction point to a reprojected frame buffer location; and
determining a location at which the reprojected ray intersects the depth data comprising a blurred depth map.

10. The method of claim 9 , wherein determining the location at which the reprojected ray intersects the blurred depth map is based on iteratively repeating determining of updated depth values and determining of updated locations on the reprojected ray.

11. The method of claim 10 , further comprising sampling a color of a pixel for display based upon a color in a frame buffer corresponding to the location at which the reprojected ray intersects the blurred depth map.

12. The method of claim 8 , wherein the reconstruction point is associated with a first stereoscopic view and a second stereoscopic view such that generating the reconstructed 3D virtual scene further comprises generating the first stereoscopic view and the second stereoscopic view.

13. The method of claim 12 , wherein the first stereoscopic view is associated with a first original frame buffer and a first reprojected frame buffer based on a cube map color data and depth data of a selected sample point; and wherein the second stereoscopic view is associated with a second original frame buffer and a second reprojected frame buffer based on the cube map color data and depth data of the selected sample point.

14. The method of claim 8 , wherein a distance between the reconstruction point and the selected sample point is greater than an intra-ocular distance for stereoscopic reprojection of the reconstructed 3D virtual scene.

15. One or more computer storage media having computer-executable instructions embodied thereon that, when executed, by one or more processors, causes the one or more processors to perform a method for reprojecting images based on three-dimensional (3D) virtual images using discontinuity and depth late stage reprojection, the method comprising:
referencing a reconstruction point that indicates camera pose information associated with a 3D virtual scene, the 3D virtual scene is associated with a plurality of sample points each having a cube map of color data and depth data;
identifying a first closest sample point of

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Boulton Michael George
• Michail Ashraf Ayman

Classifications

• H04N13/398

  Synchronisation thereof; Control thereof · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G02B2027/0134

  of stereoscopic type · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title