Techniques for performing point-based inverse rendering

Patent metadata

Field	Value
Publication number	US-11074743-B2
Application number	US-201916586746-A
Country	US
Kind code	B2
Filing date	Sep 27, 2019
Priority date	Sep 2, 2019
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

In various embodiments, a differentiable rendering application enables an inverse rendering application to infer attributes associated with a 3D scene. In operation, the differentiable rendering application renders an image based on a first set of points associated with the 3D scene. The differentiable rendering application then generates an artificial gradient that approximates a change in a value of a first pixel included in the image with respect to a change in an attribute of a first point included in the first set of points. Subsequently, the inverse rendering application performs optimization operation(s) on the first point based on the artificial gradient to generate a second set of points. Notably, an error associated with the second set of points is less than an error associated with the first set of points.

First claim

What is claimed is:

1. A computer-implemented method for inferring attributes associated with a three-dimensional (3D) scene, the method comprising:
generating a first image based on a first plurality of points associated with the 3D scene;
generating a gradient that represents a change in a value of a pixel included in the first image with respect to a given point included in the first plurality of points; and
performing one or more optimization operations on the given point based on the gradient to generate a second plurality of points, wherein an error associated with the second plurality of points is less than an error associated with the first plurality of points.

2. The computer-implemented method of claim 1 , wherein performing the one or more optimization operations comprises modifying a first position of the given point to generate a second position of a second point included in the second plurality of points, wherein the given point and the second point have a same first normal.

3. The computer-implemented method of claim 1 , wherein performing the one or more optimization operations comprises:
comparing the first image to a reference image to compute an image loss; and
modifying at least a first attribute of the given point based on the gradient to reduce the image loss.

4. The computer-implemented method of claim 1 , wherein performing the one or more optimization operations comprises:
comparing the first image to a reference image to compute an image loss;
computing one or more surface regularization losses based on the first plurality of points;
computing the error associated with the first plurality of points based on the image loss and the one or more surface regularization losses; and
moving the given point in a direction associated with the gradient to reduce the error associated with the first plurality of points.

5. The computer-implemented method of claim 1 , wherein number of distinct surfaces represented by the first plurality of points is not equal to number of distinct surfaces represented by the second plurality of points.

6. The computer-implemented method of claim 1 , wherein generating the gradient comprises:
computing a translation vector that acts to decrease an image loss between the first image and a reference image at the pixel; and
scaling the translation vector based on the change in the value of the pixel.

7. The computer-implemented method of claim 1 , wherein generating the gradient comprises:
factoring a discontinuous rasterization function to generate a visibility step function; and
computing an approximation of the change in the value of the pixel with respect to a change in a first attribute of the given point based on the visibility step function.

8. The computer-implemented method of claim 1 , wherein a first attribute of the given point comprises a normal, and generating the gradient comprises setting the gradient equal to zero.

9. The computer-implemented method of claim 1 , wherein a first attribute of the given point comprises a position or a normal, and the value of the pixel is associated with at least one of a color, a shading, and a depth value.

10. The computer implemented method of claim 1 , wherein rendering the first image comprises performing one or more elliptical filtering operations on the first plurality of points.

11. One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to infer attributes associated with a three-dimensional (3D) scene by performing the steps of:
generating a first image based on a first plurality of points associated with the 3D scene;
generating a gradient that represents a change in a value of a pixel included in the first image with respect to a given point included in the first plurality of points; and
performing one or more optimization operations on the given point based on the gradient to generate a second plurality of points, wherein an error associated with the second plurality of points is less than an error associated with the first plurality of points.

12. The one or more non-transitory computer readable media of claim 11 , wherein performing the one or more optimization operations comprises modifying a first position of the given point to generate a second position of a second point included in the second plurality of points, wherein the given point and the second point have a same first normal.

13. The one or more non-transitory computer readable media of claim 11 , wherein performing the one or more optimization operations comprises modifying a first attribute of the given point to reduce at least one of an image loss, a distance between the given point and a second point included in the first plurality of points, and a distance between the given point and a surface tangent plane.

14. The one or more non-transitory computer readable media of claim 11 , wherein a number of distinct surfaces represented by the first plurality of points is not equal to a number of distinct surfaces represented by the second plurality of points.

15. The one or more non-transitory computer readable media of claim 11 , further comprising:
computing a translation vector that acts to decrease an image loss between the first image and a reference image at the pixel; and
computing a change in the value of the pixel based on the translation vector.

16. The one or more non-transitory computer readable media of claim 11 , wherein generating the gradient comprises:
factoring a discontinuous rasterization function to generate a visibility step function; and
computing an approximation of a change in the value of the first pixel with respect to a change in an attribute of the given point based on the visibility step function.

17. The one or more non-transitory computer readable media of claim 11 , wherein an attribute of the given point comprises a normal, and generating the gradient comprises setting the gradient equal to zero.

18. The one or more non-transitory computer readable media of claim 11 , wherein an attribute of the given point comprises a position or a normal, and the value of the pixel is associated with at least one of a color, a shading, and a depth value.

19. The one or more non-transitory computer readable media of claim 11 , wherein generating the first image comprises performing one or more elliptical filtering operations on the first plurality of points.

20. A system, comprising:
one or more memories storing instructions; and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
generate a first image based on a first plurality of points associated with the 3D scene;
generating a gradient that represents a change in a value of a pixel included in the first image with respect to a given point included in the first plurality of points; and
performing one or more optimization operations on the given point based on the gradient to generate a second plurality of points, wherein an error associated with the second plurality of points is less than an error associated with the first plurality of points.

Assignees

• Disney Entpr Inc
• Eth Zuerich

Inventors

• Öztireli Ahmet Cengiz
• Sorkine-Hornung Olga
• Wu Shihao
• Wang Yifan

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T5/73

  Deblurring; Sharpening · CPC title