Learnable global bases for generating three-dimensional representations from single-view data collections

Patent metadata

Field	Value
Publication number	US-2026099998-A1
Application number	US-202519351193-A
Country	US
Kind code	A1
Filing date	Oct 6, 2025
Priority date	Oct 8, 2024
Publication date	Apr 9, 2026
Grant date	—

Abstract

Generating a three-dimensional representation from a single-view includes receiving a single-view image, generating a plurality of coefficients, generating a 3D representation from a plurality of basis elements and the plurality of coefficients, processing the 3D representation and the single-view image to generate a plurality of optimized coefficients, generating an optimized 3D representation from the plurality of coefficients and the plurality of optimized basis elements, and rendering the optimized 3D representation to generate a volume rendering, and reconstructing a 3D scene from the volume rendering.

First claim

What is claimed is:

1 . A computer-implemented method for reconstructing 3D scenes, the method comprising:
receiving a single-view image; generating a plurality of coefficients; generating an optimized 3D representation from a plurality of optimized basis elements and the plurality of coefficients; rendering the optimized 3D representation to generate a volume rendering; and reconstructing a 3D scene from the volume rendering.

2 . The computer-implemented method of claim 1 , wherein the single-view image is a 2D image.

3 . The computer-implemented method of claim 1 , wherein the plurality of optimized basis elements are voxels or triplanes.

4 . The computer-implemented method of claim 1 , wherein generating the optimized 3D representation comprises generating a linear combination of the plurality of optimized basis elements using the plurality of coefficients.

5 . The computer-implemented method of claim 1 , wherein generating the volume rendering comprises ray casting or shear warping.

6 . The computer-implemented method of claim 1 , wherein generating the plurality of coefficients comprises using a machine learning model.

7 . The computer-implemented method of claim 6 , wherein the machine learning model comprises a vision transformer.

8 . The computer-implemented method of claim 1 , wherein generating the plurality of coefficients comprises:
processing the single-view image using a machine learning model to generate a partial observation map, a depth map, and a probability distribution map; sampling the depth map to generate a dense set of 3D points; and performing Monte Carlo integration on 3D points in the dense set of 3D points based on the probability distribution map to generate a plurality of coefficients.

9 . The computer-implemented method of claim 8 , wherein the machine learning model comprises a U-Net model or a convolutional network.

10 . The computer-implemented method of claim 1 , wherein generating the plurality of optimized bases elements comprises:
generating a 3D representation from a plurality of basis elements and the plurality of coefficients; rendering the 3D representation to generate a plurality of volume renderings; and minimizing a batch reconstruction loss between the plurality of volume renderings and a plurality of single-view images to generate the plurality of optimized bases elements.

11 . The computer-implemented method of claim 10 , wherein the batch reconstruction loss comprises one or more of an L1 loss, a mean squared error, or an LPIPS metric.

12 . The computer-implemented method of claim 1 , wherein generating the plurality of optimized bases elements comprises:
generating a 3D representation from a plurality of basis elements and the plurality of coefficients; rendering the 3D representation to generate a plurality of volume renderings; and minimizing a batch reconstruction loss between the plurality of volume renderings and a partial observation map and the plurality of volume renderings and a plurality of single-view images to generate the plurality of optimized bases elements.

13 . The computer-implemented method of claim 12 , wherein the batch reconstruction loss comprises one or more of an L1 loss, a mean squared error, or an LPIPS metric.

14 . One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
receiving a single-view image; generating a plurality of coefficients; generating an optimized 3D representation from a plurality of optimized basis elements and the plurality of coefficients; rendering the optimized 3D representation to generate a volume rendering; and reconstructing a 3D scene from the volume rendering.

15 . The one or more non-transitory computer-readable media of claim 14 , wherein generating the optimized 3D representation comprises generating a linear combination of the plurality of optimized basis elements using the plurality of coefficients.

16 . The one or more non-transitory computer-readable media of claim 14 , wherein generating the plurality of coefficients comprises using a machine learning model.

17 . The one or more non-transitory computer-readable media of claim 14 , wherein generating the plurality of coefficients comprises:
processing the single-view image using a machine learning model to generate a partial observation map, a depth map, and a probability distribution map; sampling the depth map to generate a dense set of 3D points; and performing Monte Carlo integration on 3D points in the dense set of 3D points based on the probability distribution map to generate a plurality of coefficients.

18 . The one or more non-transitory computer-readable media of claim 14 , wherein generating the plurality of optimized bases elements comprises:
generating a 3D representation from a plurality of basis elements and the plurality of coefficients; rendering the 3D representation to generate a plurality of volume renderings; and minimizing a batch reconstruction loss between the plurality of volume renderings and a plurality of single-view images to generate the plurality of optimized bases elements.

19 . The one or more non-transitory computer-readable media of claim 14 , wherein generating the plurality of optimized bases elements comprises:
generating a 3D representation from a plurality of basis elements and the plurality of coefficients; rendering the 3D representation to generate a plurality of volume renderings; minimizing a batch reconstruction loss between the plurality of volume renderings and a partial observation map and the plurality of volume renderings and a plurality of single-view images to generate the plurality of optimized bases elements.

20 . A system, comprising:
one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform steps comprising:
receiving a single-view image;
generating a plurality of coefficients;
generating an optimized 3D representation from a plurality of optimized basis elements and the plurality of coefficients;
rendering the optimized 3D representation to generate a volume rendering; and
reconstructing a 3D scene from the volume rendering.

Assignees

• Nvidia Corp

Inventors

• Nagano Koki
• Jiang Kaiwen
• De Mello Shalini

Classifications

• G06T15/08

  Volume rendering · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title