Methods and systems for training quantized neural radiance field

Patent metadata

Field	Value
Publication number	US-12530840-B2
Application number	US-202318369904-A
Country	US
Kind code	B2
Filing date	Sep 19, 2023
Priority date	Mar 26, 2021
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

A computer-implemented method includes encoding a radiance field of an object onto a machine learning model; conducting, based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process includes a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points located within a threshold distance from a surface region of the object; obtaining target view parameters indicating a view direction of the object; obtaining a plurality of rays associated with a target image of the object; obtaining render sample points on the plurality of rays associated with the target image; and rendering, by inputting the render sample points to the trained machine learning model, colors associated with the pixels of the target image.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
encoding, by a computer device, a radiance field of an object onto a machine learning model; conducting, by the computer device and based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points, each of the plurality of first test sample points and the plurality of second test sample points located on a plurality of training rays derived from the training images, the plurality of second test sample points located within a threshold distance from a surface region of the object; obtaining target view parameters indicating a viewpoint and a view direction of the object; obtaining, based on the target view parameters, a plurality of sample rays associated with a target image of the object, the target image associated with the viewpoint and the view direction; obtaining, by the computer device, render sample points on the plurality of sample rays; and rendering, by inputting the render sample points to the trained machine learning model, colors associated with pixels of the target image.

2 . The computer-implemented method according to claim 1 , wherein the radiance field comprises a three-dimensional (3D) rendering space enclosing the object.

3 . The computer-implemented method according to claim 2 , wherein the machine learning model is a fully connected neural network comprising at least one node, each having an associated weight.

4 . The computer-implemented method according to claim 3 , wherein the machine learning model is configured to accept a position vector and a direction vector of a point in the 3D rendering space as an input and output a density and a radiance of the point, the position vector indicating a location of the point with respect to the viewpoint, and the direction vector indicating a relative direction of the point with respect to the viewpoint.

5 . The computer-implemented method according to claim 4 , wherein during the second training process, values of the weights, elements of the position vectors, elements of the direction vectors, the densities, and the radiances are quantized.

6 . The computer-implemented method according to claim 5 , wherein the step of obtaining the render sample points on the plurality of sample rays comprises: obtaining the render sample points, wherein the render sample points are located within the threshold distance from the surface region of the object.

7 . The computer-implemented method according to claim 5 , wherein the surface region of the object is obtained using a marching cube technique based on the set of training images, and the surface region is in a form of triangular mesh.

8 . The computer-implemented method according to claim 5 , wherein the threshold distance is 2 cm.

9 . A device comprising:
a processor; and a memory storing instructions executable by the processor, wherein the instructions are executed by the processor, the instructions cause the processor to perform operations, comprising:
encoding a radiance field of an object onto a machine learning model:
conducting, based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points, each of the plurality of first test sample points and the plurality of second test sample points located on a plurality of training rays derived from the training images, the plurality of second test sample points located within a threshold distance from a surface region of the object;
obtaining target view parameters indicating a viewpoint and a view direction of the object;
obtaining, based on the target view parameters, a plurality of sample rays associated with a target image of the object, the target image associated with the viewpoint and the view direction;
obtaining render sample points on the plurality of sample rays; and
rendering, by inputting the render sample points to the trained machine learning model, colors associated with pixels of the target image.

10 . The device according to claim 9 , wherein the radiance field comprises a three-dimensional (3D) rendering space enclosing the object.

11 . The device according to claim 10 , wherein the machine learning model is a fully connected neural network comprising at least one node, each having an associated weight.

12 . The device according to claim 11 , wherein the machine learning model is configured to accept a position vector and a direction vector of a point in the 3D rendering space as an input and output a density and a radiance at the point, the position vector indicating a location of the point with respect to the viewpoint, and the direction vector indicating a relative direction of the point with respect to the viewpoint.

13 . The device according to claim 12 , wherein during the second training process, values of the weights, elements of the position vectors, elements of the direction vectors, the densities, and the radiances are quantized.

14 . The device according to claim 13 , wherein the step of obtaining the render sample points on the plurality of sample rays comprises: obtaining the render sample points, wherein the render sample points are located within the threshold distance from the surface region of the object.

15 . The device according to claim 13 , wherein the surface region of the object is obtained using a marching cube technique based on the set of training images, and the surface region is in a form of triangular mesh.

16 . The device according to claim 13 , wherein the threshold distance is 2 cm.

17 . A non-transitory storage medium of storing instructions executable by a process, wherein, the instructions are executed by the processors, the instructions cause the processor to perform operations, comprising:
encoding a radiance field of an object onto a machine learning model; conducting, based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points, each of the plurality of first test sample points and the plurality of second test sample points located on a plurality of training rays derived from the training images, the plurality of second test sample points located within a threshold distance from a surface region of the object; obtaining target view parameters indicating a viewpoint and a view direction of the object; obtaining, based on the target view parameters, a plurality of sample rays associated with a target image of the object, the target image associated with the viewpoint and the view direction; obtaining render sample points on the plurality of sample rays associated with the target image; and rendering, by inputting the render sample points to the trained machine learning model, colors associated with pixels of the target image.

18 . The non-transitory storage medium according to claim 17 , wherein the radiance field comprises a t

Assignees

• Univ Shanghai Technology

Inventors

• Wu Minye
• Rao Chaolin
• Lou Xin
• Zhou Pingqiang
• Yu Jingyi

Classifications

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06T15/506

  Illumination models · CPC title

• G06T15/55 Primary

  Radiosity · CPC title