Methods and processors for rendering a 3D object using multi-camera image inputs

Patent metadata

Field	Value
Publication number	US-12586296-B2
Application number	US-202418597468-A
Country	US
Kind code	B2
Filing date	Mar 6, 2024
Priority date	Mar 6, 2024
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

Methods and processors for rendering a 3D object are disclosed. The method includes acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters including an initial frame-wise GS parameter and an initial camera-wise GS parameter, generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: the initial frame-wise GS parameter, the initial camera-wise GS parameter, generating, by the adjusted 3DGS model, a 3DGS output and rendering a 2D image of the 3D object using the 3DGS output.

First claim

The invention claimed is:

1 . A method rendering a three-dimensional (3D) object, the method executable by a processor, the method comprising:
acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera; acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter; generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: (i) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and (ii) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera;
wherein the adjusting the initial frame-wise GS parameter is executed in accordance with:

μ
′

=

M
R

⁢
μ

+

M
T

wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, M_R is a rotation matrix, and M_T is a translation matrix;
generating, by the adjusted 3DGS model, a 3DGS output based on the multi-camera image input;
rendering a two-dimensional (2D) image of the 3D object using the 3DGS output.

2 . The method of claim 1 , wherein the generating the adjusted 3DGS model comprises adjusting both the initial frame-wise GS parameter and the initial camera-wise GS parameter.

3 . The method of claim 1 , wherein the initial frame-wise GS parameter is an initial GS position parameter, and the initial camera-wise GS parameter is an initial GS Spherical Harmonics (SH) parameter.

4 . The method of claim 1 , wherein the plurality of initial parameters further comprises a GS rotation parameter, a GS scale parameter, and a GS opacity parameter.

5 . A method rendering a three-dimensional (3D) object, the method executable by a processor, the method comprising:
acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera; acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter; generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: (iii) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and (iv) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera; wherein the adjusting the initial camera-wise GS parameter is executed in accordance with:

SH
d

′

=

β
d

⁢
SH

+

γ
d

,

SH
r

′

=

β
r

⁢
SH

+

γ
r

,

wherein β is a scaling factor, γ is a bias factor, SH is the initial camera-wise GS SH parameter, SH d ′ is a 0th band of an adjusted camera-wise GS SH parameter, SH r ′ is a higher band of an adjusted camera-wise GS SH parameter;
generating, by the adjusted 3DGS model, a 3DGS output based on the multi-camera image input;
rendering a two-dimensional (2D) image of the 3D object using the 3DGS output.

6 . The method of claim 1 , wherein the method further comprises:
using a Multi-layer Perceptron (MLP) model to optimize the rotation matrix and the translation matrix.

7 . The method of claim 5 , wherein the method further comprises:
using a Multi-layer Perceptron (MLP) model to optimize the scaling factor and the bias factor.

8 . The method of claim 1 , wherein the rendering comprises employing a differentiable renderer to project the 3DGS output onto a 2D plane.

9 . The method of claim 1 , wherein the method further comprises:
generating a camera memory bank using the multi-camera image data; and generating a global camera index using the camera memory bank.

10 . A processor for rendering a three-dimensional (3D) object, the processing being configured to:
acquire multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera; acquire an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter; generate an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: (v) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and (vi) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera; wherein to adjust the initial frame-wise GS parameter is executed by the processor in accordance with:

μ
′

=

M
R

⁢
μ

+

M
T

wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, M R is a rotation matrix, and M T is a translation matrix;
generate, using the adjusted 3DGS model, a 3DGS output based on the multi-camera image input; and
render a two-dimensio

Assignees

• Shenzhen Yinwang Intelligent Technology Co Ltd

Inventors

• Wu Guile
• Cao Tongtong
• Liu Bingbing

Classifications

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T15/08

  Volume rendering · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title