Methods and processors for rendering a 3d object using multi-camera image inputs

Patent metadata

Field	Value
Publication number	US-2025285364-A1
Application number	US-202418597468-A
Country	US
Kind code	A1
Filing date	Mar 6, 2024
Priority date	Mar 6, 2024
Publication date	Sep 11, 2025
Grant date	—

Abstract

Methods and processors for rendering a 3D object are disclosed. The method includes acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters including an initial frame-wise GS parameter and an initial camera-wise GS parameter, generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: the initial frame-wise GS parameter, the initial camera-wise GS parameter, generating, by the adjusted 3DGS model, a 3DGS output and rendering a 2D image of the 3D object using the 3DGS output.

First claim

1 . A method rendering a 3D object, the method executable by a processor, the method comprising:
acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera; acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise GS parameter and an initial camera-wise GS parameter; generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: (i) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and (ii) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera; generating, by the adjusted 3DGS model, a 3DGS output based on the multi-camera image input; rendering a 2D image of the 3D object using the 3DGS output.

2 . The method of claim 1 , wherein the generating the adjusted 3DGS model comprises adjusting both the initial frame-wise GS parameter and the initial camera-wise GS parameter.

3 . The method of claim 1 , wherein the initial frame-wise GS parameter is an initial GS position parameter, and the initial camera-wise GS parameter is an initial GS Spherical Harmonics (SH) parameter.

4 . The method of claim 1 , wherein the plurality of initial parameters further comprises a GS rotation parameter, a GS scale parameter, and a GS opacity parameter.

5 . The method of claim 1 , wherein the adjusting the initial frame-wise GS parameter is executed in accordance with:

μ
′

=

M
R

⁢
μ

+

M
T

wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, M R is a rotation matrix, and M T is a translation matrix.

6 . The method of claim 1 , wherein the adjusting the initial camera-wise GS parameter is executed in accordance with:

SH
d

′

=

β
d

⁢
SH

+

γ
d

,

SH
r

′

=

β
r

⁢
SH

+

γ
r

,

wherein β is a scaling factor, γ is a bias factor, SH is the initial camera-wise GS SH parameter, SH d ′ is a 0th band of an adjusted camera-wise GS SH parameter, SH r ′ is a higher band of an adjusted camera-wise GS SH parameter.

7 . The method of claim 5 , wherein the method further comprises:
using a Multi-layer Perceptron (MLP) model to optimize the rotation matrix and the translation matrix.

8 . The method of claim 6 , wherein the method further comprises:
using a Multi-layer Perceptron (MLP) model to optimize the scaling factor and the bias factor.

9 . The method of claim 1 , wherein the rendering comprises employing a differentiable renderer to project the 3DGS output onto a 2D plane.

10 . The method of claim 1 , wherein the method further comprises:
generating a camera memory bank using the multi-camera image data; and generating a global camera index using the camera memory bank.

11 . A processor for rendering a 3D object, the processing being configured to:
acquire multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera; acquire an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise GS parameter and an initial camera-wise GS parameter; generate an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: (iii) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and (iv) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera; generate, using the adjusted 3DGS model, a 3DGS output based on the multi-camera image input; render a 2D image of the 3D object using the 3DGS output.

12 . The processor of claim 11 , wherein to generate the adjusted 3DGS model comprises the processor configured to adjust both the initial frame-wise GS parameter and the initial camera-wise GS parameter.

13 . The processor of claim 11 , wherein the initial frame-wise GS parameter is an initial GS position parameter, and the initial camera-wise GS parameter is an initial GS Spherical Harmonics (SH) parameter.

14 . The processor of claim 11 , wherein the plurality of initial parameters further comprises a GS rotation parameter, a GS scale parameter, and a GS opacity parameter.

15 . The processor of claim 11 , wherein to adjust the initial frame-wise GS parameter is executed by the processor in accordance with:

μ
′

=

M
R

⁢
μ

+

M
T

wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, M R is a rotation matrix, and M T is a translation matrix.

16 . The processor of claim 11 , wherein to adjust the initial camera-wise GS parameter is executed in accordance with:

SH
d

′

=

β
d

Assignees

• Shenzhen Yinwang Intelligent Technology Co Ltd

Inventors

• Wu Guile
• Cao Tongtong
• Liu Bingbing

Classifications

• G06T15/08

  Volume rendering · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T7/55 Primary

  from multiple images · CPC title