Methods and processors for differentiable rendering of three dimensional gaussians for omnidirectional cameras

Patent metadata

Field	Value
Publication number	US-2025285360-A1
Application number	US-202418891649-A
Country	US
Kind code	A1
Filing date	Sep 20, 2024
Priority date	Mar 6, 2024
Publication date	Sep 11, 2025
Grant date	—

Abstract

Methods and processors for rendering a 3D Gaussian are disclosed. The processor is configured to acquire the 3D Gaussian to be rendered, and a camera model representing an omnidirectional camera with an optical axis, updating a color of the 3D Gaussian using spherical harmonics, update a position of the 3D Gaussian by moving the 3D Gaussian towards the optical axis, update a scale of the 3D Gaussian by compressing the 3D Gaussian in at least one of a tangential direction and a polar direction relative to the optical axis, and render an updated 3D Gaussian onto a 2D plane using a 3DGS model.

First claim

1 . A method of rendering a 3D Gaussian, the method executable by a processor, the method comprising:
acquiring the 3D Gaussian to be rendered, and a camera model representing an omnidirectional camera with an optical axis; updating a color of the 3D Gaussian using spherical harmonics; updating a position of the 3D Gaussian by moving the 3D Gaussian towards the optical axis; updating a scale of the 3D Gaussian by compressing the 3D Gaussian in at least one of a tangential direction and a polar direction relative to the optical axis; and rendering an updated 3D Gaussian onto a 2D plane using a 3DGS model.

2 . The method of claim 1 , wherein the 3D Gaussian is a subset of 3D Gaussians, the omnidirectional camera further having a Field of View (FOV), and wherein the method further comprises:
acquiring a plurality of 3D Gaussians; selecting the sub-set of 3D Gaussians amongst the plurality of 3D Gaussians using the FOV, the selected sub-set being a visible subset of 3D Gaussians.

3 . The method of claim 1 , wherein the updating the position of the 3D Gaussian comprises:
rotating a camera-Gaussian center vector of the 3D Gaussian towards the optical axis.

4 . The method of claim 1 , wherein the method further comprises:
applying a weighted matrix on a rendered 3D Gaussian for controlling convergence speed of respective pixels of the rendered 3D Gaussian.

5 . The method of claim 1 , wherein the updating the scale of the 3D Gaussian comprises:
generating at least one of a tangential scaling factor and a polar scaling factor, the polar scaling factor being model-specific to the camera model; and update the scale based on the at least one of the tangential scaling factor and the polar scaling factor.

6 . The method of claim 5 , wherein the updating the scale further includes:
generating a rescaled 3D Gaussian being smaller in size along the tangential direction than the 3D Gaussian.

7 . The method of claim 5 , wherein the updating the scale further includes:
generating a rescaled 3D Gaussian being smaller in size along the polar direction than the 3D Gaussian.

8 . The method of claim 1 , wherein the camera model is MEI camera model.

9 . The method of claim 1 , wherein the camera model is Kannala-Brandt camera model.

10 . The method of claim 1 , wherein the method further comprises:
generating a training dataset including the rendered 3D Gaussian; and training a machine learning model using the training dataset for 3D scene reconstruction.

11 . A processor for rendering a 3D Gaussian, the processor being configured to:
acquire the 3D Gaussian to be rendered, and a camera model representing an omnidirectional camera with an optical axis; update a color of the 3D Gaussian using spherical harmonics; update a position of the 3D Gaussian by moving the 3D Gaussian towards the optical axis; update a scale of the 3D Gaussian by compressing the 3D Gaussian in at least one of a tangential direction and a polar direction relative to the optical axis; and render an updated 3D Gaussian onto a 2D plane using a 3DGS model.

12 . The processor of claim 11 , wherein the 3D Gaussian is a subset of 3D Gaussians, the omnidirectional camera further having a Field of View (FOV), and wherein the processor is further configured to:
acquire a plurality of 3D Gaussians; select the sub-set of 3D Gaussians amongst the plurality of 3D Gaussians using the FOV, the selected sub-set being a visible subset of 3D Gaussians.

13 . The processor of claim 11 , wherein to update the position of the 3D Gaussian the processor is configured to:
rotate a camera-Gaussian center vector of the 3D Gaussian towards the optical axis.

14 . The processor of claim 11 , wherein the processor is further configured to:
apply a weighted matrix on a rendered 3D Gaussian for controlling convergence speed of respective pixels of the rendered 3D Gaussian.

15 . The processor of claim 11 , wherein to update the scale of the 3D Gaussian the processor is configured to:
generate at least one of a tangential scaling factor and a polar scaling factor, the polar scaling factor being model-specific to the camera model; and update the scale based on the at least one of the tangential scaling factor and the polar scaling factor.

16 . The processor of claim 15 , wherein to update the scale the processor is further configured to:
generate a rescaled 3D Gaussian being smaller in size along the tangential direction than the 3D Gaussian.

17 . The processor of claim 15 , wherein to update the scale the processor is further configured to:
generate a rescaled 3D Gaussian being smaller in size along the polar direction than the 3D Gaussian.

18 . The processor of claim 11 , wherein the camera model is one of a MEI camera model and a Kannala-Brandt camera model.

19 . The processor of claim 11 , wherein the processor is further configured to:
generate a training dataset including the rendered 3D Gaussian; and train a machine learning model using the training dataset for 3D scene reconstruction.

20 . One or more non-transitory, computer-readable storage media comprising computer-executable instructions, wherein the instructions, when executed, cause one or more processors to:
acquire the 3D Gaussian to be rendered, and a camera model representing an omnidirectional camera with an optical axis; update a color of the 3D Gaussian using spherical harmonics; update a position of the 3D Gaussian by moving the 3D Gaussian towards the optical axis; update a scale of the 3D Gaussian by compressing the 3D Gaussian in at least one of a tangential direction and a polar direction relative to the optical axis; and render an updated 3D Gaussian onto a 2D plane using a 3DGS model.

Assignees

• Huawei Tech Co Ltd

Inventors

• Ren Yuan
• Chen Xingxin
• Wu Guile
• Liu Bingbing

Classifications

• G06T15/08 Primary

  Volume rendering · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T15/10 Primary

  Geometric effects · CPC title