Three-dimensional construction network training method and apparatus, and three-dimensional model generation method and apparatus

Patent metadata

Field	Value
Publication number	US-2026017885-A1
Application number	US-202519333120-A
Country	US
Kind code	A1
Filing date	Sep 18, 2025
Priority date	Mar 21, 2023
Publication date	Jan 15, 2026
Grant date	—

Abstract

This application provides a three-dimensional construction network training method and apparatus, and a three-dimensional model generation method and apparatus in the field of computer vision, to perform joint training based on a plurality of frames of images and radar point cloud data to obtain a three-dimensional construction network. More accurate depths included in the radar point cloud data may be used as deep supervision. The method includes: obtaining the plurality of frames of images and a photographing parameter used by a camera device when the plurality of frames of images are photographed, where the plurality of frames of images include images photographed from a plurality of views; obtaining the radar point cloud data including photographing scenario data; and obtaining the three-dimensional construction network based on the plurality of frames of images, the photographing parameter, and the radar point cloud data.

First claim

1 . A computer-implemented method of three-dimensional construction network training, comprising:
obtaining a plurality of frames of images and a photographing parameter used by a camera device when the plurality of frames of images are photographed, wherein the plurality of frames of images comprise images photographed from a plurality of views; obtaining radar point cloud data, wherein a capture scenario in which a radar captures the radar point cloud data has an intersection with a photographing scenario in which the camera device photographs the plurality of frames of images; and obtaining a three-dimensional construction network based on the plurality of frames of images, the photographing parameter, and the radar point cloud data, wherein the three-dimensional construction network is used to perform three-dimensional construction based on input data to output a three-dimensional model, and the radar point cloud data is used as deep supervision during three-dimensional construction of the three-dimensional construction network.

2 . The method according to claim 1 , wherein obtaining the three-dimensional construction network comprises:
constructing a virtual camera based on the plurality of frames of images and the photographing parameter, to obtain virtual camera data comprising an image at a view different from the plurality of views; and performing iterative training on an initial model to obtain the three-dimensional construction network using the virtual camera data as an input of the three-dimensional construction network and using the radar point cloud data as the deep supervision during the three-dimensional construction of the three-dimensional construction network.

3 . The method according to claim 2 , wherein constructing the virtual camera comprises:
obtaining a pose parameter of each frame of image in the plurality of frames of images; projecting the plurality of frames of images into a same space based on the pose parameter of each frame of image, to obtain image point cloud data; and projecting the image point cloud data based on an enhanced view to obtain the virtual camera data, wherein the enhanced view is a view different from the plurality of views.

4 . The method according to claim 3 , wherein using the radar point cloud data as the deep supervision during the three-dimensional construction of the three-dimensional construction network comprises:
fusing the radar point cloud data and the image point cloud data to obtain depths of a plurality of pixels; and using the depths of the plurality of pixels as the deep supervision during the three-dimensional construction of the three-dimensional construction network.

5 . The method according to claim 2 , wherein a time of iterative training of the iterative training comprises:
obtaining at least one first sampling point on a first ray in a first view in a three-dimensional model output by a three-dimensional construction network obtained through a previous iteration; obtaining at least one second sampling point on a second ray in a second view of the three-dimensional model, wherein the at least one second sampling point is a point obtained by projecting the at least one first sampling point on the second ray, and the first view and the second view are different views in the three-dimensional model; obtaining a difference between the at least one first sampling point and the at least one second sampling point; and updating the three-dimensional construction network based on the difference, to obtain a three-dimensional construction network updated in a current iterative update.

6 . The method according to claim 5 , wherein obtaining the difference between the at least one first sampling point and the at least one second sampling point comprises:
obtaining a first depth estimate of the at least one first sampling point and a second depth estimate of the at least one second sampling point; and obtaining a difference between the first depth estimate and the second depth estimate

7 . The method according to claim 1 , wherein the three-dimensional construction network comprises:
a first geometric module; a second geometric module; a first color module; a second color module; and an encoding module configured to encode a value of an input pixel, and then separately input the encoded value to the first geometric module, the second geometric module, and the second color module; wherein the first geometric module and the second geometric module are configured to perform geometric construction based on the input data, to output a geometric structure of a three-dimensional scene, wherein an output precision of the second geometric module is higher than an output precision of the first geometric module; and wherein the first color module and the second color module are configured to perform color construction based on the input data, to output a color value of each pixel in the three-dimensional scene, wherein an output precision of the second color module is higher than an output precision of the first color module, and an input of the first color module comprises an input of the first geometric module.

8 . The method according to claim 7 , wherein the input of the first color module and an input of the second color module each further comprises a depth comprising a value obtained based on the radar point cloud data.

9 . The method according to claim 7 , further comprising:
obtaining an input pose and a camera parameter; and obtaining an output image from the three-dimensional scene based on the input pose and the camera parameter.

10 . The method according to claim 1 , wherein the three-dimensional model output by the three-dimensional construction network is applied to autonomous driving of a vehicle.

11 . A computer-implemented method of three-dimensional model generation, comprising:
obtaining input view information; and outputting a three-dimensional model using the input view information as an input of a three-dimensional construction network obtained based on a plurality of frames of images, a photographing parameter, a radar point cloud data, wherein the photographing parameter is a parameter used by a camera device when the plurality of frames of images are photographed, the plurality of frames of images comprise images photographed from a plurality of views, the radar point cloud data is data captured by a radar, and a capture scenario in which the radar captures the radar point cloud data has an intersection with a photographing scenario in which the camera device photographs the plurality of frames of images.

12 . The method according to claim 11 , wherein the three-dimensional construction network comprises:
a first geometric module; a second geometric module; a first color module; a second color module; and an encoding module configured to encode a value of an input pixel, and then separately input the encoded value to the first geometric module, the second geometric module, and the second color module; wherein the first geometric module and the second geometric module are configured to perform geometric construction based on input data, to output a geometric structure in a three-dimensional scene, wherein an output precision of the second geometric module is higher than an output precision of the first geometric module; and wherein the first color module and the second color module are configured to perform color construction based on input data, to output a color value of each pixel in a three-dimensional scene, wherein an output precision of the second color module is higher than an output precision of the first color module, and an input of the first

Assignees

• Huawei Tech Co Ltd

Inventors

• Bai Dongfeng
• Qi Dexin
• Liu Bingbing

Classifications

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title