Image rendering method, electronic device, and storage medium

Patent metadata

Field	Value
Publication number	US-12573148-B2
Application number	US-202318185359-A
Country	US
Kind code	B2
Filing date	Mar 16, 2023
Priority date	Jun 24, 2022
Publication date	Mar 10, 2026
Grant date	Mar 10, 2026

Abstract

An image rendering method includes the steps below. A model of an environmental object is rendered to obtain an image of the environmental object in a target perspective. An image of a target object in the target perspective and a model of the target object are determined according to a neural radiance field of the target object. The image of the target object is fused and rendered into the image of the environmental object according to the model of the target object.

First claim

What is claimed is:

1 . An image rendering method, comprising:
rendering a model of an environmental object to obtain an image of the environmental object in a target perspective; determining an image of a target object in the target perspective and a model of the target object according to a neural radiance field of the target object; and fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object; wherein fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object comprises: determining a normal direction of a target vertex in the model of the target object according to the model of the target object; performing a light adjustment on the image of the target object to obtain a light-adjusted image of the target object according to the target perspective, environmental light, and the normal direction of the target vertex; and fusing and rendering the light-adjusted image of the target object into the image of the environmental object; or wherein fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object comprises: determining depth information of a target vertex in the model of the target object according to the target perspective and the model of the target object; comparing depth information of a respective target vertex at each pixel in the model of the target object with depth information of a respective environmental object at the each pixel; in response to determining that the depth information of the respective target vertex is smaller than the depth information of the respective environmental object, determining that the respective target vertex is not occluded by the respective environmental object, and replacing a pixel value of the respective environmental object by a pixel value of the respective target vertex; and in response to determining that the depth information of the respective target vertex is greater than or equal to the depth information of the respective environmental object, determining that the respective target vertex is occluded by the respective environmental object, and discarding the pixel value of the respective target vertex; wherein determining the image of the target object in the target perspective and the model of the target object according to the neural radiance field of the target object comprises: acquiring a point cloud model of the target object from the neural radiance field of the target object, and processing the point cloud model of the target object to obtain a mesh model of the target object; and according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using a camera projection matrix and a model view matrix in the target perspective; wherein according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using the camera projection matrix and the model view matrix in the target perspective comprises: inputting the camera projection matrix and the model view matrix in the target perspective to the neural radiance field of the target object so that the image of the target object in the target perspective is obtained by a neural rendering; and wherein the neural radiance field of the target object is determined by: acquiring, in an acquisition perspective, data of the target object to obtain a two-dimensional image of the target object and a three-dimensional point cloud of the target object using a camera and a laser radar, wherein the camera shoots around a center of visual field at different heights and in a case where the camera is rotated, the camera is rotated slowly and moved back and forth to increase coincidence rate of visual field between adjacent frames; fusing the two-dimensional image of the target object and the three-dimensional point cloud of the target object to obtain a fused image of the target object; and determining the neural radiance field of the target object according to the fused image of the target object and the acquisition perspective.

2 . The method according to claim 1 , wherein performing the light adjustment on the image of the target object to obtain the light-adjusted image of the target object according to the target perspective, the environmental light, and the normal direction of the target vertex comprises:
transforming, by using the model view matrix in the target perspective, the normal direction of the target vertex to obtain a transformed normal direction of the target vertex; determining an intensity of the target vertex according to the environmental light and the transformed normal direction of the target vertex; and performing the light adjustment on the image of the target object to obtain the light-adjusted image of the target object according to the intensity of the target vertex.

3 . The method according to claim 1 , wherein determining the depth information of the target vertex in the model of the target object according to the target perspective and the model of the target object comprises:
transforming, by using the model view matrix in the target perspective, the model of the target object to obtain a transformed model of the target object; and projecting, by using a camera projection matrix in the target perspective, the transformed model of the target object to obtain the depth information of the target vertex in the model of the target object.

4 . The method according to claim 1 , wherein processing the point cloud model of the target object to obtain the mesh model of the target object comprises:
according to a marching cubes algorithm, processing the point cloud model of the target object to obtain the mesh model of the target object.

5 . An electronic device, comprising:
at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to execute: rendering a model of an environmental object to obtain an image of the environmental object in a target perspective; determining an image of a target object in the target perspective and a model of the target object according to a neural radiance field of the target object; and fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object; wherein the at least one processor fuses and renders the image of the target object into the image of the environmental object according to the model of the target object by: determining a normal direction of a target vertex in the model of the target object according to the model of the target object; performing a light adjustment on the image of the target object to obtain a light-adjusted image of the target object according to the target perspective, environmental light, and the normal direction of the target vertex; and fusing and rendering the light-adjusted image of the target object into the image of the environmental object; or wherein the at least one processor fuses and renders the image of the target object into the image of the environmental object according to the model of the target object by: determining depth information of a target vertex in the model of the target object according to the target perspective and the model of the target object; comparing depth information of a respective target vertex at each pixel in the model of the target object with depth information of a respective environmental object at the each pixel; in response to

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Liu Xing
• Chen Ruizhi
• Zhang Yan
• Zhao Chen
• Sun Hao
• Liu Jingtuo
• Ding Errui
• Wu Tian
• Wang Haifeng

Classifications

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title