Model generation method and apparatus, image perspective determining method and apparatus, device, and medium

Patent metadata

Field	Value
Publication number	US-12394141-B2
Application number	US-202117928553-A
Country	US
Kind code	B2
Filing date	Jun 8, 2021
Priority date	Jun 8, 2020
Publication date	Aug 19, 2025
Grant date	Aug 19, 2025

Abstract

Provided are a model generation method and apparatus, an image perspective determining method and apparatus, a device, and a medium. The model generation method includes that: point cloud data and multiple image perspectives are collected to obtain coordinate data of the point cloud data and multiple image collection time points; a pose matrix corresponding to each image collection time point is determined, and a point cloud perspective at each image collection time point is generated according to the pose matrix and the coordinate data; and the point cloud perspective at each image collection time point and the respective one image perspective at the each image collection time point are used as a group of training samples, an original neural network model is trained based on multiple groups of training samples, and an image conversion model for converting a point cloud perspective into an image perspective is generated.

First claim

What is claimed is:

1. A model generation method, comprising:
collecting point cloud data and a plurality of image perspectives based on a preset collection system to obtain coordinate data of the point cloud data and a plurality of image collection time points, wherein each image collection time point of the plurality of image collection time points corresponds to a respective one image perspective of the plurality of image perspectives;
determining a pose matrix corresponding to each image collection time point of the plurality of image collection time points, and generating a point cloud perspective at each image collection time point according to the pose matrix corresponding to the each image collection time point and the coordinate data; and
using the point cloud perspective at each image collection time point and the respective one image perspective at the each image collection time point as a group of training samples, training an original neural network model based on a plurality of groups of training samples, and generating an image conversion model for converting a point cloud perspective into an image perspective;
the model generation method further comprises: using point cloud perspectives at at least two image collection time points and image perspectives corresponding to the point cloud perspectives at the at least two image collection time points as one group of training samples;
wherein using the point cloud perspectives at the at least two image collection time points and the image perspectives corresponding to the point cloud perspectives at the at least two image collection time points as the one group of training samples comprises:
using a point cloud perspective at a current image collection time point of the plurality of image collection time points as a first point cloud perspective, and using an image perspective at the current image collection time point as a first image perspective;
using a point cloud perspective at at least one image collection time point before the current image collection time point as a second point cloud perspective, and using an image perspective at the at least one image collection time point before the current image collection time point as a second image perspective; and
using the first point cloud perspective, the second point cloud perspective, the first image perspective, and the second image perspective as the one group of training samples, wherein the first point cloud perspective, the second point cloud perspective, and the second image perspective are actual input data, and the first image perspective is expected output data;
wherein the original neural network model includes a point cloud convolution excitation module, an image convolution excitation module, and a merging processing module, and wherein training the original neural network model based on the plurality of groups of training samples comprises:
inputting the plurality of groups of training samples into the original neural network model;
processing, through the point cloud convolution excitation module, a channel cascade result of the first point cloud perspective and the second point cloud perspective to obtain a point cloud feature map, and processing, through the image convolution excitation module, the second image perspective to obtain an image feature map; and
merging, through the merging processing module, the point cloud feature map and the image feature map, generating a third image perspective according to a merging processing result, and adjusting a network parameter of the original neural network model according to the third image perspective and the first image perspective.

2. The method of claim 1 , wherein determining the pose matrix corresponding to each image collection time point of the plurality of image collection time points comprises:
obtaining a pose trajectory of the preset collection system according to the point cloud data; and
sampling the pose trajectory based on the plurality of image collection time points to obtain the pose matrix corresponding to each image collection time point of the plurality of image collection time points.

3. The method of claim 1 , further comprising:
acquiring pixel points corresponding to the point cloud data in the point cloud perspective, and assigning attribute information of the point cloud data to the pixel points.

4. An image perspective determining method, comprising:
collecting point cloud data based on a preset collection system to obtain coordinate data of the point cloud data and a point cloud collection time point, determining a pose matrix corresponding to the point cloud collection time point, and generating a point cloud perspective at the point cloud collection time point according to the pose matrix and the coordinate data; and
acquiring the image conversion model generated according to the model generation method of claim 1 , inputting the point cloud perspective into the image conversion model, and determining an image perspective at the point cloud collection time point according to an output result of the image conversion model.

5. A model generation apparatus, comprising: at least one processor, and a memory which is configured to store at least one program, wherein the at least one program, when executed by the at least one processor, causes the at least one processor to implement:
collecting point cloud data and a plurality of image perspectives based on a preset collection system to obtain coordinate data of the point cloud data and a plurality of image collection time points, wherein each image collection time point of the plurality of image collection time points corresponds to a respective one image perspective of the plurality of image perspectives;
determining a pose matrix corresponding to each image collection time point of the plurality of image collection time points, and generating a point cloud perspective at each image collection time point according to the pose matrix corresponding to the each image collection time point and the coordinate data; and
using the point cloud perspective at each image collection time point and the respective one image perspective at the each image collection time point as a group of training samples, training an original neural network model based on a plurality of groups of training samples, and generating an image conversion model for converting a point cloud perspective into an image perspective;
wherein the at least one program, when executed by the at least one processor, causes the at least one processor to further implement: using point cloud perspectives at at least two image collection time points and image perspectives corresponding to the point cloud perspectives at the at least two image collection time points as one group of training samples;
wherein using the point cloud perspectives at the at least two image collection time points and the image perspectives corresponding to the point cloud perspectives at the at least two image collection time points as the one group of training samples comprises:
using a point cloud perspective at a current image collection time point of the plurality of image collection time points as a first point cloud perspective, and using an image perspective at the current image collection time point as a first image perspective;
using a point cloud perspective at at least one image collection time point before the current image collection time point as a second point cloud perspective, and using an image perspective at the at least one image collection time point before the current image collection time point as a second image perspective; and
using the first point cloud perspective, the second point cloud perspective, the first image perspective, and the second image perspective as the one group of training samples,

Assignees

• Beijing Jingdong Qianshi Tech Co Ltd

Inventors

• Li Yanli
• Liu Dongdong

Classifications

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/73

  using feature-based methods · CPC title