Dataset generation method for self-supervised learning scene point cloud completion based on panoramas

Patent metadata

Field	Value
Publication number	US-11875583-B2
Application number	US-202117533878-A
Country	US
Kind code	B2
Filing date	Nov 23, 2021
Priority date	Sep 30, 2021
Publication date	Jan 16, 2024
Grant date	Jan 16, 2024

Abstract

The present invention belongs to the technical field of 3D reconstruction in the field of computer vision, and provides a dataset generation method for self-supervised learning scene point cloud completion based on panoramas. Pairs of incomplete point cloud and target point cloud with RGB information and normal information can be generated by taking RGB panoramas, depth panoramas and normal panoramas in the same view as input for constructing a self-supervised learning dataset for training of the scene point cloud completion network. The key points of the present invention are occlusion prediction and equirectangular projection based on view conversion, and processing of the stripe problem and point-to-point occlusion problem during conversion. The method of the present invention includes simplification of the collection mode of the point cloud data in a real scene; occlusion prediction idea of view conversion; and design of view selection strategy.

First claim

The invention claimed is:

1. A dataset generation method for self-supervised learning scene point cloud completion based on panoramas, comprising the following steps:
step 1: generating initial point cloud from a panorama under a specific view by:
1.1) representing a three-dimensional world by a sphere, and representing coordinates in x, y and z directions by longitude and latitude, wherein a radius r of the sphere represents a depth value; assuming that a length of a depth panorama D 1 is a same as a range of −180° to 180° in a horizontal direction of a scene, and a width of the depth panorama D 1 corresponds to a range of −90° to 90° in a vertical direction; representing a coordinate of each pixel of the depth panorama D 1 with the longitude and the latitude, wherein a radius of a point in the sphere corresponding to each pixel is a depth value of each pixel in the depth panorama D 1 ; and in a spherical coordinate system, converting the latitude, longitude and depth values of each pixel into x, y and z coordinates in a camera coordinate system to generate point cloud P 0 ;
1.2) converting the point cloud P 0 in the camera coordinate system to a world coordinate system based on a camera extrinsic parameter corresponding to a view v 1 , and assigning a color information of a RGB panorama C 1 and a normal panorama N 1 to each point in the point cloud P 0 in a row column order of pixel points to generate initial point cloud P 1 with RGB information and initial point cloud P 2 with normal information;
step 2: selecting a new occlusion prediction view based on the initial point cloud P 1 and P 2 by:
2.1) encoding the initial point cloud P 1 by a truncated signed distance function; dividing a selected 3D space to be modeled into a plurality of blocks, and calling the block as a voxel; storing, by the voxel, a distance value between the block and a nearest object surface, and representing, by a symbol of the distance value, that the voxel is in a free space or a closed space; and conducting truncation processing if an absolute value of the distance value exceeds a set truncation distance D;
2.2) assuming that a voxel block corresponding to the view v 1 is t 0 ; updating a distance value of t 0 as 0; and updating the distance value of the voxel block near t 0 according to a distance from to, wherein if a distance from to is smaller, a decline of the distance value is larger;
2.3) traversing each voxel block to find a voxel block with the largest distance value; selecting the voxel block closest to a scene center if a plurality of the voxel blocks have a same distance value; randomly selecting from the voxel blocks which satisfy conditions if a distance from a scene center is still the same; and taking a center of a selected voxel block as a position of view v 2 to obtain a translation matrix of the view v 2 , with a rotation matrix of the view v 2 the same as a rotation matrix of the view v 1 ;
step 3: generating a panorama under the view v 2 from the initial point cloud P 1 and P 2 by:
3.1) converting the initial point cloud P 1 with the RGB information and the initial point cloud P 2 with normal information in the world coordinate system to the camera coordinate system based on a camera extrinsic parameter corresponding to the view v 2 ;
3.2) in the spherical coordinate system, converting the x, y and z coordinates of each point in the point cloud P 1 and the point cloud P 2 respectively into latitude, longitude and radius, and corresponding to a pixel position of a 2D panorama; making a color of each point correspond to the pixel position; increasing an influence range of each point, specifically, extending a calculated each pixel (x,y) outward to pixels (x,y), (x+1,y), (x,y+1) and (x+1,y+1); and copying a information carried by each pixel to a new pixels;
3.3) firstly, initializing a depth value of each pixel of depth panorama D 2 to a value 65535 that is represented by an unsigned 16-bit binary number, and initializing a color value of each pixel of a RGB panorama C 2 and a normal panorama N 2 as a background color; then conducting a following operation on all the pixels generated in step 3.2): acquiring a position of the pixel (x,y) and a corresponding depth value, and comparing with the depth value at the pixel (x,y) in the depth panorama D 2 ; if a former depth value is smaller, updating the depth value at (x,y) in the depth panorama D 2 and a color value at (x,y) in the RGB panorama C 2 and a normal panorama N 2 ; if a latter depth value is smaller, keeping unchanged; and after all updates are completed, obtaining the RGB panorama C 2 , the depth panorama D 2 and the normal panorama N 2 rendered under a new view v 2 ;
step 4: generating incomplete point cloud from the panorama under a specific view by:
4.1) generating point cloud {tilde over (P)} 0 from the depth panorama D 2 ;
4.2) calculating normal direction in the world coordinate system according to the normal panorama N 2 , and converting the normal direction in the world coordinate system to the camera coordinate system according to the camera extrinsic parameter corresponding to the view v 2 , wherein the normal panorama N 2 is rendered in the camera coordinate system corresponding to the view v 2 , but a color of the normal panorama records the normal direction in the world coordinate system;
4.3) in a process of 2D-3D rectangular projection, angle masks need to be calculated to locate a stripe area, so that a scene point cloud completion network completes a real occlusion area; calculating each point in the point cloud {tilde over (P)} 0 in the camera coordinate system; denoting a vector represented by a connecting line from an origin to a point in {tilde over (P)} 0 as {right arrow over (n)} 1 ; denoting a vector of a point in a row column order calculated from the normal panorama N 2 as {right arrow over (n)} 2 ; calculating an angle α between the vector {right arrow over (n)} 1 and the vector {right arrow over (n)} 2 ; then calculating difference values between the angle α and 90° to obtain absolute values; and filtering points with the absolute value of less than 15° as angle masks;
4.4) converting the point cloud {tilde over (P)} 0 in the camera coordinate system to the world coordinate system based on the camera extrinsic parameter corresponding to the view v 2 , and assigning the color information of the RGB panorama C 2 and the normal panorama N 2 to each point in the point cloud {tilde over (P)} 0 in the row column order of the pixel points to generate incomplete point cloud P 3 with RGB information and incomplete point cloud P 4 with normal information; and
step 5: constructing a self-supervised learning dataset by:
taking the incomplete point cloud P 3 with RGB information, the incomplete point cloud P 4 with normal information and the angle masks as input for the training of a scene point cloud completion network, wherein the targets of the scene point cloud completion network are incomplete point cloud P 1 with RGB information and incomplete point cloud P 2 with normal information.

Assignees

• Univ Dalian Tech

Inventors

• Yang Xin
• Li Tong
• Yin Baocai
• Zhang Zhaoxuan
• Wei Boyan
• Du Zhenjun

Classifications

• G06V20/647 Primary

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06T3/0062

  Physics · mapped topic

• G06T7/50

  Depth or shape recovery · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title