Method for reconstructing a 3D object based on dynamic graph network

Patent metadata

Field	Value
Publication number	US-11715258-B2
Application number	US-202117243594-A
Country	US
Kind code	B2
Filing date	Apr 29, 2021
Priority date	Mar 5, 2021
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

The present invention provides a method for reconstructing a 3D object based on dynamic graph network, first, obtaining a plurality of feature vectors from 2D image I of an object; then, preparing input data: predefining an initial ellipsoid mesh, obtaining a feature input X by filling initial features and creating a relationship matrix A corresponding to the feature input X; then, inputting the feature input X and corresponding relationship matrix A to a dynamic graph network for integrating and deducing of each vertex's feature, thus new relationship matrix is obtained and used for the later graph convoluting, which improves the initial graph information and makes the initial graph information adapted to the mesh relation of the corresponding object, therefore the accuracy and the effect of 3D object reconstruction have been improved; last, regressing the position, thus the 3D structure of the object is deduced, and the 3D object reconstruction is completed.

First claim

What is claimed is:

1. A method for reconstructing a 3D object based on a dynamic graph network, comprising:
(1): image feature extraction, comprising:
extracting image features from a 2D (2 dimensional) image I of an object, wherein the extracted image features constitute a feature map, and the feature map comprises N feature vectors of D dimensions and N center coordinates of the image areas respectively corresponding to the N feature vectors, where N and D are natural numbers, N feature vectors are denoted by F n , n=1, 2, . . . , N, feature vector F n is a column vector, and the center coordinate of the image area corresponding to feature vector F n is denoted by (x n , y n ), the center coordinate (x n , y n ) is taken as the feature vector coordinate of feature vector F n , where x n is the horizontal coordinate of the feature vector coordinate, y n is the vertical coordinate of the feature vector coordinate;
(2): input data preparation for a dynamic graph network, comprising:—
predefining an initial ellipsoid mesh, which comprises N vertices and a plurality of edges, where the coordinate of the k th vertex is (x′ k ,y′ k ,z′ k ), x′ k ,y′ k ,z′ k respectively are the x coordinate, y coordinate and z coordinate of the k th vertex, where k=1, 2, . . . , N;
filling initial features: in the feature map, finding feature vector coordinate (x k′ ,y k′ ), k′∈{1, 2, . . . N}, which has the nearest distance from the coordinate (x′ k ,y′ k ) of the k th vertex, and combining feature vector F k′ and coordinate (x′ k ,y′ k ,z′ k ) of the k th vertex into a feature vector, which is denoted by X k , where the number of dimensions of feature vector X k is c 1 , c 1 =D+3, and a feature input X is obtained, X={X 1 , X 2 , . . . , X N };
creating a relationship matrix corresponding to the feature input X, which is denoted by A and A=(A ij ) N×N , i=1, 2, . . . , N, j=1, 2, . . . , N; wherein if there is an edge connecting the i th vertex and the j th vertex, or a neighborhood relationship exists between the i th vertex and the j th vertex, element A ij =1, otherwise, element A ij =0;
(3): feature mapping and convolution in a dynamic graph network, wherein
the dynamic graph network (dynamic graph convolutional neural network) comprises a dynamic graph learning layer and two graph convolution layers, comprising:
3.1): in the dynamic graph learning layer, first performing a feature mapping for feature input X by a learnable parameter θ, for the i th feature vector X i of feature input X, the feature vector h i is obtained: h i =θ T X i , where the learnable parameter θ is a matrix with size of c 1 ×c 2 , c 2 is a natural number; then measuring the distances between vertex θ T X i and vertex θ T X j , and obtaining a relationship matrix, which is denoted by S, S=(S ij ) N×N , element S ij of relationship matrix S is:

S
ij

=

exp
⁢

{

-

d
2

(

θ
T

⁢

X
i

,

θ
T

⁢

X
j

)

}

∑

n
=
1

N

exp
⁢

{

-

d
2

(

θ
T

⁢

X
i

,

θ
T

⁢

X
n

)

}

;

where d 2 ( ) is a distance measure function used to calculate the distance between two feature vectors, and exp{ } is an exponential function;
then normalizing relationship matrix S, where the normalized element S ij is:

S
_

ij

=

S
ij

∑

n
=
1

N

S
in

;

retaining the K largest elements of the N elements S i1 , S i2 , . . . , S iN of the i th row, and setting the remaining elements of the i th row to 0, thus a relationship matrix denoted by S is obtained by the retaining and setting of the N rows, where S =( S ij ) N×N , where K is a natural number less than N;
last, integrating relationship matrix S with relationship matrix A into new relationship matrix Â:
 =(1−η) A+η S ;

where η is a hyper-parameter used to balance relationship matrix A and relationship matrix S , η is a decimal less than 1;
3.2): in the two graph convolution layers, performing two graph convolutions for feature input X, thus a feature output denoted by

Assignees

• Univ Electronic Sci & Tech China

Inventors

• Kuang Ping
• Peng Liang
• Gu Xiaofeng

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title