Methods and systems for generating three dimensional (3D) models of objects

Patent metadata

Field	Value
Publication number	US-12154226-B2
Application number	US-202217948863-A
Country	US
Kind code	B2
Filing date	Sep 20, 2022
Priority date	May 24, 2021
Publication date	Nov 26, 2024
Grant date	Nov 26, 2024

Abstract

A method for generating a three-dimensional (3D) model of an object includes receiving a two-dimensional (2D) view of at least one object as an input, measuring geometrical shape coordinates of the at least one object from the input, identifying texture parameters of the at least one object from the input, predicting geometrical shape coordinates and texture parameters of occluded portions of the at least one object in the 2D view by processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object, and generating a 3D model of the at least one object by mapping the measured geometrical shape coordinates and the identified texture parameters to the predicted geometrical shape coordinates and the predicted texture parameters of the occluded portions of the at least one object.

First claim

What is claimed is:

1. A method for generating a three-dimensional (3D) model of an object, the method comprising:
receiving, by an electronic device, a two-dimensional (2D) view of at least one object as an input;
measuring, by the electronic device, geometrical shape coordinates of the at least one object from the input;
identifying, by the electronic device, texture parameters of the at least one object from the input;
predicting, by the electronic device, geometrical shape coordinates and texture parameters of occluded portions of the at least one object in the 2D view by processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object; and
generating, by the electronic device, a 3D model of the at least one object by mapping the measured geometrical shape coordinates and the identified texture parameters to the predicted geometrical shape coordinates and the predicted texture parameters of the occluded portions of the at least one object,
wherein the predicting of the geometrical shape coordinates and the texture parameters of the occluded portions of the at least one object in the 2D view comprises:
processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object by using a first module of a third neural network to predict normal maps for the at least one object;
processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object by using a second module of the third neural network to predict occluded portions normal maps for the at least one object from the predicted normal maps; and
processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object by using a third module of the third neural network to predict a texture map of the occluded portions of the at least one object using a training input,
wherein the normal maps and the occluded portions normal maps indicate the geometrical shape coordinates of the occluded potions of the at least one object at multiple scales, and the texture map indicates the texture parameters of the occluded potions of the at least one object.

2. The method of claim 1 , wherein the first, second and third modules of the third neural network include separate convolutional neural networks or generative adversarial networks,
wherein the first module is trained to predict the normal maps from the input,
wherein the second module is trained to predict the occluded portions normal maps from the predicted normal maps, and
wherein the third module is trained to predict the texture parameters of the occluded portions using the input.

3. The method of claim 1 , wherein the third neural network is trained in a plurality of epochs, and
wherein the training of the third neural network at each epoch of the plurality of epochs comprises:
sampling surface points and texture RGB values from a training input, wherein the training input includes the batches of 2D images;
identifying misclassified surface points from previous epochs and providing weightage to the misclassified surface points;
training the third neural network based on the identified misclassified surface points to predict the geometrical shape coordinates and the texture parameters of the occluded portions of each object present in the training input;
deriving a cost function to calculate an accuracy of the third neural network in predicting the geometrical shape coordinates and the texture parameters of the occluded portions of each object present in the training input, wherein the cost function is a weighted sum of losses computed with respect to the geometrical shape coordinates and the texture parameters; and
back propagating the cost function to the third neural network to train the third neural network iteratively.

4. The method of claim 1 , wherein a fourth neural network is used to detect the occluded portions of the at least one object, and
wherein the fourth neural network includes a Conditional Generative Adversarial Network (CGAN).

5. An electronic device comprising:
a processor; and
a memory storing instructions executable by the processor,
wherein the processor is coupled to the memory and configured to:
receive a two-dimensional (2D) view of at least one object as an input;
measure geometrical shape coordinates of the at least one object from the input;
identify texture parameters of the at least one object from the input;
predict geometrical shape coordinates and texture parameters of occluded portions of the at least one object in the 2D view by processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object; and

generate a three-dimensional (3D) model of the at least one object by mapping the measured geometrical shape coordinates and the identified texture parameters to the predicted geometrical shape coordinates and the predicted texture parameters of the occluded portions of the at least one object, wherein the processor is further configured to:
process the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object by using a first module of a third neural network to predict normal maps for the at least one object;
process the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object by using a second module of the third neural network to predict occluded portions normal maps for the at least one object from the predicted normal maps; and
process the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object by using a third module of the third neural network to predict a texture map of the occluded portions of the at least one object using a training input,
wherein the normal maps and the occluded portions normal maps indicate the geometrical shape coordinates of the occluded potions of the at least one object at multiple scales, and the texture map indicates the texture parameters of the occluded potions of the at least one object.

6. An electronic device comprising:
a processor; and
a memory storing instructions executable by the processor,
wherein the processor is coupled to the memory and configured to:
receive a two-dimensional (2D) view of at least one object as an input;
measure geometrical shape coordinates of the at least one object from the input;
identify texture parameters of the at least one object from the input;
predict geometrical shape coordinates and texture parameters of occluded portions of the at least one object in the 2D view by processing the measured geometrical shape coordinates of the at least one object, the identified texture parameters of the at least one object, and the occluded portions of the at least one object; and
generate a three-dimensional (3D) model of the at least one object by mapping the measured geometrical shape coordinates and the identified texture parameters to the predicted geometrical shape coordinates and the predicted texture parameters of the occluded portions of the at least one objec

Assignees

• Samsung Electronics Co Ltd

Inventors

• Saha Sujoy
• Ambi Mahantesh Mallappa
• Jain Rajat Kumar
• Badhwar Amita
• Singhal Aditi
• Abothula Rakesh
• Boregowda Lokesh Rayasandra

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title