Methods, devices and computer program products for generating 3d models

Patent metadata

Field	Value
Publication number	US-2020349754-A1
Application number	US-201816962665-A
Country	US
Kind code	A1
Filing date	Feb 23, 2018
Priority date	Feb 23, 2018
Publication date	Nov 5, 2020
Grant date	—

Abstract

A method of generating a 3D model may include receiving a plurality of 2D images of a physical object captured from a respective plurality of viewpoints in a 3D scan of the physical object in a first process. The method may include receiving a first process 3D mesh representation of the physical object and calculating respective second process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images. The method may include generating a second process 3D mesh representation of the physical object using the plurality of 2D images, the second process estimated position and/or orientation information, and the first process 3D mesh representation of the physical object. The method may include generating a 3D model of the physical object by applying surface texture information from the plurality of 2D images to the second process 3D mesh representation of the physical object.

First claim

1 . A method comprising:
receiving a plurality of two-dimensional (2D) images of a physical object captured from a respective plurality of viewpoints in a three-dimensional (3D) scan of the physical object in a first process; receiving a first process 3D mesh representation of the physical object; calculating respective second process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images; generating a second process 3D mesh representation of the physical object using the plurality of 2D images, the second process estimated position and/or orientation information, and the first process 3D mesh representation of the physical object; and generating a 3D model of the physical object by applying surface texture information from the plurality of 2D images to the second process 3D mesh representation of the physical object.

2 . The method of claim 1 , wherein the calculating the second process estimated position and/or orientation information and generating the second process 3D mesh representation are iteratively repeated.

3 . The method of claim 1 , further comprising:
receiving camera parameters corresponding to the plurality of 2D images; and calculating updated camera parameters based on the plurality of 2D images.

4 . The method of claim 1 , further comprising:
determining an initial depth map using the first process 3D mesh representation from a perspective of a reference one of the plurality of 2D images; selecting one or more surrounding ones of the plurality of 2D images based on the second process estimated position and/or orientation information; warping the one or more surrounding ones of the plurality of 2D images to appear from the perspective of the reference one of the plurality of 2D images; and calculating an updated depth map based on a combination of the reference one of the plurality of 2D images and the warped surrounding ones of the plurality of 2D images.

5 . The method of claim 1 , further comprising:
calculating surface normal vectors corresponding to positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images, wherein the generating the second process 3D mesh representation of the physical object is based on the calculated surface normal vectors.

6 . The method of claim 1 , further comprising:
receiving respective first process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images; and receiving a feature point map comprising positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images, wherein the calculating of the respective second process estimated position and/or orientation information comprises calculating the respective second process estimated position and/or orientation information for each of the respective plurality of viewpoints of the plurality of 2D images using the plurality of 2D images, the first process estimated position and/or orientation information, and the feature point map.

7 . The method of claim 6 , wherein the calculating of the respective second process estimated position and/or orientation information comprises:
calculating an estimated location of a feature point of the plurality of feature points within a 2D image of the plurality of 2D images based on the first process estimated position and/or orientation information; identifying the feature point within the 2D image to determine an actual location of the feature point within the 2D image; determining a displacement between the estimated location of the feature point within the 2D image and the actual location of the feature point within the 2D image; and calculating a second process estimated position and/or orientation corresponding to the 2D image that yields a smaller displacement than the displacement determined using the estimated location based on the first process estimated position and/or orientation information.

8 . The method of claim 7 , wherein the calculating of the respective second process estimated position and/or orientation information further comprises minimizing a sum of displacements between a plurality of estimated locations of feature points of the plurality of feature points within ones of the plurality of 2D images.

9 . The method of claim 1 , wherein the first process comprises:
capturing the plurality of 2D images using at least one camera; calculating first process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images; calculating a feature point map comprising positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images; and generating the first process 3D mesh representation of the physical object.

10 . The method of claim 9 , wherein the first process 3D mesh representation of the physical object is generated while capturing the plurality of 2D images.

11 . The method of claim 9 ,
wherein the 3D model is a second process 3D model, and wherein the first process further comprises generating a first process 3D model of the physical object by applying surface texture information from the plurality of 2D images to the first process 3D mesh representation of the physical object.

12 . A computer program product comprising a non-transitory computer readable storage medium having computer readable program code embodied in the medium that when executed by a processor causes the processor to perform the method of claim 1 .

13 . A processing device, the processing device configured to perform the method of claim 1 .

14 . A method, performed by an electronic device, comprising:
capturing a plurality of two-dimensional (2D) images of a physical object captured from a respective plurality of viewpoints in a three-dimensional (3D) scan of the physical object using at least one camera; calculating first process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images; calculating a feature point map comprising positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images; generating the first process 3D mesh representation of the physical object; receiving a first process 3D model of the physical object by applying surface texture information from the plurality of 2D images to the first process 3D mesh representation of the physical object; displaying the first process 3D model on a display of the electronic device; providing an option to a user of the electronic device to further process the first process 3D model; and receiving a second process 3D model responsive to receiving instruction from the user to further process the first process 3D model.

15 . The method of claim 14 , wherein the receiving of the second process 3D model comprises:
calculating respective second process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images; generating a second process 3D mesh representation of the physical object using the plurality of 2D images, the second process estimated position and/or orientation information, and the first process 3D mesh representation of the physical object; and generating the second process 3D model of the physical object by applying surface texture information from the plurality of 2D images to the second process 3D mesh

Assignees

• Sony Corp

Inventors

• Michielin Francesco
• Novak Lars
• Mattisson Fredrik

Classifications

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/579

  from motion · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/30201

  Face · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title