3D motion effect from a 2D image

Patent metadata

Field	Value
Publication number	US-11017586-B2
Application number	US-201916388187-A
Country	US
Kind code	B2
Filing date	Apr 18, 2019
Priority date	Apr 18, 2019
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

Systems and methods are described for generating a three dimensional (3D) effect from a two dimensional (2D) image. The methods may include generating a depth map based on a 2D image, identifying a camera path, generating one or more extremal views based on the 2D image and the camera path, generating a global point cloud by inpainting occlusion gaps in the one or more extremal views, generating one or more intermediate views based on the global point cloud and the camera path, and combining the one or more extremal views and the one or more intermediate views to produce a 3D motion effect.

First claim

What is claimed is:

1. A method for generating a three dimensional (3D) motion effect, comprising:
identifying semantic information for a two dimensional (2D) image;
generating a first depth estimate based on the 2D image and the semantic information;
identifying image segmentation information for the 2D image;
generating a second depth estimate based on the first depth estimate and the image segmentation information; and
refining the second depth estimate based on a high resolution version of the 2D image,
identifying a camera path;
generating one or more extremal views based on the 2D image and the camera path;
generating a global point cloud by inpainting occlusion gaps in the one or more extremal views based on the second depth estimate, wherein the occlusion gaps are a result of warping the 2D image to generate the one or more extremal views;
generating one or more intermediate views based on the global point cloud and the camera path; and
combining the one or more extremal views and the one or more intermediate views to produce a 3D motion effect.

2. The method of claim 1 , further comprising:
extracting a feature map from a layer of a VGG-19 convolutional neural network (CNN), wherein the semantic information is identified based on the feature map.

3. The method of claim 1 , further comprising:
upsampling the feature map using a linear upsampling function to produce the semantic information.

4. The method of claim 1 , further comprising:
extracting object information using a mask regional convolutional neural network (R-CNN), wherein the image segmentation information is based on the object information.

5. The method of claim 1 , wherein:
the second depth estimate is refined using a CNN.

6. The method of claim 1 , further comprising:
identifying a point on the camera path, wherein the one or more extremal views are generated by warping the 2D image according to the point on the camera path, and the occlusion gaps are a result of warping the 2D image.

7. The method of claim 6 , wherein:
the inpainting comprises generating one or more additional points in the global cloud corresponding to the occlusion gaps.

8. The method of claim 7 , wherein:
the one or more additional points are generated using a CNN.

9. The method of claim 7 , wherein:
each point in the global point cloud comprises color information, position information, and depth information.

10. The method of claim 1 , wherein:
the camera path comprises a plurality of camera positions, wherein each of the plurality of camera positions comprises a center point and a camera rotation.

11. The method of claim 1 , further comprising:
selecting a number of intermediate views based at least in part on a target frame rate and a target video length of the 3D motion effect.

12. The method of claim 1 , wherein:
the 2D image comprises the only input for the 3D motion effect.

13. An apparatus for generating a three dimensional (3D) motion effect, comprising:
a processor and a memory storing instructions and in electronic communication with the processor, the processor being configured to execute the instructions to:
identify semantic information for a two dimensional (2D) image;
generate a first depth estimate based the 2D image and the semantic information;
identify image segmentation information for the 2D image;
generate a second depth estimate based on the first depth estimate and the image segmentation information;
refine the second depth estimate based on a high resolution version of the 2D image;
identify a camera path;
generate one or more extremal views based on the 2D image and the camera path;
generate a global point cloud by inpainting occlusion gaps in the one or more extremal views based at least in part on the refined second depth estimate, wherein the occlusion gaps are a result of warping the 2D image to generate the one or more extremal views;
generate one or more intermediate views based on the global point cloud and the camera path; and
combine the one or more extremal views and the one or more intermediate views to produce a 3D motion effect.

14. The apparatus of claim 13 , the processor being further configured to execute the instructions to:
extract a feature map from a layer of a VGG-19 convolutional neural network (CNN), wherein the semantic information is identified based on the feature map.

15. The apparatus of claim 13 , the processor being further configured to execute the instructions to:
upsample the feature map using a linear upsampling function to produce the semantic information.

16. The apparatus of claim 13 , the processor being further configured to execute the instructions to:
extract object information using a mask regional convolutional neural network (R-CNN), wherein the image segmentation information is based on the object information.

17. The apparatus of claim 13 , wherein:
the second depth estimate is refined using a CNN.

Assignees

• Adobe Inc

Inventors

• Long Mai
• Niklaus Simon
• Yang Jimei

Classifications

• G06T7/55 Primary

  from multiple images · CPC title

• G06V10/806

  of extracted features · CPC title

• G06V10/267

  by performing operations on regions, e.g. growing, shrinking or watersheds · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title