Three dimensional scene inpainting using stereo extraction

Patent metadata

Field	Value
Publication number	US-11094108-B2
Application number	US-201916559852-A
Country	US
Kind code	B2
Filing date	Sep 4, 2019
Priority date	Sep 27, 2018
Publication date	Aug 17, 2021
Grant date	Aug 17, 2021

Abstract

Systems and methods for rendering three-dimensional (3D) scenes having improved visual characteristics from a pair of 2D images having different viewpoints. The 3D scene is created by obtaining a first two-dimensional (2D) image of a scene object from a first viewpoint, obtaining a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint, creating a depth map from the first and second 2D images, creating a 3D scene from the depth map and the first and second 2D images, detecting regions of the initial 3D scene with incomplete image information, reconstructing the detected regions of the 3D scene, determining replacement information and modify the reconstructed regions, and rendering the 3D scene with the modified reconstructed regions from a plurality of viewpoints.

First claim

What is claimed:

1. A system for creating a three-dimensional (3D) scene, the system comprising:
eyewear including a first imager and a second imager spaced from the first imager, the first imager configured to obtain a first two-dimensional (2D) image of a scene object from a first viewpoint and the second imager configured to obtain a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint;
a processing system coupled to the eyewear, the processing system configured to:
obtain the first 2D image and the second 2D image;
create a depth map from the first and second 2D images;
create a 3D scene from the depth map and the first and second 2D images;
detect regions of the 3D scene with incomplete image information;
reconstruct the detected regions of the 3D scene;
determine replacement information and modify the reconstructed regions; and
render the 3D scene with the modified reconstructed regions from a plurality of viewpoints.

2. The system of claim 1 , wherein the processing system is further configured to:
identify holes in the rendered 3D scene from one or more viewpoints; and
refine the rendered 3D scene to fill in the holes.

3. The system of claim 1 , wherein the eyewear includes a first temple and a second temple and wherein the first imager is adjacent the first temple and the second imager is adjacent the second temple.

4. The system of claim 1 , wherein to determine replacement information for the detected regions with incomplete image information the processing system is configured to:
identify a boundary surrounding each detected region;
identify background information in the boundary surrounding each detected region;
identify foreground information in the boundary surrounding each detected region; and
blend the background and foreground information through each detected region giving the background information higher weight than the foreground information.

5. The system of claim 4 , wherein to blend the background and foreground boundary information, the processing system is configured to:
diffuse missing information from the background information through each detected region into the foreground information.

6. The system of claim 1 , wherein the depth map includes pixel vertices and corresponding image information from the first and second 2D images, to create the 3D scene the processing system is configured to connect the vertices to form first faces, and to reconstruct the detected regions the processing system is configured to connect the vertices of the boundary regions to form second faces, the second faces being different than the first faces.

7. The system of claim 6 , wherein the first and second faces include at least one of triangular faces or quadrangular faces.

8. The system of claim 1 , wherein the depth map includes pixel vertices and confidence values corresponding to each vertex, to create the 3D scene the processing system is configured to connect the vertices to form faces, and wherein to detect the regions of the 3D scene with incomplete information the processing system is configured to identify contiguous faces including a least one of degenerated faces or low confidence faces.

9. The system of claim 8 , wherein the degenerated faces each have at least one angle that is less than a threshold value.

10. The system of claim 8 , wherein each low confidence face includes at least one vertex generated with inconsistent values between the first 2D image and the second 2D image.

11. A method for creating a three-dimensional (3D) scene, the method comprising:
obtaining a first two-dimensional (2D) image of a scene object from a first viewpoint;
obtaining a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint;
creating a depth map from the first and second 2D images;
creating a 3D scene from the depth map and the first and second 2D images;
detecting regions of the 3D scene with incomplete image information;
reconstructing the detected regions of the 3D scene;
determining replacement information and modify the reconstructed regions; and
rendering the 3D scene with the modified reconstructed regions from a plurality of viewpoints.

12. The method of claim 11 , further comprising:
identifying holes in the rendered 3D scene from one or more viewpoints; and
refining the rendered 3D scene to fill in the holes.

13. The method of claim 11 , wherein the first 2D image is obtained from a first imager adjacent a first temple of eyewear and the second 2D image is obtained from a second imager adjacent a second temple of the eyewear.

14. The method of claim 11 , wherein the determining step comprises:
identifying a boundary surrounding each detected region;
identifying background information in the boundary surrounding each detected region;
identifying foreground information in the boundary surrounding each detected region; and
blending the background and foreground information through each detected region giving the background information higher weight than the foreground information.

15. The method of claim 14 , wherein the blending comprises:
diffusing missing information from the background information through each detected region into the foreground information.

16. The method of claim 11 , wherein the creating the 3D scene step comprises creating a depth map including pixel vertices and corresponding image information from the first and second 2D images and connecting the vertices to form first faces, and wherein the reconstructing step further includes:
connecting the vertices of the boundary regions to form second faces, wherein the second faces are different than the first faces.

17. The method of claim 16 , wherein the first and second faces include at least one of triangular faces or quadrangular faces.

18. The method of claim 11 , wherein the depth map includes pixel vertices and corresponding image information from the first and second 2D images and creating the 3D scene comprises connecting the vertices to form multi-angular faces, and wherein the detecting step comprises:
identifying contiguous faces including a least one of degenerated faces or low confidence faces.

19. The method of claim 18 , wherein the degenerated faces each have at least one angle that is less than a threshold.

20. The method of claim 18 , wherein each low confidence face includes at least one vertex generated with inconsistent values exceeding a threshold value between the first 2D image and the second 2D image.

Assignees

• Snap Inc

Inventors

• Daube Nir
• Karni Zachi

Classifications

• G06T5/77

  Retouching; Inpainting; Scratch removal · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/136

  involving thresholding · CPC title