Modifying two-dimensional images utilizing segmented three-dimensional object meshes of the two-dimensional images

Patent metadata

Field	Value
Publication number	US-12277652-B2
Application number	US-202218055585-A
Country	US
Kind code	B2
Filing date	Nov 15, 2022
Priority date	Nov 15, 2022
Publication date	Apr 15, 2025
Grant date	Apr 15, 2025

Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for generating three-dimensional meshes representing two-dimensional images for editing the two-dimensional images. The disclosed system utilizes a first neural network to determine density values of pixels of a two-dimensional image based on estimated disparity. The disclosed system samples points in the two-dimensional image according to the density values and generates a tessellation based on the sampled points. The disclosed system utilizes a second neural network to estimate camera parameters and modify the three-dimensional mesh based on the estimated camera parameters of the pixels of the two-dimensional image. In one or more additional embodiments, the disclosed system generates a three-dimensional mesh to modify a two-dimensional image according to a displacement input. Specifically, the disclosed system maps the three-dimensional mesh to the two-dimensional image, modifies the three-dimensional mesh in response to a displacement input, and updates the two-dimensional image.

First claim

What is claimed is:

1. A method comprising:
generating, by at least one processor utilizing one or more neural networks, a three-dimensional mesh by determining displacement of vertices of a tessellation of a two-dimensional image based on pixel depth values of the two-dimensional image;
segmenting, by the at least one processor utilizing the one or more neural networks, the three-dimensional mesh into a plurality of three-dimensional object meshes corresponding to a plurality of distinct objects of the two-dimensional image;
modifying, by the at least one processor in response to a displacement input to the two-dimensional image within a graphical user interface displaying the two-dimensional image, a selected three-dimensional object mesh of the plurality of three-dimensional object meshes based on a displaced portion of the selected three-dimensional object mesh corresponding to the displacement input to the two-dimensional image; and
generating, by the at least one processor, a modified two-dimensional image comprising at least one modified portion according to the displaced portion of the selected three-dimensional object mesh.

2. The method of claim 1 , wherein generating the three-dimensional mesh comprises:
generating the three-dimensional mesh by determining the displacement of the vertices of the tessellation of the two-dimensional image based on the pixel depth values and estimated camera parameters; or
generating the three-dimensional mesh based on a plurality of points sampled in the two-dimensional image according to density values determined from the pixel depth values of the two-dimensional image.

3. The method of claim 1 , wherein segmenting the three-dimensional mesh comprises:
detecting one or more objects in the two-dimensional image or in the three-dimensional mesh; and
separating a first portion of the three-dimensional mesh from a second portion of the three-dimensional mesh based on the one or more objects detected in the two-dimensional image or in the three-dimensional mesh.

4. The method of claim 3 , wherein segmenting the three-dimensional mesh comprises:
determining a semantic map comprising labels indicating object classifications of pixels in the two-dimensional image; and
detecting the one or more objects in the two-dimensional image based on the labels of the semantic map.

5. The method of claim 3 , wherein segmenting the three-dimensional mesh comprises:
determining a depth discontinuity at a portion of the three-dimensional mesh based on corresponding pixel depth values of the two-dimensional image; and
detecting the one or more objects in the three-dimensional mesh based on the depth discontinuity at the portion of the three-dimensional mesh.

6. The method of claim 1 , wherein modifying the selected three-dimensional object mesh comprises:
determining a projection from the two-dimensional image onto the plurality of three-dimensional object meshes in a three-dimensional environment; and
determining the selected three-dimensional object mesh based on a two-dimensional position of the displacement input relative to the two-dimensional image and the projection from the two-dimensional image onto the plurality of three-dimensional object meshes.

7. The method of claim 1 , wherein modifying the selected three-dimensional object mesh comprises:
determining that the displacement input indicates a displacement direction for a portion of the selected three-dimensional object mesh; and
modifying a portion of the selected three-dimensional object mesh by displacing the portion of the selected three-dimensional object mesh according to the displacement direction.

8. The method of claim 1 , wherein generating the modified two-dimensional image comprises:
determining a two-dimensional position of the two-dimensional image corresponding to a three-dimensional position of the displaced portion of the selected three-dimensional object mesh based on a mapping between the plurality of three-dimensional object meshes and the two-dimensional image; and
generating the modified two-dimensional image comprising the at least one modified portion at the two-dimensional position based on the three-dimensional position of the displaced portion of the selected three-dimensional object mesh.

9. The method of claim 1 , wherein modifying the selected three-dimensional object mesh comprises modifying the selected three-dimensional object mesh according to the displacement input without modifying one or more additional three-dimensional object meshes adjacent to the selected three-dimensional object mesh within a three-dimensional environment.

10. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
generating, utilizing one or more neural networks, a three-dimensional mesh by determining displacement of vertices of a tessellation of a two-dimensional image based on pixel depth values of the two-dimensional image;
detecting, utilizing one or more object detection models, a plurality of distinct objects of the two-dimensional image;

segmenting, in response to detecting the plurality of distinct objects, the three-dimensional mesh into a plurality of three-dimensional object meshes corresponding to the plurality of distinct objects of the two-dimensional image;
modifying, in response to a displacement input to the two-dimensional image within a graphical user interface displaying the two-dimensional image, a selected three-dimensional object mesh of the plurality of three-dimensional object meshes based on a displaced portion of the selected three-dimensional object mesh corresponding to the displacement input to the two-dimensional image; and
generating a modified two-dimensional image comprising at least one modified portion according to the displaced portion of the selected three-dimensional object mesh.

11. The system of claim 10 , wherein detecting the plurality of distinct objects comprises:
generating, utilizing the one or more object detection models, a semantic map comprising labels indicating object classifications of pixels in the two-dimensional image; and
detecting the plurality of distinct objects of the two-dimensional image based on the labels of the semantic map.

12. The system of claim 10 , wherein detecting the plurality of distinct objects comprises:
determining, based on the pixel depth values of the two-dimensional image, a portion of the two-dimensional image comprising a depth discontinuity between adjacent regions of the two-dimensional image; and
determining that a first region of the adjacent regions corresponds to a first object and a second region of the adjacent regions corresponds to a second object.

13. The system of claim 10 , wherein modifying the selected three-dimensional object mesh comprises:
determining a two-dimensional position of the displacement input relative to the two-dimensional image; and
determining a three-dimensional position corresponding to a three-dimensional object mesh of the plurality of three-dimensional object meshes based on a mapping between the two-dimensional image and a three-dimensional environment comprising the plurality of three-dimensional object meshes.

14. The system of claim 10 , wherein modifying the selected three-dimensional object mesh comprises:
determining, based on an attribute of the displacement input, that the displacement input indicates one or more displacement directions for a portion of the selected three-dimensional object mesh; and
displacing the portion of the selected three-dimensional object mesh in the

Assignees

• Adobe Inc

Inventors

• Mech Radomir
• Carr Nathan
• Gadelha Matheus

Classifications

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20228

  Disparity calculation for image-based rendering · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title