Modifying two-dimensional images utilizing three-dimensional meshes of the two-dimensional images

Patent metadata

Field	Value
Publication number	US-12367626-B2
Application number	US-202218055584-A
Country	US
Kind code	B2
Filing date	Nov 15, 2022
Priority date	Nov 15, 2022
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for generating three-dimensional meshes representing two-dimensional images for editing the two-dimensional images. The disclosed system utilizes a first neural network to determine density values of pixels of a two-dimensional image based on estimated disparity. The disclosed system samples points in the two-dimensional image according to the density values and generates a tessellation based on the sampled points. The disclosed system utilizes a second neural network to estimate camera parameters and modify the three-dimensional mesh based on the estimated camera parameters of the pixels of the two-dimensional image. In one or more additional embodiments, the disclosed system generates a three-dimensional mesh to modify a two-dimensional image according to a displacement input. Specifically, the disclosed system maps the three-dimensional mesh to the two-dimensional image, modifies the three-dimensional mesh in response to a displacement input, and updates the two-dimensional image.

First claim

What is claimed is:

1. A method comprising:
determining, by at least one processor utilizing one or more neural networks, pixel depth values representing estimated depths for pixels of a two-dimensional image;
generating, by the at least one processor utilizing the one or more neural networks, a three-dimensional mesh by incorporating depth information from the pixel depth values representing the estimated depths at the pixels of the two-dimensional image into a tessellation of the two-dimensional image according to camera parameters extracted from the two-dimensional image;
detecting, by the at least one processor, a displacement input at a two-dimensional position on the two-dimensional image;
modifying, by the at least one processor in response to the displacement input at the two-dimensional position on the two-dimensional image, the three-dimensional mesh by determining a displaced portion of the three-dimensional mesh according to a mapping of the two-dimensional position of the displacement input to a three-dimensional position of the three-dimensional mesh; and
generating, by the at least one processor, a modified two-dimensional image comprising a displacement of a portion of the two-dimensional image by re-rendering the two-dimensional image according to the displaced portion of the three-dimensional mesh and the camera parameters extracted from the two-dimensional image.

2. The method of claim 1 , wherein:
detecting the displacement input comprises determining the two-dimensional position of the displacement input within the two-dimensional image comprising two-dimensional coordinates of the displacement input relative to the two-dimensional image; and
modifying the three-dimensional mesh comprises determining three-dimensional coordinates of the displacement input relative to the three-dimensional mesh based on the mapping of the two-dimensional coordinates of the displacement input to the three-dimensional position according to the camera parameters extracted from the two-dimensional image.

3. The method of claim 1 , wherein modifying the three-dimensional mesh comprises:
determining, based on an attribute of the displacement input, that the displacement input indicates a displacement direction for a portion of the three-dimensional mesh; and
displacing the displaced portion of the three-dimensional mesh in the displacement direction.

4. The method of claim 3 , wherein modifying the three-dimensional mesh comprises:
determining, based on an additional attribute of the displacement input, that the displacement input indicates an additional displacement direction of the portion of the three-dimensional mesh; and
displacing the displaced portion of the three-dimensional mesh according to the additional displacement direction.

5. The method of claim 4 , wherein modifying the three-dimensional mesh comprises:
selecting, in response to an additional input in connection with the displacement input, a new portion of the three-dimensional mesh to displace according to a mapping of an additional two-dimensional position of the additional input to an additional three-dimensional position of the three-dimensional mesh; and
displacing the new portion of the three-dimensional mesh according to the displacement input in the displacement direction.

6. The method of claim 5 , wherein modifying the three-dimensional mesh comprises displacing, based on movement of the displacement input, one or more additional portions of the three-dimensional mesh from the portion of the three-dimensional mesh to the new portion of the three-dimensional mesh in the displacement direction.

7. The method of claim 1 , wherein:
generating the three-dimensional mesh comprises generating a mapping of the pixels of the two-dimensional image to three-dimensional coordinates of the three-dimensional mesh based on the pixel depth values and the camera parameters extracted from the two-dimensional image; and
modifying the three-dimensional mesh comprises determining the mapping of the two-dimensional position of the displacement input to the three-dimensional position based on the mapping of the pixels of the two-dimensional image to the three-dimensional coordinates of the three-dimensional mesh.

8. The method of claim 1 , wherein generating the modified two-dimensional image comprises:
providing a preview two-dimensional image comprising the displacement of the portion of the two-dimensional image in response to the displacement input;
providing, for display within a graphical user interface, an option to commit the displacement of the portion of the two-dimensional image; and
generating the modified two-dimensional image comprising the displacement of the portion of the two-dimensional image in response to detecting a selection of the option to commit the displacement of the portion of the two-dimensional image.

9. The method of claim 1 , wherein modifying the three-dimensional mesh comprises:
determining a displacement filter indicating a shape associated with the displacement input; and
displacing the displaced portion of the three-dimensional mesh according to the shape of the displacement input in a direction of the displacement input.

10. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
determining, utilizing one or more neural networks, pixel depth values representing estimated depths for pixels of a two-dimensional image based on objects of the two-dimensional image;
generating, utilizing the one or more neural networks, a three-dimensional mesh by incorporating depth information from the pixel depth values corresponding to the objects of the two-dimensional image into a tessellation of the two-dimensional image according to camera parameters extracted from the two-dimensional image;
determining a three-dimensional position of the three-dimensional mesh based on a corresponding position of a displacement input at a two-dimensional position on the two-dimensional image according to a mapping of the two-dimensional image to the three-dimensional mesh;
modifying, in response to the displacement input at the two-dimensional position on the two-dimensional image, the three-dimensional mesh by determining a displaced portion of the three-dimensional mesh at the three-dimensional position of the three-dimensional mesh; and
generating a modified two-dimensional image comprising a displacement of a portion of the two-dimensional image by re-rendering the two-dimensional image according to the displaced portion of the three-dimensional mesh at the three-dimensional position of the three-dimensional mesh and the camera parameters extracted from the two-dimensional image.

11. The system of claim 10 , wherein determining the three-dimensional position of the three-dimensional mesh comprises:
determining the two-dimensional position of the displacement input comprises determining two-dimensional coordinates of the displacement input relative to the two-dimensional image; and
determining the three-dimensional position of the three-dimensional mesh based on the two-dimensional coordinates of the displacement input relative to the two-dimensional image and a projection from the two-dimensional image onto the three-dimensional mesh corresponding to the mapping of the two-dimensional image to the three-dimensional mesh.

12. The system of claim 11 , wherein modifying the three-dimensional mesh comprises:
determining that the displacement input indicates a displacement of a portion of the three-dimensional mesh according to the three-dimensional position of the three-dimensional m

Assignees

• Adobe Inc

Inventors

• Mech Radomir
• Carr Nathan
• Gadelha Matheus

Classifications

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title