Generating shadows for placed objects in depth estimated scenes of two-dimensional images

Patent metadata

Field	Value
Publication number	US-12469194-B2
Application number	US-202318304113-A
Country	US
Kind code	B2
Filing date	Apr 20, 2023
Priority date	Oct 3, 2022
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images. The disclosed systems further use three-dimensional representations of two-dimensional images to customize focal points for the two-dimensional images.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
determining, by at least one processor, estimated three-dimensional characteristics of one or more background objects in a scene of a two-dimensional image; determining, by the at least one processor, a request to place a two-dimensional object at a selected position within the scene of the two-dimensional image; generating, in response to placing the two-dimensional object at the selected position according to the request, a proxy three-dimensional mesh of the two-dimensional object by generating a tessellation of the two-dimensional object according to estimated three-dimensional characteristics of the two-dimensional object at the selected position within the scene of the two-dimensional image; and generating, by the at least one processor, a modified two-dimensional image comprising one or more updated shadows according to the proxy three-dimensional mesh of the two-dimensional object at the selected position of the two-dimensional object and the estimated three-dimensional characteristics of the one or more background objects.

2 . The computer-implemented method of claim 1 , wherein determining the estimated three-dimensional characteristics of the one or more background objects comprises generating, utilizing one or more neural networks, a three-dimensional mesh for the two-dimensional image based on pixel depth values corresponding to one or more foreground objects and the one or more background objects of the scene of the two-dimensional image and estimated camera parameters of a camera position corresponding to the two-dimensional image.

3 . The computer-implemented method of claim 2 , wherein determining the estimated three-dimensional characteristics of the one or more background objects comprises:
generating an object segmentation map for one or more foreground objects and the one or more background objects of the scene of the two-dimensional image; and generating a plurality of separate three-dimensional meshes for the one or more foreground objects and the one or more background objects according to the object segmentation map.

4 . The computer-implemented method of claim 1 , wherein determining the request to place the two-dimensional object at the selected position comprises determining that the request comprises moving the two-dimensional object from a first position in the two-dimensional image to a second position in the two-dimensional image.

5 . The computer-implemented method of claim 4 , wherein determining the estimated three-dimensional characteristics of the one or more background objects comprises generating, according to a shape of the two-dimensional object, the proxy three-dimensional mesh at a three-dimensional position corresponding to the selected position within the scene of the two-dimensional image.

6 . The computer-implemented method of claim 5 , wherein generating the modified two-dimensional image comprises:
removing, from the two-dimensional image, a first shadow corresponding to the two-dimensional object at the first position in the two-dimensional image; and generating, utilizing the proxy three-dimensional mesh, a second shadow corresponding to the two-dimensional object at the second position in the two-dimensional image.

7 . The computer-implemented method of claim 5 , wherein generating the proxy three-dimensional mesh comprises:
determining a symmetric axis corresponding to the two-dimensional object according to features of a visible portion of the two-dimensional object within the two-dimensional image; and generating, based on the symmetric axis, a three-dimensional mesh comprising a first three-dimensional portion corresponding to the visible portion of the two-dimensional object and a mirrored three-dimensional portion of the first three-dimensional portion corresponding to a non-visible portion of the two-dimensional object.

8 . The computer-implemented method of claim 5 , wherein generating the proxy three-dimensional mesh comprises:
determining that the two-dimensional object corresponds to a predetermined subset of objects; and generating, utilizing a machine-learning model trained for the predetermined subset of objects, a three-dimensional mesh representing the two-dimensional object.

9 . The computer-implemented method of claim 1 , wherein generating the modified two-dimensional image comprises:
determining an object shadow map for the two-dimensional object at the selected position according to the proxy three-dimensional mesh of the two-dimensional object and an estimated shadow map according to one or more additional three-dimensional meshes representing the one or more background objects; and generating the modified two-dimensional image comprising a rendered shadow of the two-dimensional object at the selected position based on estimated camera parameters of the two-dimensional image, and estimated lighting parameters of the two-dimensional image by merging the object shadow map for the two-dimensional object and the estimated shadow map.

10 . A system comprising:
one or more memory devices comprising a two-dimensional image; and one or more processors configured to cause the system to:
generate, utilizing one or more neural networks, a three-dimensional mesh for the two-dimensional image based on pixel depth values corresponding to one or more background objects in a scene of the two-dimensional image;
determine, based on the three-dimensional mesh of the two-dimensional image and in response to placing a two-dimensional object at a selected position within the scene of the two-dimensional image according to a request, estimated three-dimensional characteristics of the two-dimensional object relative to the one or more background objects;
generate a proxy three-dimensional mesh of the two-dimensional object by generating a tessellation of the two-dimensional object according to the estimated three-dimensional characteristics of the two-dimensional object at the selected position within the scene of the two-dimensional image; and
generate a modified two-dimensional image comprising one or more updated shadows according to the proxy three-dimensional mesh of the two-dimensional object at the selected position of the two-dimensional object and the estimated three-dimensional characteristics of the two-dimensional object relative to the one or more background objects.

11 . The system of claim 10 , wherein the one or more processors are configured to generate the three-dimensional mesh for the two-dimensional image by:
determining an object segmentation for the one or more background objects in the scene of the two-dimensional image; generating one or more three-dimensional meshes representing the one or more background objects within a three-dimensional space; and generating the proxy three-dimensional mesh representing the two-dimensional object within the three-dimensional space.

12 . The system of claim 11 , wherein the one or more processors are configured to determine the estimated three-dimensional characteristics by:
determining a three-dimensional position based on the selected position within the scene of the two-dimensional image; and placing the proxy three-dimensional mesh representing the two-dimensional object within the three-dimensional space at the three-dimensional position.

13 . The system of claim 10 , wherein the one or more processors are configured to determine the estimated three-dimensional characteristics by:
determining that the two-dimensional object comprises a foreground object in the two-dimensional image; and generating the proxy three-dimensio

Assignees

• Adobe Inc

Inventors

• Hold-Geoffroy Yannick
• Krs Vojtech
• Mech Radomir
• Carr Nathan
• Gadelha Matheus

Classifications

• G06T15/60

  Shadow generation · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/68

  of symmetry · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title