Editing shadows in digital images utilizing machine learning models

Patent metadata

Field	Value
Publication number	US-2025336100-A1
Application number	US-202418651176-A
Country	US
Kind code	A1
Filing date	Apr 30, 2024
Priority date	Apr 30, 2024
Publication date	Oct 30, 2025
Grant date	—

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for editing shadows in digital images. In particular, in some embodiments, the disclosed systems determine, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light. In addition, in some embodiments, the disclosed systems generate, utilizing a lighting diffusion network, a diffused image from the digital image, the diffused image comprising smoothed shading. Moreover, in some embodiments, the disclosed systems generate, utilizing a shadow synthesis network, a shadowed image from the diffused image and a modified environment map comprising a modified dominant light. Furthermore, in some embodiments, the disclosed systems generate, from the diffused image and the shadowed image, a modified digital image comprising an edited shadow.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
determining, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light; generating, utilizing a lighting diffusion network, a diffused image from the digital image, the diffused image comprising smoothed shading; generating, utilizing a shadow synthesis network, a shadowed image from the diffused image and a modified environment map comprising a modified dominant light; and generating, from the diffused image and the shadowed image, a modified digital image comprising an edited shadow.

2 . The computer-implemented method of claim 1 , wherein determining the environment map comprises determining at least one of a position, a size, or an intensity of the dominant light.

3 . The computer-implemented method of claim 1 , further comprising modeling the dominant light as a two-dimensional isotropic Gaussian light source.

4 . The computer-implemented method of claim 1 , wherein generating the diffused image comprises removing hard shadows and specular highlights of the digital image.

5 . The computer-implemented method of claim 1 , further comprising generating the modified environment map by changing at least a position, a size, or an intensity of the dominant light of the environment map.

6 . The computer-implemented method of claim 1 , wherein generating the shadowed image comprises applying the modified dominant light to the diffused image.

7 . The computer-implemented method of claim 1 , wherein generating the modified digital image comprises compositing the diffused image and the shadowed image as a weighted combination.

8 . A system comprising:
one or more memory devices; and one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
determining, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light;
generating a diffused image from the digital image by utilizing a lighting diffusion network to remove hard shadows and specular highlights of the digital image;
generating a shadowed image from the diffused image and a modified environment map by utilizing a shadow synthesis network to apply a modified dominant light to the diffused image; and
generating a modified digital image comprising an edited shadow for the digital image by compositing the diffused image and the shadowed image.

9 . The system of claim 8 , wherein determining the environment map comprises determining a three-dimensional intensity of the dominant light.

10 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
providing, for display via a user interface of a client device, a lighting control element; and determining, based on a user interaction with the lighting control element, at least one of a position, a size, or an intensity of the modified dominant light for the modified environment map.

11 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining the modified environment map comprising the modified dominant light and a new dominant light, wherein generating the shadowed image from the diffused image and the modified environment map comprises utilizing the shadow synthesis network to apply the modified dominant light and the new dominant light to the diffused image.

12 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining a measure of lighting estimation loss based on a difference between the environment map and a ground truth environment map; and modifying parameters of the lighting estimation network based on the measure of lighting estimation loss.

13 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining a measure of lighting diffusion loss based on a difference between the diffused image and a ground truth diffused image; and modifying parameters of the lighting diffusion network based on the measure of lighting diffusion loss.

14 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining a measure of shadow synthesis loss based on a difference between the shadowed image and a ground truth shadowed image; and modifying parameters of the shadow synthesis network based on the measure of shadow synthesis loss.

15 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
determining, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light; generating, utilizing a lighting diffusion network, a diffused image from the digital image, the diffused image comprising smoothed shading; generating, utilizing a shadow synthesis network, a shadowed image from the diffused image and a modified environment map comprising a modified dominant light; and generating, from the diffused image and the shadowed image, a modified digital image comprising an edited shadow.

16 . The non-transitory computer-readable medium of claim 15 , wherein determining the environment map for the digital image comprises converting nonparametric information for the dominant light into position, size, and intensity parameters for the dominant light.

17 . The non-transitory computer-readable medium of claim 15 , wherein:
generating the diffused image from the digital image comprises generating an ambient-lighted image, and generating the shadowed image from the diffused image and the modified environment map comprises generating a dominant-lighted image.

18 . The non-transitory computer-readable medium of claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to perform additional operations comprising:
generating a lighting data set that correlates facial shadows to parameters of dominant lights; determining ground truth environment maps comprising the dominant lights of the lighting data set; and modifying parameters of the lighting estimation network based on the ground truth environment maps.

19 . The non-transitory computer-readable medium of claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to perform additional operations comprising:
receiving a user interaction with a lighting control element via a user interface of a client device; and determining, based on the user interaction with the lighting control element, a change to at least one of a position, a size, or an intensity of the dominant light for the modified environment map.

20 . The non-transitory computer-readable medium of claim 15 , wherein the digital image comprises a portrait, and wherein generating the modified digital image comprises generating an updated portrait with the edited shadow.

Assignees

• Adobe Inc

Inventors

• Shu Zhixin
• Hou Andrew
• Zhang He
• Zhang Xuaner
• Hold-Geoffroy Yannick
• Yoon Jae Shin

Classifications

• G06T2211/441

  AI-based methods, deep learning or artificial neural networks · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title