Inserting three-dimensional objects into digital images with consistent lighting via global and local lighting information

Patent metadata

Field	Value
Publication number	US-12561896-B2
Application number	US-202318354619-A
Country	US
Kind code	B2
Filing date	Jul 18, 2023
Priority date	Jul 22, 2021
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

This disclosure describes methods, non-transitory computer readable storage media, and systems that generate realistic shading for three-dimensional objects inserted into digital images. The disclosed system utilizes a light encoder neural network to generate a representation embedding of lighting in a digital image. Additionally, the disclosed system determines points of the three-dimensional object visible within a camera view. The disclosed system generates a self-occlusion map for the digital three-dimensional object by determining whether fixed sets of rays uniformly sampled from the points intersects with the digital three-dimensional object. The disclosed system utilizes a generator neural network to determine a shading map for the digital three-dimensional object based on the representation embedding of lighting in the digital image and the self-occlusion map. Additionally, the disclosed system generates a modified digital image with the three-dimensional object inserted into the digital image with consistent lighting of the three-dimensional object and the digital image.

First claim

What is claimed is:

1 . A method comprising:
extracting lighting parameters representing lighting from at least one light source generating light and shadows in an initial digital image utilizing a light encoder neural network; generating a self-occlusion map for a digital three-dimensional object indicating whether a plurality of sampled rays originating from each of a plurality of points on the digital three-dimensional object will intersect with the digital three-dimensional object; generating, utilizing a generator neural network, a shading map for the digital three-dimensional object from the lighting parameters and the self-occlusion map; and generating, based on the shading map, a modified digital image comprising the digital three-dimensional object with lighting consistent with the initial digital image inserted into the initial digital image.

2 . The method of claim 1 , wherein generating the self-occlusion map for the digital three-dimensional object indicating whether the plurality of sampled rays will intersect with the digital three-dimensional object comprises:
determining the plurality of points on the digital three-dimensional object visible within a field of view based on a camera position in a three-dimensional space; and generating, for a plurality of uniformly sampled rays from a point of the plurality of points, a vector comprising a plurality of intersection values, an intersection value of the plurality of intersection values indicating whether a sampled ray of the plurality of uniformly sampled rays intersects with the digital three-dimensional object.

3 . The method as recited in claim 2 , wherein determining the plurality of points on the digital three-dimensional object comprises determining one or more portions of the digital three-dimensional object visible within the field of view of a camera view based on the camera position in the three-dimensional space.

4 . The method as recited in claim 2 , wherein generating the vector comprises:
determining, for each point of the plurality of points, a fixed set of rays sampled uniformly at equal angle intervals from the point; and determining whether the fixed set of rays intersect with one or more portions of the digital three-dimensional object.

5 . The method as recited in claim 1 , wherein generating the self-occlusion map comprises generating a plurality of vectors for the plurality of points, each vector of the plurality of vectors comprising a plurality of intersection values indicating whether a plurality of sets of rays sampled from the plurality of points intersect with at least a portion of the digital three-dimensional object.

6 . The method as recited in claim 1 , further comprising:
extracting a normal map and an albedo map of the digital three-dimensional object rendered from a camera view based on a camera position in a three-dimensional space; and determining, utilizing the generator neural network, the shading map for the digital three-dimensional object based on the lighting parameters, the self-occlusion map, the normal map, and the albedo map.

7 . The method as recited in claim 1 , wherein extracting the lighting parameters from the initial digital image utilizing the light encoder neural network comprises predicting a light representation embedding for lighting in the initial digital image utilizing the light encoder neural network comprising parameters learned utilizing contrastive learning based on a plurality of configuration parameters and a plurality of predicted light representation embeddings for a plurality of digital background images.

8 . The method as recited in claim 7 , further comprising:
generating, utilizing a first light encoder, a first light representation embedding based on the plurality of configuration parameters comprising image-based lighting parameters and camera parameters stored for a digital background image of the plurality of digital background images; generating, utilizing a second light encoder, a second light representation embedding predicted from the digital background image of the plurality of digital background images; determining a contrastive loss based on a difference between the first light representation embedding and the second light representation embedding; and learning the parameters of the light encoder neural network based on the contrastive loss.

9 . The method as recited in claim 1 , wherein generating the modified digital image comprises:
generating a digital mask based on the digital three-dimensional object based on a camera view corresponding to a camera position; and inserting the digital three-dimensional object into the initial digital image with lighting consistent to the lighting in the initial digital image based on the digital mask, the shading map based on the self-occlusion map, and an albedo map for the digital three-dimensional object.

10 . A system comprising:
a memory component; and one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising:
extracting lighting parameters representing lighting from at least one light source generating light and shadows in an initial digital image utilizing a light encoder neural network;
generating a self-occlusion map for a digital three-dimensional object indicating whether a plurality of sampled rays originating from each of a plurality of points on the digital three-dimensional object will intersect with the digital three-dimensional object;
generating, utilizing a generator neural network, a shading map for the digital three-dimensional object from the lighting parameters and the self-occlusion map; and
generating, based on the shading map, a modified digital image comprising the digital three-dimensional object with lighting consistent with the initial digital image inserted into the initial digital image.

11 . The system of claim 10 , wherein generating the self-occlusion map for the digital three-dimensional object indicating whether the plurality of sampled rays will intersect with the digital three-dimensional object comprises:
determining the plurality of points on the digital three-dimensional object visible within a field of view based on a camera position in a three-dimensional space; and generating, for a plurality of uniformly sampled rays from a point of the plurality of points, a vector comprising a plurality of intersection values, an intersection value of the plurality of intersection values indicating whether a sampled ray of the plurality of uniformly sampled rays intersects with the digital three-dimensional object.

12 . The system of claim 11 , wherein determining the plurality of points on the digital three-dimensional object visible within the field of view comprises:
determining one or more visible surfaces of the digital three-dimensional object by ray-marching from the camera position to the digital three-dimensional object in a three-dimensional space; and determining the plurality of points on the one or more visible surfaces of the digital three-dimensional object.

13 . The system of claim 11 , wherein generating the self-occlusion map comprises:
determining the plurality of sampled rays by sampling rays at equal angle intervals from each point of the plurality of points; determining whether the rays intersect with one or more surfaces of the digital three-dimensional object; and generating a vector including the plurality of intersection values in response to determining whether the rays intersect with the one or more surfaces of the digital three-dimensional object.

14 . The s

Assignees

• Adobe Inc

Inventors

• Villegas Ruben
• Jang Yunseok
• Aksit Duygu Ceylan
• Yang Jimei
• Sun Xin

Classifications

• G06T15/20

  Perspective computation · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T15/06

  Ray-tracing · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title