Systems and methods for modifying lighting in three-dimensional models

Patent metadata

Field	Value
Publication number	US-11847736-B2
Application number	US-202117237372-A
Country	US
Kind code	B2
Filing date	Apr 22, 2021
Priority date	Apr 22, 2021
Publication date	Dec 19, 2023
Grant date	Dec 19, 2023

Abstract

The consistent use of lighting in different instances of digital media may help ensure that objects are depicted in a similar manner in the digital media. However, in some cases, a three-dimensional (3D) model may depict an object under lighting conditions that differ from the lighting conditions depicted in other digital media. The present disclosure provides systems and methods for generating 3D models to include lighting that is consistent with the lighting used in other digital media. According to an embodiment, a lighting template is determined based on digital media depicting a first physical object. A modified 3D model of a second physical object is then generated based on the lighting template to light the second physical object according to the lighting template.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
determining, based on previously captured digital media depicting a real-world first physical object as lit by first lighting, a lighting template representing the first lighting, wherein determining the lighting template includes determining a position of a light source relative to the real-world first physical object as depicted in the digital media and further includes determining an environment map for an environment containing the real-world first physical object;
obtaining a three-dimensional (3D) model that represents the real-world first physical object that is depicted in the digital media;
obtaining a plurality of other 3D models that represent other real-world physical objects, wherein the other real-world physical objects are related to the real-world first physical object;
normalizing second lighting of the 3D model to obtain normalized lighting;
normalizing a plurality of lightings of the plurality of other 3D models, each of the plurality of lightings corresponding to a respective one of the plurality of other 3D models, to obtain the normalized lighting; and
applying the lighting template to the 3D model that represents the real-world first physical object and to each one of the plurality of other 3D models so that the 3D model that represents the real-world first physical object and the plurality of other 3D model are all depicted with consistent lighting, wherein the applying the lighting template includes:
generating, based on both (i) the 3D model that represents the real-world first physical object depicted in the digital media and (ii) the lighting template representing the first lighting in the digital media depicting the real-world first physical object, a modified 3D model that represents the real-world first physical object depicted in the digital media lit according to the lighting template, wherein generating the modified 3D model includes modifying the 3D model to have a virtual light source at a virtual position corresponding to the position of the light source relative to the real-world first physical object as depicted in the digital media, applying the environment map to the 3D model, and modifying the normalized lighting based on the lighting template; and
generating, based on both (i) the plurality of other 3D models that represent other real-world physical objects and (ii) the lighting template representing the first lighting in the digital media depicting the real-world first physical object, a plurality of other modified 3D models lit according to the lighting template, wherein generating the plurality of other modified 3D models includes modifying each of the plurality of other 3D models to have a virtual light source at a virtual position corresponding to the position of the light source relative to the real-world first physical object as depicted in the digital media, applying the environment map to each of the plurality of other 3D models, and for each of the plurality of other 3D models, modifying the normalized lighting based on the lighting template.

2. The method of claim 1 , wherein:
determining the lighting template further comprises determining other properties of the light source other than the position of the light source relative to the real-world first physical object as depicted in the digital media; and
wherein properties of the virtual light source are also based on the other properties of the light source.

3. The method of claim 2 , wherein:
determining the other properties of the light source comprises determining a type of light source; and
the virtual light source comprises the same type of light source.

4. The method of claim 1 , wherein determining the lighting template comprises:
comparing the modified 3D model to the digital media;
determining, based on the comparing, that the modified 3D model matches the digital media; and
responsive to determining that the modified 3D model matches the digital media, determining that the modified 3D model includes the lighting template.

5. The method of claim 4 , wherein determining the lighting template comprises multiple iterations of:
generating a respective modified 3D model based on the 3D model and a respective lighting template; and
comparing the respective modified 3D model to the digital media.

6. The method of claim 1 , wherein generating the modified 3D model comprises modifying a texture map of the 3D model based on the lighting template; and wherein generating the plurality of other modified 3D models comprises modifying a plurality of texture maps of the plurality of other 3D models based on the lighting template, each of the plurality of texture maps corresponding to a respective one of the plurality of other 3D models.

7. The method of claim 1 , wherein:
the digital media comprises multiple different instances of digital media; and
determining the lighting template comprises averaging different lighting depicted in the different instances of digital media.

8. The method of claim 7 , wherein the multiple different instances of digital media depict multiple different physical objects including the real-world first physical object.

9. The method of claim 1 , wherein the real-world first physical object is the same as the 3D model that represents the real-world first physical object.

10. The method of claim 1 , wherein the real-world first physical object corresponds to a product sold by a merchant.

11. The method of claim 1 , wherein the digital media comprises at least one of an image, a video or a further 3D model.

12. A system comprising:
memory to store a three-dimensional (3D) model that represents a real-world first physical object that is depicted in previously captured digital media and a plurality of other 3D models that represent other real-world physical objects, wherein the other real-world physical objects are related to the real-world first physical object; and
at least one processor to:
determine, based on the previously captured digital media depicting the real-world first physical object as lit by first lighting, a lighting template representing the first lighting, wherein determining the lighting template includes determining a position of a light source relative to the real-world first physical object as depicted in the digital media and further includes determining an environment map for an environment containing the real-world first physical object;
normalize second lighting of the 3D model to obtain normalized lighting;
normalize a plurality of lightings of the plurality of other 3D models, each of the plurality of lightings corresponding to a respective one of the plurality of other 3D models, to obtain the normalized lighting; and
apply the lighting template to the 3D model that represents the real-world first physical object and to each one of the plurality of other 3D models so that the 3D model that represents the real-world first physical object and the plurality of other 3D model are all depicted with consistent lighting, wherein the applying the lighting template includes:
generating, based on both (i) the 3D model that represents the real-world first physical object depicted in the digital media and (ii) the lighting template representing the first lighting in the digital media depicting the real-world first physical object, a modified 3D model that represents the real-world first physical object depicted in the digital media lit according to the lighting template, wherein generating the modified 3D model includes modifying the 3D model to have a virtual light source at a virtual position corresponding to the position of the light source rel

Assignees

• Shopify Inc

Inventors

• Delgado Byron Leonel
• Leroux Stephan
• Beauchamp Daniel

Classifications

• G06T15/506 Primary

  Illumination models · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T17/205

  Re-meshing · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title