Albedo reconstruction based on inverse shading prediction

Patent metadata

Field	Value
Publication number	US-2025078229-A1
Application number	US-202318241039-A
Country	US
Kind code	A1
Filing date	Aug 31, 2023
Priority date	Aug 31, 2023
Publication date	Mar 6, 2025
Grant date	—

Abstract

Various disclosed embodiments are directed to deriving an albedo output image from an input image based on deriving an inverse shading map. For example, an input image can be a photograph of a human face (i.e., the geometric features) with RGB values representing the color values of the face as well as pixels representing shadows (i.e., the shadow features) underneath the chin of the human face. The inverse shading map may be a black and white pixel value image that contains pixels representing the same human face without the RGB values and the shadows underneath the chin. The inverse shading map thus relies on the geometric space, rather than RGB space. Geometric space, for example, allows embodiments to capture the geometric features of a face, as opposed to those geometric features' RGB or shadow details.

First claim

What is claimed is:

1 . A system comprising:
at least one computer processor; and one or more computer storage media storing computer-useable instructions that, when used by the at least one computer processor, cause the at least one computer processor to perform operations comprising: receiving an input image, the input image including: a set of geometric features that define one or more portions of one or more real-world objects, a set of shadow features associated with the one or more real-world objects, and a set of color features that indicate one or more colors of the one or more real-world objects; based on the input image, deriving, via a first model, an inverse shading map, the inverse shading map indicating the set of geometric features without the set of shadow features and without the set of color features; and based on the inverse shading map and the input image, deriving an albedo output image, the albedo output image indicating the set of geometric features and the set of color features but not the set of shadow features.

2 . The system of claim 1 , wherein the deriving of the inverse shading map is further based on generating, via a second model, a normal map and generating, via a third model, a segmentation map, the normal map indicating a microstructure texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects, wherein the normal map and the segmentation map are provided as input to the first model to produce the inverse shading map.

3 . The system of claim 1 , wherein the deriving of the albedo output image is further based on generating the albedo output image via a second model, and wherein the second model is a Generative Adversarial Network (GAN).

4 . The system of claim 1 , wherein the deriving of the albedo output image is further based on multiplying the input image by the inverse shading map.

5 . The system of claim 1 , wherein the operations further comprising training the first model by learning an inverse shading prediction function based on minimizing a perceptual loss between a ground truth image and an albedo training image and minimizing a discrimination loss between the ground truth image and a normal training image or a segmentation training image.

6 . The system of claim 1 , wherein the set of geometric features are features of a human face and hair, and wherein the one or more real-world objects include the human face and hair, and wherein the set of shadow features include shadows on the human face, and wherein the set of color features include a skin color of the human face, or hair color of the hair.

7 . The system of claim 1 , wherein the input image further includes a set of lighting features that represent highlights or lighting on the one or more real-world objects, and wherein the inverse shading map is further without the lighting features, and wherein the albedo output image does not include the lighting features.

8 . The system of claim 7 , wherein the inverse shading map is an image that includes negative lighting and shading relative to the input image.

9 . The system of claim 1 , wherein the albedo output image includes a second set of color features in a same position as the set of shadow features, and wherein the second set of color features are indicative of shadow removal from the input image.

10 . A computer-implemented method comprising:
receiving an input image, the input image including: a set of geometric features that define one or more portions of one or more real-world objects, a set of shadow features associated with the one or more real-world objects, and a set of color features that indicate one or more colors of the one or more real-world objects; based on the input image, deriving an inverse shading map, the inverse shading map indicating the set of geometric features without the set of shadow features; and based on the inverse shading map and the input image, deriving an albedo output image, the albedo output image includes a second set of color features in a same position as the set of shadow features, and wherein second set of color features is indicative of shadow removal from the input image.

11 . The computer-implemented method of claim 10 , wherein the deriving of the inverse shading map is based on generating, via a first model, the inverse shading map, and wherein the deriving of the inverse shading map is further based on generating, via a second model, a normal map and generating, via a third model, a segmentation map, the normal map indicating a microstructure texture of the one or more real-world objects, the segmentation map indicating, via a unique pixel mask, different features of the one or more real-world objects, wherein the normal map and the segmentation map are provided as input to the first model to produce the inverse shading map.

12 . The computer-implemented method of claim 10 , wherein the deriving of the albedo output image is further based on generating or modifying the albedo output image via a Generative Adversarial Network (GAN).

13 . The computer-implemented method of claim 10 , wherein the deriving of the albedo output image is further based on multiplying the input image by the inverse shading map.

14 . The computer-implemented method of claim 10 , wherein the deriving of the inverse shading map is based on a first model generating the inverse shading map, and wherein the method further comprising training the first model by learning an inverse shading prediction function based on minimizing a perceptual loss between a ground truth image and an albedo training image and minimizing a discrimination loss between the ground truth image and a normal training image or a segmentation training image.

15 . The computer-implemented method of claim 10 , wherein the set of geometric features are features of a human face and hair, and wherein the one or more real-world objects include the human face and the hair, and wherein the set of shadow features include shadows on the human face, and wherein the set of color features include a skin color of the human face, or hair color of the hair.

16 . The computer-implemented method of claim 10 , wherein the input image further includes a set of lighting features that represent highlights or lighting on the one or more real-world objects, and wherein the inverse shading map is further without the lighting features, and wherein the albedo output image does not include the lighting features.

17 . The computer-implemented method of claim 10 , wherein the inverse shading map is an image that includes negative lighting and shading relative to the input image.

18 . The computer-implemented method of claim 10 , wherein the albedo output image indicates the set of geometric features and the set of color features but not the set of shadow features.

19 . A computerized system, the system comprising:
an inverse shading map means for receiving an input image, the input image including: a set of geometric features that define one or more portions of one or more real-world objects, a set of shadow features associated with the one or more real-world objects, and a set of color features that indicate one or more colors of the one or more real-world objects; at least one of a segmentation map means and a normal map means for generating at least one of a segmentation map and a normal map respectively, the normal map indicating a texture of the one or more real-world objects, the segmentation

Assignees

• Adobe Inc

Inventors

• Zhang He
• Yoon Jae Shin
• Jung Hyun Joon
• Sun Xin

Classifications

• G06T5/94 Primary

  based on local image properties, e.g. for local contrast enhancement · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title