Generating and compositing hair pixels using generative neural networks

Patent metadata

Field	Value
Publication number	US-12475623-B2
Application number	US-202318338964-A
Country	US
Kind code	B2
Filing date	Jun 21, 2023
Priority date	Jun 21, 2023
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating and composting pixels of a digital image that depict hair of an individual using generative neural networks. In some embodiments, the disclosed systems receive a modification to a face crop enclosing a face depicted within a digital image. In some cases, the disclosed systems determine, from the modification, modified hair pixels within the face crop of the digital image and unmodified hair pixels outside of the face crop of the digital image. The disclosed systems generate, for the unmodified hair pixels outside of the face crop, replacement hair pixels that resemble the modified hair pixels utilizing a generative neural network. Additionally, the disclosed systems generate a modified digital image by replacing the unmodified hair pixels outside of the face crop with the replacement hair pixels.

First claim

What is claimed is:

1 . A method comprising:
identifying a modification to a face depicted within a face region of a digital image; determining, from the modification utilizing a deep hair matte, modified hair pixels within the face region of the digital image and unmodified hair pixels outside of the face region of the digital image; generating, utilizing a generative neural network for the unmodified hair pixels outside of the face region, replacement hair pixels that resemble the modified hair pixels; and generating, utilizing an image compositing model, a modified digital image by replacing the unmodified hair pixels outside of the face region with the replacement hair pixels.

2 . The method of claim 1 , wherein:
determining the modified hair pixels within the face region comprises determining hair pixels affected by the modification to the face region; and determining the unmodified hair pixels outside of the face region comprises determining hair pixels unaffected by the modification to the face region.

3 . The method of claim 1 , further comprising generating the deep hair matte from the digital image by utilizing a deep matting model to process image segmentations for the digital image.

4 . The method of claim 3 , further comprising:
generating a cropped deep hair mask by cropping the deep hair matte according to the face region of the digital image; and wherein generating the replacement hair pixels comprises utilizing the generative neural network to generate pixels that resemble hair pixels indicated by the cropped deep hair mask.

5 . The method of claim 3 , wherein generating the deep hair matte comprises extracting, from the digital image, an instance segmentation map indicating one or more faces depicted within the digital image.

6 . The method of claim 5 , wherein generating the deep hair matte further comprises encoding, from the instance segmentation map, a part segmentation map defining hair pixels for the one or more faces depicted within the digital image.

7 . The method of claim 1 , wherein generating the modified digital image comprises utilizing a mesh warp to modify a resolution of the replacement hair pixels to match a resolution of the digital image.

8 . A system comprising:
a memory component; and one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising:
identifying a modification to a face depicted within a face region of a digital image;
determining, from the modification, modified hair pixels within the face region of the digital image and unmodified hair pixels outside of the face region of the digital image by generating a deep hair matte from a part segmentation map of the digital image;
generating, utilizing a generative neural network for the unmodified hair pixels outside of the face region, replacement hair pixels that resemble the modified hair pixels; and
generating, utilizing an image compositing model, a modified digital image by replacing the unmodified hair pixels outside of the face region with the replacement hair pixels.

9 . The system of claim 8 , wherein identifying the modification to the face depicted within the face region of the digital image comprises receiving an indication of applying a style modification neural network to pixels within the face region.

10 . The system of claim 8 , wherein generating the deep hair matte comprises:
extracting, from the digital image, an instance segmentation map indicating the face depicted within the digital image; encoding, from the instance segmentation map, a part segmentation map defining hair pixels for the face depicted within the digital image; and utilizing a deep matting model to generate the deep hair matte from the part segmentation map.

11 . The system of claim 8 , wherein generating the replacement hair pixels comprises utilizing the generative neural network to generate hair pixels indicated by the deep hair matte for replacing the unmodified hair pixels outside of the face region.

12 . The system of claim 8 , wherein generating the modified digital image comprises utilizing a mesh warp to align textures of the replacement hair pixels with textures of the digital image.

13 . The system of claim 8 , wherein:
determining the modified hair pixels within the face region comprises determining hair pixels affected by the modification to the face region; and determining the unmodified hair pixels outside of the face region comprises determining hair pixels unaffected by the modification to the face region.

14 . The system of claim 8 , further comprising providing the modified digital image for display on a client device.

15 . A non-transitory computer readable medium storing instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
identifying a modification to a face depicted within a face region of a digital image; determining, from the modification utilizing a deep hair matte, modified hair pixels within the face region of the digital image and unmodified hair pixels outside of the face region of the digital image; generating, utilizing a generative neural network for the unmodified hair pixels outside of the face region, replacement hair pixels that resemble the modified hair pixels; and generating, utilizing an image compositing model, a modified digital image by replacing the unmodified hair pixels outside of the face region with the replacement hair pixels.

16 . The non-transitory computer readable medium of claim 15 , wherein:
determining the modified hair pixels within the face region comprises determining hair pixels affected by the modification to the face region; and determining the unmodified hair pixels outside of the face region comprises determining hair pixels unaffected by the modification to the face region.

17 . The non-transitory computer readable medium of claim 15 , further comprising generating the deep hair matte from the digital image by utilizing a deep matting model to process image segmentations for the digital image.

18 . The non-transitory computer readable medium of claim 17 , further comprising:
generating a cropped deep hair mask by cropping the deep hair matte according to the face region of the digital image; and wherein generating the replacement hair pixels comprises utilizing the generative neural network to generate pixels that resemble hair pixels indicated by the cropped deep hair mask.

19 . The non-transitory computer readable medium of claim 17 , wherein generating the deep hair matte comprises extracting, from the digital image, an instance segmentation map indicating one or more faces depicted within the digital image.

20 . The non-transitory computer readable medium of claim 19 , wherein generating the deep hair matte further comprises encoding, from the instance segmentation map, a part segmentation map defining hair pixels for the one or more faces depicted within the digital image.

Assignees

• Adobe Inc

Inventors

• Khodadadeh Siavash
• Xie Jinrong
• Kalarot Ratheesh
• Ghadar Shabnam

Classifications

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T5/77

  Retouching; Inpainting; Scratch removal · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30201

  Face · CPC title

• G06T7/11

  Region-based segmentation · CPC title