Repairing irregularities in computer-generated images

Patent metadata

Field	Value
Publication number	US-12417518-B2
Application number	US-202218057930-A
Country	US
Kind code	B2
Filing date	Nov 22, 2022
Priority date	Nov 22, 2022
Publication date	Sep 16, 2025
Grant date	Sep 16, 2025

Abstract

Systems and methods for image processing are provided. Embodiments include identifying an image of a face that includes an artifact in a part of the face. A machine learning model generates an intermediate image based on the original image. The intermediate image depicts the part of the face in a closed position. Then the model generates a corrected image based on the intermediate image. The corrected image depicts the face with the part of the face in an open position and without the artifact.

First claim

What is claimed is:

1. A method comprising:
identifying an image of a face with an expression comprising an open position of a part of the face, wherein the image includes an artifact in the part of the face;
selecting a first value for an input attribute, wherein the first value indicates a modified expression comprising a closed position of the part of the face;
generating an intermediate image based on the image and the first value for the input attribute using an image generation network trained to modify a position of the part of the face based on the input attribute, wherein the intermediate image depicts the face with the modified expression comprising the part in the closed position;
selecting a second value for the input attribute, wherein the second value indicates the expression comprising the open position of the part of the face; and
generating a corrected image based on the intermediate image and the second value for the input attribute using the image generation network, wherein the corrected image depicts the face with the expression comprising the part of the face in the open position and without the artifact.

2. The method of claim 1 , wherein:
the image is generated using a diffusion model and the artifact is a product of the diffusion model.

3. The method of claim 1 , further comprising:
providing the first value for the input attribute to the image generation network.

4. The method of claim 3 , further comprising:
providing the second value for the input attribute to the image generation network.

5. The method of claim 1 , wherein:
the part of the image comprises an eye or a mouth.

6. The method of claim 1 , further comprising:
generating a high-resolution image based on the corrected image using a super-resolution network.

7. The method of claim 1 , further comprising:
identifying a portion of the image including an additional artifact; and
generating a subsequent corrected image based on the corrected image and the identified portion using an inpainting network.

8. The method of claim 1 , further comprising:
displaying the corrected image to a user;
receiving a user input indicating a portion of the image; and
generating a binary mask indicating the portion of the image, wherein a subsequent corrected image is generated based on the binary mask.

9. The method of claim 1 , further comprising:
identifying training data including first training images showing faces including artifacts in the part of the face and second training images corresponding to the first training images without the artifacts in the part of the face, wherein the image generation network is trained based on the training data.

10. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
identifying an image of a face with an expression comprising an open position of a part of the face, wherein the image includes an artifact in the part of the face;
selecting a first value for an input attribute, wherein the first value indicates a modified expression comprising a closed position of the part of the face;
generating an intermediate image based on the image and the first value for the input attribute using an image generation network trained to modify a position of the part of the face based on the input attribute, wherein the intermediate image depicts the face with the modified expression comprising the part in the closed position;
selecting a second value for the input attribute, wherein the second value indicates the expression comprising the open position of the part of the face;
generating a corrected image based on the intermediate image and the second value for the input attribute using the image generation network, wherein the corrected image depicts the face with the expression comprising the part of the face in the open position and does not include the artifact in the part of the face;
generating a high-resolution image based on the corrected image using a super-resolution network; and
generating a subsequent corrected image based on the high-resolution image and a mask indicating portion of the high-resolution image for inpainting using an inpainting network.

11. The non-transitory computer-readable medium of claim 10 , wherein the instructions are further executable to perform operations comprising:
providing the first value for the input attribute to the image generation network.

12. The non-transitory computer-readable medium of claim 11 , wherein the instructions are further executable to perform operations comprising:
providing the second value for the input attribute to the image generation network.

13. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device configured to perform operations comprising:
identifying an image of a face with an expression comprising an open position of a part of the face, wherein the image includes an artifact in the part of the face;
selecting a first value for an input attribute, wherein the first value indicates a modified expression comprising a closed position of the part of the face;
generating an intermediate image based on the image and the first value for the input attribute using an image generation network trained to modify a position of the part of the face based on the input attribute, wherein the intermediate image depicts the face with the modified expression comprising the part in the closed position;
selecting a second value for the input attribute, wherein the second value indicates the expression comprising the open position of the part of the face; and
generating a corrected image based on the intermediate image and the second value for the input attribute using the image generation network, wherein the corrected image depicts the face with the expression comprising the part of the face in the open position and without the artifact.

14. The system of claim 13 , wherein:
the image is generated using a diffusion model and the artifact is a product of the diffusion model.

15. The system of claim 13 , the processing device further configured to perform operations comprising:
providing the first value for the input attribute to the image generation network.

16. The system of claim 15 , the processing device further configured to perform operations comprising:
providing the second value for the input attribute to the image generation network.

17. The system of claim 13 , wherein:
the part of the image comprises an eye or a mouth.

18. The system of claim 13 , the processing device further configured to perform operations comprising:
generating a high-resolution image based on the corrected image using a super-resolution network.

19. The system of claim 13 , the processing device further configured to perform operations comprising:
identifying a portion of the image including an additional artifact; and
generating a subsequent corrected image based on the corrected image and the identified portion using an inpainting network.

20. The system of claim 13 , the processing device further configured to perform operations comprising:
displaying the corrected image to a user;
receiving a user input indicating a portion of the image; and
generating a binary mask indicating the portion of the image, wherein a subsequent corrected image is generated based on the binary mask.

Assignees

• Adobe Inc

Inventors

• Agarwal Anjali
• Khodadadeh Siavash
• Kalarot Ratheesh
• Qu Hui
• Olsen Sven C
• Ghadar Shabnam

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30201

  Face · CPC title