Segmented differentiable optimization with multiple generators

Patent metadata

Field	Value
Publication number	US-2023086807-A1
Application number	US-202217723879-A
Country	US
Kind code	A1
Filing date	Apr 19, 2022
Priority date	Sep 17, 2021
Publication date	Mar 23, 2023
Grant date	—

Abstract

Embodiments are disclosed for segmented image generation. The method may include receiving an input image and a segmentation mask, projecting, using a differentiable machine learning pipeline, a plurality of segments of the input image into a plurality of latent spaces associated with a plurality of generators to obtain a plurality of projected segments, and compositing the plurality of projected segments into an output image.

First claim

We claim:

1 . A computer-implemented method comprising:
receiving an input image and a segmentation mask; projecting, using a differentiable machine learning pipeline, a plurality of segments of at least an object depicted in the input image into a plurality of latent spaces associated with a plurality of generators to obtain a plurality of projected segments; and compositing the plurality of projected segments into an output image.

2 . The computer-implemented method of claim 1 , further comprising:
receiving a request to edit a first portion of the input image; determining a segment of the input image corresponding to the first portion of the input image; generating, by a generator corresponding to the segment of the input image, an edited image by exploring the latent space associated with the generator; and generating an edited output image by compositing the edited image with the input image.

3 . The computer-implemented method of claim 1 , wherein the plurality of generators are clones of a generator model.

4 . The computer-implemented method of claim 1 , wherein the plurality of generators includes two or more different generator models.

5 . The computer-implemented method of claim 1 , wherein the segmentation mask is dynamically updated based on a stitching loss calculated by a stitching layer.

6 . The computer-implemented method of claim 1 , wherein receiving an input image and a segmentation mask further comprises:
processing the input image using a semantic segmentation model to generate the segmentation mask.

7 . The computer-implemented method of claim 1 , wherein receiving an input image and a segmentation mask further comprises:
receiving an input identifying at least one segment of the segmentation mask via a user interface, wherein the input includes painting the at least one segment on a representation of the input image.

8 . A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:
receive an input image and a segmentation mask; project, using a differentiable machine learning pipeline, a plurality of segments of the input image into a plurality of latent spaces associated with a plurality of generators to obtain a plurality of projected segments; and composite the plurality of projected segments into an output image.

9 . The non-transitory computer readable storage medium of claim 8 , wherein the instructions, when executed, further cause the processor to:
receive a request to edit a first portion of the input image; determine a segment of the input image corresponding to the first portion of the input image; generate, by a generator corresponding to the segment of the input image, an edited image by exploring the latent space associated with the generator; and generate an edited output image by compositing the edited image with the input image.

10 . The non-transitory computer readable storage medium of claim 8 , wherein the plurality of generators are clones of a generator model.

11 . The non-transitory computer readable storage medium of claim 8 , wherein the plurality of generators includes two or more different generator models.

12 . The non-transitory computer readable storage medium of claim 8 , wherein the segmentation mask is dynamically updated based on a stitching loss calculated by a stitching layer.

13 . The non-transitory computer readable storage medium of claim 8 , wherein to receive an input image and a segmentation mask, the instructions, when executed, further cause the processor to:
process the input image using a semantic segmentation model to generate the segmentation mask.

14 . The non-transitory computer readable storage medium of claim 8 , wherein to receive an input image and a segmentation mask, the instructions, when executed, further cause the processor to:
receive an input identifying at least one segment of the segmentation mask via a user interface, wherein the input includes painting the at least one segment on a representation of the input image.

15 . A system comprising:
at least one processor; and a memory including instructions stored thereon which, when executed by the at least one processor, cause the system to:
receive an input image and a segmentation mask;
project, using a differentiable machine learning pipeline, a plurality of segments of the input image into a plurality of latent spaces associated with a plurality of generators to obtain a plurality of projected segments; and
composite the plurality of projected segments into an output image.

16 . The system of claim 15 , wherein the instructions, when executed, further cause the system to:
receive a request to edit a first portion of the input image; determine a segment of the input image corresponding to the first portion of the input image; generate, by a generator corresponding to the segment of the input image, an edited image by exploring the latent space associated with the generator; and generate an edited output image by compositing the edited image with the input image.

17 . The system of claim 15 , wherein the plurality of generators are clones of a generator model.

18 . The system of claim 15 , wherein the plurality of generators includes two or more different generator models.

19 . The system of claim 15 , wherein the segmentation mask is dynamically updated based on a stitching loss calculated by a stitching layer.

20 . The system of claim 15 , wherein to receive an input image and a segmentation mask, the instructions, when executed, further cause the system to:
process the input image using a semantic segmentation model to generate the segmentation mask.

Assignees

• Adobe Inc
• Czech Technical Univ In Prague

Inventors

• Lukác Michal
• Shechtman Elya
• Sýkora Daniel
• Futschik David

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T2207/20104

  Interactive definition of region of interest [ROI] · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/11

  Region-based segmentation · CPC title