Preset style transfer

Patent metadata

Field	Value
Publication number	US-12406334-B2
Application number	US-202318303271-A
Country	US
Kind code	B2
Filing date	Apr 19, 2023
Priority date	Apr 19, 2023
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

Systems and methods for image processing are described. Embodiments of the present disclosure include an image generation network configured to encode a plurality of abstract images using a style encoder to obtain a plurality of abstract style encodings, wherein the style encoder is trained to represent image style separately from image content. A clustering component clusters the plurality of abstract style encodings to obtain an abstract style cluster comprising a subset of the plurality of abstract style encodings. A preset component generates an abstract style transfer preset representing the abstract style cluster.

First claim

What is claimed is:

1. A method comprising:
encoding a plurality of abstract images using a style encoder to obtain a plurality of abstract style encodings, wherein the style encoder is trained to represent image style separately from image content;
clustering the plurality of abstract style encodings to obtain an abstract style cluster comprising a subset of the plurality of abstract style encodings; and
generating an abstract style transfer preset representing the abstract style cluster by identifying a plurality of style tags associated with images in the abstract style cluster and selecting one or more style tags for the abstract style cluster from the plurality of style tags, wherein the abstract style transfer preset includes the one or more style tags.

2. The method of claim 1 , further comprising:
obtaining a content image including structural content; and
generating a combined image based on the content image and the abstract style transfer preset, wherein the combined image includes the structural content from the content image and a style from the abstract style transfer preset.

3. The method of claim 2 , further comprising:
encoding the content image to obtain a content encoding representing the structural content; and
identifying an abstract style encoding of the abstract style transfer preset, wherein the combined image is generated based on the content encoding and the abstract style encoding.

4. The method of claim 2 , further comprising:
receiving a user input identifying the abstract style transfer preset from a plurality of abstract style transfer presets, wherein the combined image is generated based on the user input.

5. The method of claim 2 , further comprising:
receiving a user input indicating a style transfer weight corresponding to the abstract style transfer preset, wherein the combined image is generated based on the style transfer weight.

6. The method of claim 1 , wherein:
the style encoder is trained using unsupervised learning based on the plurality of abstract images.

7. The method of claim 1 , further comprising:
generating image embeddings corresponding to the plurality of abstract images;
performing zero-shot classification on the plurality of abstract images to obtain probability values corresponding to the plurality of style tags for each of the abstract images; and
ranking the plurality of style tags based on the probability values, wherein the one or more style tags are selected based on the ranking.

8. The method of claim 1 , further comprising:
computing a relevance score based on a distance between the images and a center of the abstract style cluster, wherein the one or more style tags are selected based on the relevance score.

9. The method of claim 1 , further comprising:
computing a tag frequency score based on a cluster tag frequency for the abstract style cluster and an overall tag frequency for the plurality of abstract images, wherein the one or more style tags are selected based on the tag frequency score.

10. A method comprising:
obtaining training data including a plurality of abstract images;
initializing an image generation network including a style encoder and a content encoder;
training the image generation network to generate images that include an abstract style based on an abstract style input and content based on a content input, wherein the style encoder encodes the abstract style from the abstract style input and the content encoder encodes the content from the content input;
generating an abstract style transfer preset based on abstract style encodings of the plurality of abstract images from the style encoder by clustering the abstract style encodings to obtain an abstract style cluster comprising a subset of the abstract style encodings and generating the abstract style transfer preset representing the abstract style cluster by identifying a plurality of style tags associated with images in the abstract style cluster; and
selecting one or more style tags for the abstract style cluster from the plurality of style tags, wherein the abstract style transfer preset includes the one or more style tags.

11. The method of claim 10 , further comprising:
generating a reconstructed image using the image generation network based on a style encoding and a content encoding of an input image; and
computing a reconstruction loss based on the reconstructed image, wherein the image generation network is trained based on the reconstruction loss.

12. The method of claim 10 , further comprising:
generating a style transfer image using the image generation network based on a style encoding of a style image and a content encoding of a content image; and
computing a style loss based on the style transfer image, wherein the image generation network is trained based on the style loss.

13. An apparatus comprising:
a processor;
a memory comprising instructions executable by the processor;
an image generation network including a style encoder and a content encoder configured to generate images that include an abstract style based on an abstract style input and content based on a content input, wherein the style encoder encodes the abstract style from the abstract style input and the content encoder encodes the content from the content input;
a clustering component configured to cluster a plurality of abstract style encodings from the style encoder to obtain an abstract style cluster comprising a subset of the plurality of abstract style encodings; and
a preset component configured to generate an abstract style transfer preset representing the abstract style cluster by identifying a plurality of style tags associated with images in the abstract style cluster and selecting one or more style tags for the abstract style cluster from the plurality of style tags, wherein the abstract style transfer preset includes the one or more style tags.

14. The apparatus of claim 13 , further comprising:
a multi-modal encoder configured to generate image embeddings for a plurality of abstract images and to generate text embeddings for the plurality of style tags.

15. The apparatus of claim 13 , further comprising:
a training component configured to train the image generation network using unsupervised learning based on a plurality of abstract images.

16. The apparatus of claim 13 , further comprising:
an image editing interface configured to receive a user input identifying the abstract style transfer preset from a plurality of abstract style transfer presets, wherein a combined image is generated based on the user input.

Assignees

• Adobe Inc

Inventors

• Ravi Hareesh
• Harikumar Midhun
• Park Taesung
• Kale Ajinkya Gorakhnath

Classifications

• G06T5/00

  Image enhancement or restoration · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/20092

  Interactive image processing based on input by user · CPC title