Method, electronic device, and computer program product for image generation

Patent metadata

Field	Value
Publication number	US-12511795-B2
Application number	US-202318129136-A
Country	US
Kind code	B2
Filing date	Mar 31, 2023
Priority date	Feb 28, 2023
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

A method in an illustrative embodiment includes: acquiring an image set, where the image set includes a first plurality of images that can be classified into at least two categories; determining a corner case image set in the image set, where the corner case image set includes a second plurality of images that tend to be incorrectly classified; training an image generator with at least some images in the second plurality of images and first guidance associated with the at least some images; and generating an additional image by the trained image generator with a first image in the second plurality of images and additional guidance, where the additional guidance is different from the first guidance. By means of the technical solutions of the present disclosure, an image generation efficiency can be improved, and the quality of generated images can be enhanced, thereby improving user experience.

First claim

What is claimed is:

1 . A method for image generation, comprising:
acquiring an image set, wherein the image set comprises a first plurality of images that can be classified into at least two categories; determining a corner case image set in the image set, wherein the corner case image set comprises a second plurality of images that tend to be incorrectly classified; training an image generator with at least some images in the second plurality of images and first guidance associated with the at least some images; and generating an additional image by the trained image generator with a first image in the second plurality of images and additional guidance, wherein the additional guidance is different from the first guidance, the additional guidance comprising input information that at least partially mischaracterizes a visual element detected in the first image, and further wherein the additional image is generated by the trained image generator as a modified version of the first image in the second plurality of images, at least in part by: (i) applying a segmentation process to the first image to segment at least the detected visual element into a plurality of segmented portions, and (ii) applying a semantic alignment process to the first image to alter at least a subset of the segmented portions of the first image to provide semantic consistency between the altered segmented portions and the input information, the additional image thereby differing from the first image in the second plurality of images in a manner controlled at least in part by the additional guidance, to provide an expansion of the image set.

2 . The method for image generation according to claim 1 , wherein determining the corner case image set in the image set comprises:
determining the corner case image set utilizing a distance-based surprise adequacy method.

3 . The method for image generation according to claim 2 , wherein determining the corner case image set utilizing the distance-based surprise adequacy method comprises:
determining an image classification space for the at least two categories based on the image set; determining, for a first image in the first plurality of images, a first Euclidean distance between the first image and other images belonging to the same category in the image classification space and a second Euclidean distance between the first image and images belonging to other categories in the image classification space; and determining, based on the first Euclidean distance and the second Euclidean distance, whether the first image belongs to the corner case image set.

4 . The method for image generation according to claim 3 , wherein determining, based on the first Euclidean distance and the second Euclidean distance, whether the first image belongs to the corner case image set comprises:
determining, based on a ratio of the first Euclidean distance to the second Euclidean distance, whether the first image belongs to the corner case image set.

5 . The method for image generation according to claim 1 , wherein the first guidance comprises at least one of the following:
image guidance; and word guidance.

6 . The method for image generation according to claim 1 , further comprising:
acquiring a prompt associated with the first image in the second plurality of images; generating a processed first image with the first image and the prompt; and generating the additional image with the first image in the second plurality of images and the additional guidance comprising: generating the additional image with the processed first image and the additional guidance.

7 . The method for image generation according to claim 6 , wherein the prompt comprises at least one of the following:
an image prompt; and a word prompt.

8 . The method for image generation according to claim 6 , wherein the first image comprises an object, and the prompt is associated with the object.

9 . The method for image generation according to claim 8 , wherein generating the processed first image with the first image and the prompt comprises:
removing the object from the first image based on the prompt.

10 . An electronic device, comprising:
at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform actions comprising: acquiring an image set, wherein the image set comprises a first plurality of images that can be classified into at least two categories; determining a corner case image set in the image set, wherein the corner case image set comprises a second plurality of images that tend to be incorrectly classified; training an image generator with at least some images in the second plurality of images and first guidance associated with the at least some images; and generating an additional image by the trained image generator with a first image in the second plurality of images and additional guidance, wherein the additional guidance is different from the first guidance, the additional guidance comprising input information that at least partially mischaracterizes a visual element detected in the first image, and further wherein the additional image is generated by the trained image generator as a modified version of the first image in the second plurality of images, at least in part by: (i) applying a segmentation process to the first image to segment at least the detected visual element into a plurality of segmented portions, and (ii) applying a semantic alignment process to the first image to alter at least a subset of the segmented portions of the first image to provide semantic consistency between the altered segmented portions and the input information, the additional image thereby differing from the first image in the second plurality of images in a manner controlled at least in part by the additional guidance, to provide an expansion of the image set.

11 . The electronic device according to claim 10 , wherein determining the corner case image set in the image set comprises:
determining the corner case image set utilizing a distance-based surprise adequacy method.

12 . The electronic device according to claim 11 , wherein determining the corner case image set utilizing the distance-based surprise adequacy method comprises:
determining an image classification space for the at least two categories based on the image set; determining, for a first image in the first plurality of images, a first Euclidean distance between the first image and other images belonging to the same category in the image classification space and a second Euclidean distance between the first image and images belonging to other categories in the image classification space; and determining, based on the first Euclidean distance and the second Euclidean distance, whether the first image belongs to the corner case image set.

13 . The electronic device according to claim 12 , wherein determining, based on the first Euclidean distance and the second Euclidean distance, whether the first image belongs to the corner case image set comprises:
determining, based on a ratio of the first Euclidean distance to the second Euclidean distance, whether the first image belongs to the corner case image set.

14 . The electronic device according to claim 10 , wherein the first guidance comprises at least one of the following:
image guidance; and word guidance.

15 . The electronic device according to clai

Assignees

• Dell Products Lp

Inventors

• Wang Zijia
• Liu Zhisong
• Ni Jiacheng
• Jia Zhen

Classifications

• G06T5/77

  Retouching; Inpainting; Scratch removal · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/764 Primary

  using classification, e.g. of video objects · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title