Computing apparatus and operation method of the same

Patent metadata

Field	Value
Publication number	US-11640650-B2
Application number	US-202017030905-A
Country	US
Kind code	B2
Filing date	Sep 24, 2020
Priority date	Oct 16, 2019
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

Provided are a computing apparatus for constructing a mosaic image and an operation method of the same. The computing apparatus includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to: segment an input image into a plurality of sub areas to obtain a plurality of sub area images, extract a feature from each of the plurality of sub area images, generate a plurality of source images respectively corresponding to the plurality of sub areas using an image generation neural network, the image generation neural network using, as a condition, the feature extracted from each of the plurality of sub area images, and combine the plurality of source images respectively corresponding to the plurality of sub areas to generate a mosaic image.

First claim

What is claimed is:

1. A computing apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory to:
obtain a plurality of sub area images by segmenting an input image into a plurality of sub areas,
extract a feature from each of the plurality of sub area images and generate feature vectors corresponding to the extracted features of the plurality of sub area images,
generate a plurality of source images respectively corresponding to the plurality of sub areas using an image generation neural network, wherein each source image of the plurality of source images is generated using an image generation neural network using, as a condition, the feature vector from each of the plurality of sub area images to generate the source images including a feature designated by the respective feature vector, and
generate a mosaic image at least by locating each source image of the plurality of source images at the corresponding sub area.

2. The computing apparatus of claim 1 , wherein the processor is further configured to execute the one or more instructions to: extract the feature of each of the plurality of sub area images from each of the plurality of sub area images and generate the feature vectors for the plurality of sub area images using a feature extraction neural network.

3. The computing apparatus of claim 2 , wherein the feature of each of the plurality of sub area images includes at least one of color, texture, or geometric information.

4. The computing apparatus of claim 1 , wherein the image generation neural network includes a generative adversarial network (GAN).

5. A computing apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory to:
segment an area of an empty image including no content into a plurality of sub areas,
generate a plurality of source images respectively corresponding to the plurality of sub areas, wherein each source image of the plurality of source images is generated using an image generation neural network, and
generate a mosaic image by locating each source image of the plurality of source images at the corresponding sub area,
wherein to generate the plurality of source images the processor is configured to execute the one or more instructions stored in the memory to:
generate a first source image of the plurality of source images corresponding to a first sub area of the plurality of sub areas using the image generation neural network without the image generation neural network using a condition related to esthetics of other images of the plurality of source images,
after generating the first source image, obtain an esthetics score of a previously generated source image at least by evaluating esthetics of the previously generated source image, and
generate other source images of the plurality of source images respectively corresponding to the plurality of sub areas using the image generation neural network, wherein each of other source image of the plurality of source images is generated using the image generation neural network using, as a condition, the esthetics score of the previously generated source image of the plurality of source images.

6. The computing apparatus of claim 5 , wherein the processor is further configured to execute the one or more instructions to: evaluate, based on obtaining the esthetics score of the previously generated source image, the esthetics of the previously generated source image based on the previously generated source image and a partially completed mosaic image.

7. A computing apparatus comprising: a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory to:
receive a template representing a composition of a mosaic image to be generated,
segment an area of an empty image including no content into a plurality of sub areas,
generate a plurality of source images respectively corresponding to the plurality of sub areas using an image generation neural network, wherein each source image of the plurality of source images is generated using the image generation neural network using, as a condition, information about the template, and
generate a mosaic image by locating each source image of the plurality of source images at the corresponding sub area,
wherein to generate the plurality of source images the processor is configured to execute the one or more instructions stored in the memory to:
generate a first source image of the plurality of source images corresponding to a first sub area of the plurality of sub areas using the image generation neural network without the image generation neural network using a condition related to esthetics of other images of the plurality of source images,
after generating the first source image, obtain an esthetics score of a previously generated source image by evaluating esthetics of the previously generated source image, and
generate other source images of the plurality of source images respectively corresponding to the plurality of sub areas using the image generation neural network, the image generation neural network using, as a condition, the esthetics score of the previously generated source image of the plurality of source images.

8. The computing apparatus of claim 7 , wherein the information about the template includes object information corresponding to each of the plurality of sub areas in the template.

9. The computing apparatus of claim 7 , wherein the template is a user designated template.

10. A method of operating a computing apparatus, comprising:
obtaining a plurality of sub area images by segmenting an input image into a plurality of sub areas;
extracting a feature from each of the plurality of sub area images and generating feature vectors corresponding to the extracted features of the plurality of sub area images;
generating a plurality of source images respectively corresponding to the plurality of sub areas using an image generation neural network, wherein each source image of the plurality of source images is generated using the image generation neural network using, as a condition, the feature vector from each of the plurality of sub area images to generate the source images including a feature designated by the respective feature vector; and
generating a mosaic image by locating each source image of the plurality of source images at the corresponding sub areas.

11. The method of claim 10 , further comprising extracting the feature of each of the plurality of sub area images from each of the plurality of sub area images and generating the feature vectors for the plurality of sub area images using a feature extraction neural network.

12. The method of claim 11 , wherein the feature of each of the plurality of sub area images includes at least one of color, texture, or geometric information.

13. The method of claim 10 , wherein the image generation neural network includes a generative adversarial network (GAN).

14. A method of operating a computing apparatus, comprising:
segmenting an area of an empty image including no content into a plurality of sub areas;
generating a plurality of source images respectively corresponding to the plurality of sub areas using an image generation neural network, wherein each source image of the plurality of source images is generated using the image generation neural network; and
generating a mosaic image by locating each source image of the plurality of source images at the corresponding sub area, wherein

Assignees

• Samsung Electronics Co Ltd

Inventors

• Baijal Anant
• Agarwal Vivek
• Koo Jayoon

Classifications

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06V10/40

  Extraction of image or video features · CPC title