Sky editing based on image composition

Patent metadata

Field	Value
Publication number	US-10762608-B2
Application number	US-201816119709-A
Country	US
Kind code	B2
Filing date	Aug 31, 2018
Priority date	Apr 8, 2016
Publication date	Sep 1, 2020
Grant date	Sep 1, 2020

Abstract

Embodiments of the present disclosure relate to a sky editing system and related processes for sky editing. The sky editing system includes a composition detector to determine the composition of a target image. A sky search engine in the sky editing system is configured to find a reference image with similar composition with the target image. Subsequently, a sky editor replaces content of the sky in the target image with content of the sky in the reference image. As such, the sky editing system transforms the target image into a new image with a preferred sky background.

First claim

What is claimed is:

1. A computer-implemented method for editing images, comprising:
constructing, by one or more processors, a first composite label representing a first composition of a foreground of a first image and a second composite label representing a second composition of a background of the first image;
constructing, based on the first composite label and the second composite label, a composite label to represent a first layout of the first image;
identifying, by the one or more processors, based on the composite label, a second image having a second layout similar to the first layout; and
generating, by the one or more processors, a composite image including a first portion of the first image and a second portion of the second image, wherein the generating comprises extracting, from the second image, a maximum rectangular region without a foreground of the second image, and rescaling the maximum rectangular region to a size of a minimum rectangle region to cover all the background of the first image.

2. The method of claim 1 , further comprising:
dividing the first image to a plurality of areas; and
generating scene parsing labels for the plurality of areas based on a scene classification model.

3. The method of claim 2 , further comprising:
generating respective histograms based on the scene parsing labels; and
concatenating information of the respective histograms to form the first composite label and the second composite label.

4. The method of claim 2 , further comprising:
forming respective one-dimensional vectors based on the scene parsing labels; and
forming the first composite label and the second composite label based on the respective one-dimensional vectors.

5. The method of claim 1 , wherein constructing the composite label comprises constructing the composite label to represent composition information of the foreground and the background of the first image.

6. The method of claim 1 , wherein constructing the composite label comprises constructing the composite label to represent information of a ratio between the foreground and the background of the first image.

7. The method of claim 1 , further comprising:
computing a color of a boundary between the first portion of the first image and the second portion of the second image based on a linear combination of a first color of the first portion of the first image and a second color of the second portion of the second image.

8. The method of claim 1 , further comprising:
clustering, based at least in part on the composite label, the first image into a subclass of training images to train a neural network; and
using the neural network to determine the second image having the second layout similar to the first layout.

9. The method of claim 1 , wherein the first portion of the first image is a foreground, and the second portion of the second image is a background.

10. A non-transitory computer storage medium comprising computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
constructing a first label representing a first composition of a foreground of a first image and a second label representing a second composition of a background of the first image;
constructing, based on the first label and the second label, a composite label to represent a first layout of the first image;
identifying, based on the composite label, a second image having a second layout similar to the first layout; and
generating a composite image based on a first portion of the first image and a second portion of the second image, wherein the generating comprises extracting, from the second image, a maximum rectangular region without a foreground of the second image, and rescaling the maximum rectangular region to match a size of the background of the first image.

11. The non-transitory computer storage medium of claim 10 , wherein the operations further comprising:
clustering the first image into a plurality of training images based on the composite label; and
determining the second image having the second layout similar to the first layout based on a supervised learning with the plurality of training images.

12. The non-transitory computer storage medium of claim 10 , wherein the operations further comprising:
determining characteristics of a plurality of areas of the first image;
selecting the foreground and the background from the plurality of areas.

13. The non-transitory computer storage medium of claim 12 , wherein the selecting comprises selecting the foreground and the background based on scene parsing labels of the plurality of areas.

14. The non-transitory computer storage medium of claim 12 , wherein the selecting comprises selecting the foreground based on one or more areas of the plurality of areas being characterized as foreground parts.

15. The non-transitory computer storage medium of claim 12 , wherein the selecting comprises selecting the foreground and the background based on color histograms of the plurality of areas.

16. The non-transitory computer storage medium of claim 10 , wherein the generating comprises adjusting a foreground of the composite image based on a luminance of the second portion of the second image.

17. The non-transitory computer storage medium of claim 10 , wherein the generating comprises adjusting a foreground of the composite image based on a color temperature of the second portion of the second image.

18. The non-transitory computer storage medium of claim 10 , wherein the generating comprises adjusting a foreground of the composite image based on a saturation of the second portion of the second image.

19. A computer system, comprising:
a computer memory; and
a processor, operationally coupled with the computer memory, configured to execute operations comprising:
constructing a first label representing a first composition of a foreground of a first image and a second label representing a second composition of a background of the first image;
constructing, based on the first label and the second label, a composite label to represent a first layout of the first image;
identifying, based on the composite label, a second image having a second layout similar to the first layout;
extracting, from the second image, a maximum rectangular region without a foreground of the second image;
rescaling the maximum rectangular region to match a size of the background of the first image; and
generating a composite image based on the rescaled maximum rectangular region and the foreground of the first image.

20. The computer system of claim 19 , wherein the operations further comprising:
clustering the first image into a plurality of training images based on the composite label; and
determining the second image having the second layout similar to the first layout based on a supervised learning with the plurality of training images.

Assignees

• Adobe Inc

Inventors

• Shen Xiaohui
• Tsai Yi-Hsuan
• Sunkavalli Kalyan K
• Lin Zhe

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06F16/5838

  using colour · CPC title