Image enhancement system

Patent metadata

Field	Value
Publication number	US-11758079-B2
Application number	US-202217709508-A
Country	US
Kind code	B2
Filing date	Mar 31, 2022
Priority date	Aug 30, 2017
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

Disclosed are systems, methods, and computer-readable storage media to enhance an image on a mobile computing device. In some aspects, a user interface on the mobile computing device provides a plurality of image filters, each of the image filters associated with a different replacement sky type. Upon application of one of the plurality of image filters to an image, the mobile computing device is configured to identify a sky portion of the image, modify a color mapping of a non-sky portion of the image, and replace the sky portion of the image with replacement sky data corresponding to the applied image filter.

First claim

We claim:

1. A device, comprising:
a memory; and
one or more electronic hardware processors coupled to the memory and configured to perform operations comprising:
receiving input that selects a filter from a plurality of filters, the plurality of filters comprising a sunset mode filter;
generating an image according to a skytone associated with the selected filter, the generating of the image comprising:
modifying a first color of the image based on a first of a plurality of color curves associated with the first color that increases the first color of a first pixel in the image that has a first color value that transgresses a first threshold; and
modifying a second color of the image based on a second of the plurality of color curves associated with the second color that decreases the second color of a second pixel in the image that has a second color value that fails to transgress a second threshold.

2. The device of claim 1 , wherein each of the plurality of filters has an associated replacement sky type and the operations further comprise:
receiving a particular image; and
generating the image by replacing a sky portion of the particular image based on the replacement sky type associated with the selected filter.

3. The device of claim 1 , wherein when the image is captured by the device after receiving the filter selection, the operations further comprise:
performing a rough segmentation of the image to generate a first segmented image having ambiguity between a sky portion and non-sky portion of the image; and
performing, using the image captured by the device as a reference, refined segmentation of the first segmented image to generate a second segmented image in which the ambiguity between the sky portion and the non-sky portion of the image has been reduced, the refined segmentation being performed by applying a guided filter to the image and the first segmented image, the guided filter based on a guidance image comprising a grayscale version of the image, wherein the image is generated by replacing the sky portion of the image based on the second segmented image.

4. The device of claim 3 , wherein the guided filter is further based on a joint bilateral filter.

5. The device of claim 1 , wherein the sunset mode filter causes a yellow tone to be added to one or more regions of a given image having a brightness or luminance above a predetermined threshold and causes blue tones to be added to one or more regions of the given image that have a brightness or luminance below the predetermined threshold.

6. The device of claim 1 , wherein the operations further comprise:
presenting a plurality of filter selection control indicators simultaneously with a given image presented on a display of the device, each of the plurality of filter selection control indicators being associated with a respective one of the plurality of filters; and
visually distinguishing one of the filter selection control indicators that corresponds to the one of the plurality of filters selected by received input from a user.

7. The device of claim 6 , wherein a first filter selection control of the plurality of filter selection control indicators textually represents a nighttime sky filter, a second filter selection control of the plurality of filter selection control indicators textually represents a daytime sky filter, and a third filter selection control of the plurality of filter selection control indicators textually represents the sunset mode filter.

8. The device of claim 1 , wherein the operations further comprise:
applying a trained classifier to the image, wherein the trained classifier is trained based on training data that includes training images and corresponding annotations that indicate sky portions in the training images and non-sky portions in the training images, wherein the corresponding annotations are received by user input that identifies different portions in each of the training images, the sky portion of the image being identified by identifying similarities between a filter response of the image and filter responses of the sky and non-sky portions of the training images.

9. The device of claim 8 , wherein the operations further comprise:
identifying the sky portion by determining a weight of each pixel of the image after the sky portion is replaced.

10. The device of claim 9 , wherein replacing the sky portion comprises:
determining a first weight for a first pixel value in a first location in the sky portion of the image;
determining a second weight for a second pixel value in the sky data, the second pixel value having a location in the sky data corresponding to the first location; and
multiplying the first weight by the first pixel value and the second weight by the second pixel value and adding a result of the multiplying to determine a pixel value for the image.

11. The device of claim 1 , wherein the plurality of filters comprises a daytime mode filter that causes an overall brightness of the image to be increased and a warm color tone to be added to the image.

12. The device of claim 1 , wherein the operations further comprise:
retrieving the plurality of color curves each associated with a different color.

13. The device of claim 1 , wherein the plurality of filters comprises a nighttime mode filter that causes an overall brightness of the image to be decreased and a cold color tone to be added to the image.

14. A method, comprising:
receiving input that selects a filter from a plurality of filters, the plurality of filters comprising a sunset mode filter;
generating an image according to a skytone associated with the selected filter, the generating of the image comprising:
modifying a first color of the image based on a first of a plurality of color curves associated with the first color that increases the first color of a first pixel in the image that has a first color value that transgresses a first threshold; and
modifying a second color of the image based on a second of the plurality of color curves associated with the second color that decreases the second color of a second pixel in the image that has a second color value that fails to transgress a second threshold.

15. The method of claim 14 , wherein when the image is captured after receiving the filter selection, further comprising:
performing a rough segmentation of the image to generate a first segmented image having ambiguity between a sky portion and non-sky portion of the image; and
performing, using the image captured as a reference, refined segmentation of the first segmented image to generate a second segmented image in which the ambiguity between the sky portion and the non-sky portion of the image has been reduced, the refined segmentation being performed by applying a guided filter to the image and the first segmented image, the guided filter based on a guidance image comprising a grayscale version of the image, wherein the image is generated by replacing the sky portion of the image based on the second segmented image.

16. The method of claim 15 , wherein the guided filter is further based on a joint bilateral filter.

17. The method of claim 14 , further comprising:
converting the image to a lab color space; and
shifting color values in A and B channels of the lab color space away from a center value in each of the A and B channels respectively.

18. The method of claim 14 , wherein the plurality of filters comprises a daytime mode filter that causes an overall brightness of the image to be increased and a warm color tone to be added to the ima

Assignees

• Snap Inc

Inventors

• Ma Chongyang
• Duan Kun
• Mei Xing
• Hu Nan

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• H04N1/628 Primary

  Memory colours, e.g. skin or sky · CPC title

• H04N5/2621 Primary

  Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T7/11

  Region-based segmentation · CPC title