Systems, devices, and methods for image enhancement

Patent metadata

Field	Value
Publication number	US-11323634-B2
Application number	US-202016921289-A
Country	US
Kind code	B2
Filing date	Jul 6, 2020
Priority date	Aug 30, 2017
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

Disclosed are systems, methods, and computer-readable storage media to enhance an image on a mobile computing device. In some aspects, a user interface on the mobile computing device provides a plurality of image filters, each of the image filters associated with a different replacement sky type. Upon application of one of the plurality of image filters to an image, the mobile computing device is configured to identify a sky portion of the image, modify a color mapping of a non-sky portion of the image, and replace the sky portion of the image with replacement sky data corresponding to the applied image filter.

First claim

We claim:

1. A device, comprising:
a memory; and
one or more electronic hardware processors coupled to the memory and configured to perform operations comprising:
providing a plurality of filters in a user interface, each of the filters having an associated replacement sky type;
selecting a filter from the plurality of filters, the plurality of filters comprising a sunset mode filter that causes a yellow tone to be added to one or more regions of a given image having a brightness or luminance above a predetermined threshold and causes blue tones to be added to one or more regions of the given image that have a brightness or luminance below the predetermined threshold;
receiving an image;
replacing a sky portion of the image based on the replacement sky type associated with the selected filter.

2. The device of claim 1 , wherein the operations further comprise:
when an image is captured by the device after receiving the selection of the filter:
performing a rough segmentation of the image to generate a first segmented image having ambiguity between a sky portion and non-sky portion of the image;
performing, using the image captured by the device as a reference, refined segmentation of the first segmented image to generate a second segmented image in which the ambiguity between the sky portion and the non-sky portion of the image has been reduced, the refined segmentation being performed by applying a guided filter to the image and the first segmented image, the guided filter based on a guidance image comprising a grayscale version of the image; and
identifying the sky portion of the image based on the second segmented image.

3. The device of claim 2 , wherein the guided filter is further based on a joint bilateral filter.

4. The device of claim 1 , wherein the operations further comprise:
converting the image to a lab color space; and
shifting color values in A and B channels of the lab color space away from a center value in each of the A and B channels respectively.

5. The device of claim 1 , wherein the operations further comprise retrieving a plurality of color curves each associated with a different color.

6. The device of claim 1 , wherein the operations further comprise:
presenting a plurality of filter selection control indicators simultaneously with a given image presented on a display of the device, each of the plurality of filter selection control indicators being associated with a respective one of the plurality of filters; and
visually distinguishing one of the filter selection control indicators that corresponds to the one of the plurality of filters selected by received input from a user.

7. The device of claim 6 , wherein a first of the plurality of filter selection control indicators textually represents a first of the plurality of filters as a nighttime sky filter, a second of the plurality of filter selection control indicators textually represents a second of the plurality of filters as a daytime sky filter, and a third of the plurality of filter selection control indicators textually represents a third of the plurality of filters as a sunset sky filter.

8. The device of claim 1 , wherein the operations further comprise applying a trained classifier to the image, wherein the trained classifier is trained based on training data that includes training images and corresponding annotations that indicate sky portions in the training images and non-sky portions in the training images, wherein the corresponding annotations are received by user input that identifies different portions in each of the training images, the sky portion of the image being identified by identifying similarities between a filter response of the image and filter responses of the sky and non-sky portions of the training images.

9. The device of claim 8 , wherein the operations further comprise identifying the sky portion by determining a weight of each pixel of the image after the sky portion is replaced.

10. The device of claim 9 , wherein replacing the sky portion comprises:
determining a weight for a first pixel value in a first location in the sky portion of the image;
determining a second weight for a second pixel value in the sky data, the second pixel value having a location in the sky data corresponding to the first location; and
multiplying the first weight by the first pixel value and the second weight by the second pixel value and adding a result of the multiplying to determine a pixel value for the image.

11. The device of claim 1 , wherein the plurality of filters comprises a daytime mode filter that causes an overall brightness of the image to be increased and a warm color tone to be added to the image.

12. The device of claim 1 , wherein the operations further comprise:
retrieving a plurality of color curves each associated with a different color;
modifying a first color of the image based on a first of the plurality of color curves associated with the first color that increases the first color of a first pixel in the image that has a first color value that exceeds a first threshold; and
modifying a second color of the image based on a second of the plurality of color curves associated with the second color that decreases the second color of a second pixel in the image that has a second color value that is below a second threshold.

13. The device of claim 1 , wherein the plurality of filters comprises a nighttime mode filter that causes an overall brightness of the image to be decreased and a cold color tone to be added to the image.

14. A method, comprising:
providing a plurality of filters in a user interface, each of the filters having an associated replacement sky type;
selecting a filter from the plurality of filters, the plurality of filters comprising a sunset mode, filter that causes a yellow tone to be added to one or more regions of a given image having a brightness or luminance above a predetermined threshold and causes blue tones to be added to one or more regions of the given image that have a brightness or luminance below the predetermined threshold;
receiving an image;
replacing a sky portion of the image based on the replacement sky type associated with the selected filter.

15. The method of claim 14 , further comprising:
when an image is captured by a device after receiving the selection of the filter:
performing a rough segmentation of the image to generate a first segmented image having ambiguity between a sky portion and non-sky portion of the image;
performing, using the image captured by the device as a reference, refined segmentation of the first segmented image to generate a second segmented image in which the ambiguity between the sky portion and the non-sky portion of the image has been reduced, the refined segmentation being performed by applying a guided filter to the image and the first segmented image, the guided filter based on a guidance image comprising a grayscale version of the image; and
identifying the sky portion of the image based on the second segmented image.

16. The method of claim 15 , wherein the guided filter is further based on a joint bilateral filter.

17. The method of claim 14 , further comprising:
converting the image to a lab color space; and
shifting color values in A and B channels of the lab color space away from a center value in each of the A and B channels respectively.

18. The method of claim 14 , wherein the plurality of filters comprises a daytime mode filter that causes an overall brightness of the image to be increased and a warm color tone to be added to t

Assignees

• Snap Inc

Inventors

• Ma Chongyang
• Duan Kun
• Mei Xing
• Hu Nan

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T13/80

  Two-dimensional [2D] animation, e.g. using sprites · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/11

  Region-based segmentation · CPC title