Electronic device applying bokeh effect to image and controlling method thereof

Patent metadata

Field	Value
Publication number	US-11315224-B2
Application number	US-202016793473-A
Country	US
Kind code	B2
Filing date	Feb 18, 2020
Priority date	Feb 20, 2019
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

An electronic device and method are disclosed. The electronic device includes a memory and processor, which implements the method, including: obtaining image data for an image, set in the image a region of interest and a background region, using depth information, both including one or more sub-regions, determining respective bokeh characteristics for the one or more first sub-regions and the one or more second sub-regions based on a first characteristic criterion and a second characteristic criterion, respectively, and processing the image to apply a bokeh effect to the plurality of pixels based on the respective bokeh characteristics for the one or more first sub-regions and the one or more second sub-regions.

First claim

What is claimed is:

1. An electronic device, comprising:
a display;
a memory; and
a processor electrically connected to the memory,
wherein, the memory stores instructions that are executable by the processor to cause the electronic device to:
obtain image data for an image including a plurality of pixels,
set, within the image, a region of interest and a background region, using depth information associated with each of the plurality of pixels, wherein the region of interest includes one or more first sub-regions, and the background region includes one or more second sub-regions,
determine respective bokeh characteristics for the one or more first sub-regions and the one or more second sub-regions based on a first characteristic criterion and a second characteristic criterion, respectively, and
process the image to apply a bokeh effect to the plurality of pixels based on the respective bokeh characteristics for the one or more first sub-regions and the one or more second sub-regions,
wherein the respective bokeh characteristics for the one or more first sub-regions and the one or more second sub-regions indicate respective intensities of the bokeh effect, and
wherein the instructions are executable by the processor to cause the electronic device to:
identify a drag input on the display on which the image is displayed,
identify at least one third sub-region through which the drag input passes among the one or more first sub-regions and at least one fourth sub-region through which the drag input passes among the one or more second sub-regions,
determine an intensity of the effect applied to at least one third sub-region that is higher than an intensity of the effect applied to the rest of the one or more first sub-regions, and
determine an intensity of the effect applied to at least one fourth sub-region that is higher than an intensity of the effect applied to the rest of the one or more second sub-regions.

2. The electronic device of claim 1 ,
wherein the indicated intensity of the bokeh effect increases based on an increase in depth as indicated by the depth information.

3. The electronic device of claim 1 , wherein the instructions are further executable by the processor to cause the electronic device to:
obtain a height of the image, wherein the respective bokeh characteristics are determined based at least in part on the obtained height, and whether a sub-region is included in the region of interest or the background region,
wherein when a radial bokeh effect is to be applied to the image data, a plurality of concentric circular regions is set for determining the respective bokeh characteristics to be applied to the image data and
wherein when a first concentric circular region overlaps both the region of interest and the background, the first concentric circular region is subdivided into a first portion corresponding to the region of interest and a second portion corresponding to the background region, such that the bokeh effect is applied differently between the first and second portion.

4. The electronic device of claim 1 , further comprising:
a camera for capturing the image, the camera including at least one of a plurality of image sensors, and a depth-detection sensor.

5. The electronic device of claim 1 , wherein the depth information includes a depth value for each pixel of the plurality of pixels, and
wherein the instructions are executable by the processor to cause the electronic device to:
generate the respective bokeh characteristics for each pixel by processing depth values for each pixel using a depth-to-characteristic conversion function.

6. The electronic device of claim 1 , wherein at least one of the respective bokeh characteristics includes a direction for the bokeh effect to be applied to each pixel of the plurality of pixels.

7. The electronic device of claim 6 , further comprising:
an input device,
wherein the instructions are executable by the processor to: determine the direction for the bokeh effect based on another user input received through the input device.

8. The electronic device of claim 7 , wherein the user input selects a bokeh application mode,
wherein the instructions are further executable by the processor to cause the electronic device to:
when the selected bokeh application mode is a first mode, detect a direction of a tangential vector of a pixel on a circumference, the circumference centered on a center of the image and having a radius defined as an interpixel distance from the center to the pixel, and set the detected direction of the tangential vector as the direction of the bokeh effect for the pixel, and
when the selected bokeh application is a second mode, set a direction of a straight line extending toward the pixel from the center position as the direction of the bokeh effect for the pixel.

9. The electronic device of claim 3 , wherein the instructions are executable by the processor to: determine a starting position for the bokeh effect within the region of interest,
wherein the bokeh characteristics are further based at least on the determined starting position,
wherein a second concentric circular region overlaps both the region of interest and the background, the second concentric circular region is subdivided into a third portion corresponding to the region of interest and a fourth portion corresponding to the background region, and
wherein an intensity of the bokeh effect applied to the first portion corresponding to the region of interest differs from an intensity of the bokeh effect applied to the third portion corresponding to the region of interest.

10. The electronic device of claim 9 , wherein the bokeh characteristics are determined based on at least one of:
respective differences between a height value for the starting position and height values for each of the plurality of pixels,
a difference between a depth value indicated in the depth information for a pixel disposed at the starting position and depth values indicated in the depth information a remainder of the plurality of pixels, and
an interpixel distance from the pixel disposed at the starting position to the remainder of the plurality of pixels.

11. The electronic device of claim 9 , wherein, the instructions are executable by the processor to cause the electronic device to:
analyze the image to detect a subject of interest included in the region of interest,
wherein the start position is set within a sub-region of the region of interest in which the subject of interest is disposed.

12. The electronic device of claim 11 , wherein the starting position is determined to be at least one of a centerpoint of the image, and a centerpoint of the sub-region in which the subject of interest isid disposed.

13. The electronic device of claim 1 , wherein, the instructions are executable by the processor to cause the electronic device to:
determine a movement direction of a subject within the region of interest; and
determine bokeh characteristics for each image sub-region with respect to the background region based on the movement direction.

14. The electronic device of claim 13 , wherein, the instructions are executable by the processor to cause the electronic device to:
apply the bokeh effect to a boundary region including a boundary line, the boundary region disposed adjacent to both the region of interest and the background region.

15. The electronic device of claim 1 , wherein, the instructions are executable by the processor to cause the electronic device to:
divide the image into a plurality of shift regions, each shift region including a sh

Assignees

• Samsung Electronics Co Ltd

Inventors

• Hwang Jeena
• Hwang Insung
• Pandey Subash
• Hyun Jongmin
• Won Jonghun

Classifications

• G06T5/20 Primary

  using local operators · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/11

  Region-based segmentation · CPC title