Kernel reshaping-powered splatting-based efficient image space lens blur

Patent metadata

Field	Value
Publication number	US-11501413-B2
Application number	US-202016950320-A
Country	US
Kind code	B2
Filing date	Nov 17, 2020
Priority date	Nov 17, 2020
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

Embodiments are disclosed for generating lens blur effects. The disclosed systems and methods comprise receiving a request to apply a lens blur effect to an image, the request identifying an input image and a first disparity map, generating a plurality of disparity maps and a plurality of distance maps based on the first disparity map, splatting influences of pixels of the input image using a plurality of reshaped kernel gradients, gathering aggregations of the splatted influences, and determining a lens blur for a first pixel of the input image in an output image based on the gathered aggregations of the splatted influences.

First claim

We claim:

1. A computer-implemented method comprising:
receiving, by a user interface manager of a lens blur rendering system, a request to apply a lens blur effect to an image, the request identifying an input image and a first disparity map;
generating, by a disparity map pre-processing manager of the lens blur rendering system, a plurality of disparity maps and a plurality of distance maps based on the first disparity map;
splatting, by a kernel splatting manager of the lens blur rendering system, influences of pixels of the input image using a plurality of reshaped kernel gradients;
gathering, by the kernel splatting manager of the lens blur rendering system, aggregations of the splatted influences; and
determining, by the kernel splatting manager of the lens blur rendering system, a lens blur for a first pixel of the input image in an output image based on the gathered aggregations of the splatted influences.

2. The computer-implemented method of claim 1 , wherein determining, by the kernel splatting manager of the lens blur rendering system, a lens blur for a first pixel of the input image in an output image based on the gathered aggregations of the splatted influences, further comprises:
solving a linear equation system constructed from the gathered aggregations of the splatted influences.

3. The computer-implemented method of claim 2 , wherein solving a linear equation system constructed from the gathered aggregations of the splatted influences, further comprises:
determining a first color splatting influence excluding occluded content;
determining a second color splatting influence that restrains the influence inside object boundaries;
determining a first count splatting influence excluding occluded content; and
determining a second count splatting influence that restrains the influence inside object boundaries.

4. The computer-implemented method of claim 1 , wherein generating, by a disparity map pre-processing manager of the lens blur rendering system, a plurality of disparity maps and a plurality of distance maps based on the first disparity map, further comprises:
generating a disparity alpha map and a solid disparity map based on the first disparity map; and
generating a gradient disparity map, a disparity edge map, and a local min disparity map based on the solid disparity map.

5. The computer-implemented method of claim 4 , further comprising:
generating the plurality of distance maps including directional distance maps and a Euclidean distance map using the disparity edge map, wherein each pixel of a directional distance map indicates a distance to a nearest edge in a direction associated with the distance map and each pixel of the Euclidean distance map indicates a Euclidean distance to a nearest edge, wherein the directional distance maps include an up distance map, a down distance, a left distance map, and a right distance map.

6. The computer-implemented method of claim 5 , further comprising:
generating, by the kernel splatting manager of the lens blur rendering system, the reshaped kernel gradients based on a plurality of splatting strategies by:
identifying, for at least one pixel of at least one of the kernel gradients, a nearest qualified edge corresponding to at least one of the plurality of splatting strategies inside the kernel gradient using the plurality of distance maps; and
moving the at least one pixel of the at least one of the kernel gradients beyond the nearest qualified edge based on the distances indicated in the plurality of distance maps.

7. The computer-implemented method of claim 1 , further comprising:
generating, by a highlight manager, a highlight map of the input image, the highlight map including a highlight weight for each pixel of the input image indicating that pixel's relative intensity in the input image.

8. The computer-implemented method of claim 1 , wherein gathering, by the kernel splatting manager of the lens blur rendering system, aggregations of the splatted influences, further comprises:
integrating over gradient fields representing the splatted influences of the pixels of the input image.

9. A system comprising:
a computing device implementing a lens blur rendering system, the lens blur rendering system comprising:
a user interface manager to receive a request to apply a lens blur effect to an image, the request identifying an input image and a first disparity map;
a disparity map pre-processing manager to generate a plurality of disparity maps based on the first disparity map and a plurality of distance maps based on the plurality of disparity maps; and
a kernel splatting manager to:
splat influences of pixels of the input image using a plurality of reshaped kernel gradients;
gather aggregations of the splatted influences; and
determine a lens blur for a first pixel of the input image in an output image based on the gathered aggregations of the splatted influences.

10. The system of claim 9 , wherein to determine a lens blur for a first pixel of the input image in an output image based on the gathered aggregations of the splatted influences, the kernel splatting manager is further to:
solve a linear equation system constructed from the gathered aggregations of the splatted influences.

11. The system of claim 10 , wherein to solve a linear equation system constructed from the gathered aggregations of the splatted influences, the kernel splatting manager is further to:
determine a first color splatting influence excluding occluded content;
determine a second color splatting influence that restrains the influence inside object boundaries;
determine a first count splatting influence excluding occluded content; and
determine a second count splatting influence that restrains the influence inside object boundaries.

12. The system of claim 9 , wherein to generate a plurality of disparity maps based on the disparity map and a plurality of distance maps based on the plurality of disparity maps, the disparity map pre-processing manager is further to:
generate a disparity alpha map and a solid disparity map based on the disparity map; and
generate a gradient disparity map, a disparity edge map, and a local min disparity map based on the solid disparity map.

13. The system of claim 12 , wherein the disparity map pre-processing manager is further to:
generate the plurality of distance maps including directional distance maps and an Euclidean distance map using the disparity edge map, wherein each pixel of a directional distance map indicates a distance to a nearest edge in a direction associated with the distance map and each pixel of the Euclidean distance map indicates a Euclidean distance to a nearest edge, wherein the directional distance maps include an up distance map, a down distance, a left distance map, and a right distance map.

14. The system of claim 13 , wherein the kernel splatting manager is further to:
generate the reshaped kernel gradients based on a plurality of splatting strategies by:
identifying, for at least one pixel of at least one of the kernel gradients, a nearest qualified edge corresponding to at least one of the plurality of splatting strategies inside the kernel gradient using the plurality of distance maps; and
moving the at least one pixel of the at least one of the kernel gradients beyond the nearest qualified edge based on the distances indicated in the plurality of distance maps.

15. The system of claim 9 , wherein the lens blur rendering system further comprises:
a highlight manager to generate a highlight map of the input image, the highlight map including a highlight weight

Assignees

• Adobe Inc

Inventors

• Lin Haiting
• Jia Yumin
• Chien Jen-Chan

Classifications

• G06T5/20

  using local operators · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T7/50

  Depth or shape recovery · CPC title