Image refocusing for camera arrays

Patent metadata

Field	Value
Publication number	US-9712720-B2
Application number	US-201414293611-A
Country	US
Kind code	B2
Filing date	Jun 2, 2014
Priority date	Jun 2, 2014
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

Techniques related to image refocusing for camera arrays using a space variant filter based on a pixel by pixel blur level associated with a difference between a reference image and a basic refocused image are discussed. Such techniques may include taking the difference between the reference image and the basic refocused image and applying a space variant filter to the basic refocused image to generate a final refocused image having a region of increased blur with respect to the all in focus reference image. The blur level implemented by the space variant filter may be based on the difference and, optionally, the reference image noise level and the reference image gradient.

First claim

What is claimed is:

1. A computer-implemented method for providing image refocusing comprising:
determining an all in focus reference image and generating a basic refocused image based on a plurality of images received via a camera array;
taking a difference of the all in focus reference image and the basic refocused image to generate a difference image; and
applying a space variant filter based at least in part on the difference image to the basic refocused image to generate a final refocused image having a region of increased blur with respect to the all in focus reference image, wherein applying the space variant filter based on the difference image comprises increasing blur by a greater amount in a first region than in a second region of the basic refocused image, wherein the first region is associated with a first difference in the difference image and the second region is associated with a second difference in the difference image, and wherein the first difference is greater than the second difference.

2. The method of claim 1 , further comprising:
applying a low pass filter to the difference image to generate a low pass difference image, wherein applying the space variant filter based at least in part on the difference image comprises applying the space variant filter based on the low pass difference image.

3. The method of claim 1 , wherein applying the space variant filter comprises applying the space variant filter based on a Laplacian pyramid approximation.

4. The method of claim 3 , wherein applying the space variant filter based on the Laplacian pyramid approximation comprises:
generating a basic refocused image Laplacian pyramid based on the basic refocused image;
generating a mask pyramid based on the difference image;
determining a product of the mask pyramid and the basic refocused image Laplacian pyramid; and
reconstructing the product to generate the final refocused image.

5. The method of claim 4 , further comprising:
applying a low pass filter to the difference image to generate a low pass difference image;
generating a noise level mask and a gradient mask based on the all in focus reference image; and
applying the noise level mask and the gradient mask to the low pass difference image to generate a processed low pass difference image having reduced noise and gradient effects, wherein the mask pyramid is generated based on the processed low pass difference image.

6. The method of claim 4 , wherein generating the mask pyramid comprises applying a kernel with a circularly symmetric frequency response.

7. The method of claim 1 , wherein the space variant filter comprises at least one of a noise level mask or a gradient mask.

8. The method of claim 1 , wherein taking the difference of the all in focus reference image and the basic refocused image comprises taking the difference of grayscale versions of the all in focus reference image and the basic refocused image.

9. The method of claim 1 , wherein applying the space variant filter to the basic refocused image comprises applying the space variant filter to each color channel of the basic refocused image independently.

10. The method of claim 1 , wherein the all in focus reference image comprises one of the plurality of images, and wherein determining the basic refocused image comprises shifting the remaining plurality of images based on a displacement associated with a desired depth and averaging the all in focus reference image and the shifted images to generate the basic refocused image.

11. A system for providing image refocusing on a computing device, comprising:
a memory configured to store image data;
a central processing unit coupled to the memory, wherein the central processing unit comprises:
reference image and basic refocused image determination circuitry configured to determine an all in focus reference image and generate a basic refocused image based on a plurality of images received via a camera array; and

a graphics processing unit coupled to the memory, wherein the graphics processing unit comprises:
differencer circuitry configured to take a difference of the all in focus reference image and the basic refocused image to generate a difference image; and
space variant filter circuitry configured to apply a space variant filter based at least in part on the difference image to the basic refocused image to generate a final refocused image having a region of increased blur with respect to the all in focus reference image, wherein to apply the space variant filter based on the difference image comprises the space variant filter circuitry to increase blur by a greater amount in a first region than in a second region of the basic refocused image, wherein the first region is associated with a first difference in the difference image and the second region is associated with a second difference in the difference image, and wherein the first difference is greater than the second difference.

12. The system of claim 11 , wherein the graphics processing unit further comprises:
low pass filter circuitry configured to apply a low pass filter to the difference image to generate a low pass difference image, wherein the space variant filter circuitry being configured to apply the space variant filter based at least in part on the difference image comprises the space variant filter circuitry configured to apply the space variant filter based on the low pass difference image.

13. The system of claim 11 , wherein the space variant filter circuitry is configured to apply the space variant filter based on a Laplacian pyramid approximation.

14. The system of claim 13 , wherein the graphics processing unit further comprises:
pyramid circuitry configured to generate a basic refocused image Laplacian pyramid based on the basic refocused image and a weighting Gaussian pyramid based on the difference image;
product circuitry configured to determine a product of the weighting Gaussian pyramid and the basic refocused image Laplacian pyramid; and
pyramid reconstruction circuitry configured to reconstruct the product to generate the final refocused image.

15. The system of claim 14 , wherein the pyramid circuitry is configured to generate the weighting Gaussian pyramid based on a kernel with a circularly symmetric frequency response.

16. The system of claim 11 , wherein the differencer circuitry is configured to take the difference of the all in focus reference image and the basic refocused image based on grayscale versions of the all in focus reference image and the basic refocused image, and wherein the space variant filter circuitry is configured to apply the space variant filter to each color channel of the basic refocused image independently.

17. The system of claim 11 , further comprising:
a display configured to display the final refocused image in real time to a user.

18. At least one non-transitory machine readable medium comprising a plurality of instructions that in response to being executed on a computing device, cause the computing device to provide image refocusing by:
determining an all in focus reference image and generating a basic refocused image based on a plurality of images received via a camera array;
taking a difference of the all in focus reference image and the basic refocused image to generate a difference image; and
applying a space variant filter based at least in part on the difference image to the basic refocused image to generate a final refocused image having a region of increased blur with respect to the all in focus reference image, wherein app

Assignees

• Intel Corp

Inventors

• Nestares Oscar
• Jiang Yong

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T5/20 Primary

  using local operators · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• H04N1/409

  Edge or detail enhancement; Noise or error suppression · CPC title