Motion blur for light-field images

Patent metadata

Field	Value
Publication number	US-9900510-B1
Application number	US-201615372968-A
Country	US
Kind code	B1
Filing date	Dec 8, 2016
Priority date	Dec 8, 2016
Publication date	Feb 20, 2018
Grant date	Feb 20, 2018

Abstract

Motion blur may be applied to a light-field image. The light-field image may be captured with a light-field camera having a main lens, an image sensor, and a plurality of microlenses positioned between the main lens and the image sensor. The light-field image may have a plurality of lenslet images, each of which corresponds to one microlens of the microlens array. The light-field image may be used to generate a mosaic of subaperture images, each of which has pixels from the same location on each of the lenslet images. Motion vectors may be computed to indicate motion occurring within at least a primary subaperture image of the mosaic. The motion vectors may be used to carry out shutter reconstruction of the mosaic to generate a mosaic of blurred subaperture images, which may then be used to generate a motion-blurred light-field image.

First claim

What is claimed is:

1. A method for blurring a light-field image, the method comprising:
in a data store, receiving a light-field image that has been captured with a light-field camera comprising an aperture, an image sensor, and a microlens array positioned between the aperture and the image sensor, wherein the light-field image comprises a plurality of lenslet images, each of which corresponds to one microlens of the microlens array;
in a processor, using the light-field image to generate a mosaic of subaperture images, each of which comprises pixels from the same location on each of the lenslet images;
in the processor, computing motion vectors indicative of motion of at least a primary subaperture image of the mosaic of subaperture images;
in the processor, using the motion vectors to carry out shutter reconstruction of the mosaic of subaperture images to generate a blur-ready mosaic of subaperture images;
in the processor, using the blur-ready mosaic of subaperture images to generate a motion-blurred light-field image; and
outputting the motion-blurred light-field image.

2. The method of claim 1 , further comprising, prior to receiving the light-field image, capturing the light-field image in the light-field camera.

3. The method of claim 1 , wherein:
the primary subaperture image comprises pixels from a primary location on each of the lenslet images; and
computing the motion vectors comprises calculating a temporal disparity between the primary subaperture image and a plurality of proximate subaperture images, each of which comprises pixels from a proximate location on each of the lenslet images that is proximate the primary location.

4. The method of claim 3 , wherein computing the motion vectors further comprises calculating additional disparities between additional primary subaperture images of the mosaic of subaperture images, and additional subaperture images of the mosaic of subaperture images.

5. The method of claim 3 , wherein the proximate subaperture images are selected such that the proximate locations are positioned on opposite sides of the primary location on each of the lenslet images.

6. The method of claim 5 , wherein the proximate subaperture images are further selected such that the proximate locations define a ring that encircles the primary location on each of the lenslet images.

7. The method of claim 1 wherein computing the motion vectors further comprises applying an optical flow to at least the primary subaperture image.

8. The method of claim 1 , wherein the motion vectors comprise a full four-dimensional motion vector set defining a three-dimensional flow of objects represented in the light-field image.

9. The method of claim 8 , wherein the motion vectors comprise forward motion vectors indicative of motion of the objects in time subsequent to capture of the light-field image, and backward motion vectors indicative of motion of the objects in time prior to capture of the light-field image.

10. The method of claim 1 , wherein using the motion vectors to carry out shutter reconstruction of the mosaic of subaperture images comprises applying back projection to the blur-ready mosaic of subaperture images to generate a four-dimensional light-field.

11. The method of claim 1 , wherein using the blur-ready mosaic of subaperture images to generate the motion-blurred light-field image comprises transforming the blur-ready mosaic of subaperture images back to a lenslet domain, with motion data embedded for each microlens.

12. A non-transitory computer-readable medium for blurring a light-field image, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:
causing a data store to receive a light-field image that has been captured with a light-field camera comprising an aperture, an image sensor, and a microlens array positioned between the aperture and the image sensor, wherein the light-field image comprises a plurality of lenslet images, each of which corresponds to one microlens of the microlens array;
using the light-field image to generate a mosaic of subaperture images, each of which comprises pixels from the same location on each of the lenslet images;
computing motion vectors indicative of motion of at least a primary subaperture image of the mosaic of subaperture images;
using the motion vectors to carry out shutter reconstruction of the mosaic of subaperture images to generate a blur-ready mosaic of subaperture images;
using the blur-ready mosaic of subaperture images to generate a motion-blurred light-field image; and
causing an output device to output the motion-blurred light-field image.

13. The non-transitory computer-readable medium of claim 12 , wherein:
the primary subaperture image comprises pixels from a primary location on each of the lenslet images; and
computing the motion vectors comprises calculating a temporal disparity between the primary subaperture image and a plurality of proximate subaperture images, each of which comprises pixels from a proximate location on each of the lenslet images that is proximate the primary location.

14. The non-transitory computer-readable medium of claim 13 , wherein computing the motion vectors further comprises calculating additional disparities between additional primary subaperture images of the mosaic of subaperture images, and additional subaperture images of the mosaic of subaperture images.

15. The non-transitory computer-readable medium of claim 13 , wherein the proximate subaperture images are selected such that the proximate locations are positioned on opposite sides of the primary location on each of the lenslet images and define a ring that encircles the primary location on each of the lenslet images.

16. The non-transitory computer-readable medium of claim 12 , wherein the motion vectors comprise a full four-dimensional motion vector set defining a three-dimensional flow of objects represented in the light-field image.

17. The non-transitory computer-readable medium of claim 16 , wherein the motion vectors comprise forward motion vectors indicative of motion of the objects in time subsequent to capture of the light-field image, and backward motion vectors indicative of motion of the objects in time prior to capture of the light-field image.

18. The non-transitory computer-readable medium of claim 12 , wherein using the motion vectors to carry out shutter reconstruction of the mosaic of subaperture images comprises applying back projection to the blur-ready mosaic of subaperture images to generate a four-dimensional light-field.

19. The non-transitory computer-readable medium of claim 12 , wherein using the blur-ready mosaic of subaperture images to generate the motion-blurred light-field image comprises transforming the blur-ready mosaic of subaperture images back to a lenslet domain, with motion data embedded for each microlens.

20. A system for blurring a light-field image, the system comprising:
a data store configured to receive a light-field image that has been captured with a light-field camera comprising an aperture, an image sensor, and a microlens array positioned between the aperture and the image sensor, wherein the light-field image comprises a plurality of lenslet images, each of which corresponds to one microlens of the microlens array;
a processor, communicatively coupled to the data store, configured to:
use the light-field image to generate a mosaic of subaperture images, each of which comprises pixels from the same location on each of the lensle

Assignees

• Lytro Inc

Inventors

• Karafin Jon
• Nonn Thomas
• Pan Gang
• Wang Zejing

Classifications

• G06T2200/21

  involving computational photography · CPC title

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title