Systems and method for performing depth based image editing

Patent metadata

Field	Value
Publication number	US-10311649-B2
Application number	US-201715694700-A
Country	US
Kind code	B2
Filing date	Sep 1, 2017
Priority date	Feb 21, 2012
Publication date	Jun 4, 2019
Grant date	Jun 4, 2019

Abstract

Systems and methods for the manipulation of captured light fields and captured light field image data in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system for manipulating captured light field image data includes a processor, a display, a user input device, and a memory, wherein a depth map includes depth information for one or more pixels in the image data, and wherein an image manipulation application configures the processor to display a first synthesized image, receive user input data identifying a region within the first synthesized image, determine boundary data for the identified region using the depth map, receive user input data identifying at least one action, and perform the received action using the boundary data and the captured light field image data.

First claim

What is claimed is:

1. An array camera system for capturing and manipulating captured image data, comprising:
an array camera comprising a plurality of cameras, where each camera includes separate optics, and a plurality of sensor elements, and each camera is configured to independently capture an image of a scene;
a processor;
a display connected to the processor and configured to display images;
a memory connected to the processor;
software connected to the processor and directs the processor to:
capture images;
store the captured images in the memory; and
generate a depth map having depth information associated with the captured images of the scene; and
an image manipulation application within the memory that directs the processor to:
select a collection of pixels within at least one captured image based upon depth information, wherein the collection of pixels is selected by identifying a boundary of the collection of pixels based upon color and intensity values, and the associated depth information, where the boundary is determined by separately clustering each given pixel based upon depth, color, and intensity;
modify the pixels of the selected collection of pixels of the at least one captured image;
copy the modified pixels of the selected collection of pixels; and
paste the modified pixels of the selected collection of pixels into another image.

2. The system of claim 1 , wherein the image manipulation application directs the processor to modify at least one of: brightness, luminosity, color values and transparency values of the pixels of the selected collection of pixels of the at least one captured image.

3. The system of claim 1 , wherein the collection of pixels is an object.

4. The system of claim 1 , wherein the collection of pixels excludes the pixels of an object.

5. The system of claim of claim 1 , wherein the image manipulation application directs the processor to modify a focus of the selected collection of pixels of at least one captured image by:
identifying a focal plane of the selected collection of pixels using a depth of field; and
refocusing the pixels of the selected collection of pixels using the depth of field, the identified focal plane, and blur modification data, where the blur modification data affects bokeh of the selected collection of pixels.

6. The system of claim 5 , wherein the depth of field is determined by at least one of: utilizing the associated depth information of the at least one captured image, setting the depth of field to a default value, and using a user-provided depth of field through a user input device connected to the processor and configured to provide user generated input data.

7. The system of claim 1 , wherein the image manipulation application further directs the processor to:
display the captured image using the display; and
identify the selected collection of pixels within the displayed image.

8. The system of claim 1 further comprising a user input device connected to the processor and configured to provide user generated input data, wherein identifying the collection of pixels within the displayed image comprises receiving user input data from the user input device identifying at least one pixel of the selected collection of pixels within the captured image.

9. The system of claim 1 further comprising an eyeball tracking device, wherein the associated depth of the selected collection of pixels is determined based upon where a user is focusing and dynamically adjusted as the user focuses at different parts of the captured image.

10. The system of claim 1 , further comprising:
a user input device connected to the processor and configured to generate user input data in response to user input;
wherein the image manipulation application further configures the processor to receive user input data.

11. The system of claim 10 , wherein the input device is a touchscreen device.

12. The system of claim 10 , wherein the received input data comprises the selected collection of pixels.

13. The system of claim 10 , wherein the received input data comprises modification of the pixels of the selected collection of pixels.

14. The system of claim 1 , wherein the captured image and associated depth information are stored in memory in an image data file.

15. The system of claim 14 , wherein the image data file also includes at least one auxiliary map selected from a group consisting of a confidence map, an edge map, and a missing pixel map.

16. The system of claim 1 , wherein the image into which the selected collection of pixels having modified pixels is copied is another image of the scene.

17. A method for manipulating captured image data using an array camera having a processor, comprising:
capturing, using the processor directed by software, a plurality of images of a scene;
generating, using the processor directed by the software, a depth map having depth information associated with the captured images of the scene;
selecting, using the processor directed by an image manipulation application, a collection of pixels within at least one captured image, based upon the depth information, wherein the collection of pixels is selected by identifying a boundary of the collection of pixels based upon color and intensity values, and the associated depth information, where the boundary is determined by separately clustering each given pixel based upon depth, color, and intensity;
modifying, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels of the at least one captured image;
copying, using the processor directed by an image manipulation application, the modified pixels of the selected collection of pixels; and
pasting, using the processor directed by an image manipulation application, the modified pixels of the selected collection of pixels into another image.

18. The method of claim 17 wherein modifying, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels comprises modifying at least one of: brightness, luminosity, color values or transparency values of the pixels.

19. The method of claim 17 , wherein the collection of pixels is an object.

20. The method of claim 17 , wherein the collection of pixels excludes an object.

21. The method of claim of claim 17 further comprising:
identifying, using the processor directed by an image manipulation application, a focal plane of the selected collection of pixels using a depth of field; and
refocusing, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels using the depth of field, the identified focal plane, and blur modification data, where the blur modification data affects bokeh of the selected collection of pixels.

22. The method of claim 21 , wherein the depth of field is determined by at least one of: utilizing the associated depth information of the at least one captured image, setting the depth of field to a default value, and using a user-provided depth of field through a user input device connected to the processor and configured to provide user generated input data.

23. The method of claim 17 , further comprising:
displaying, using the processor directed by an image manipulation application, the captured image using the display; and
identifying, using the processor directed by an image manipulation application, the selected collection of pix

Assignees

• Fotonation Ltd

Inventors

• Mcmahon Andrew Kenneth John
• Venkataraman Kartik
• Mullis Robert

Classifications

• G06T2200/21

  involving computational photography · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F3/041

  Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title