Systems and methods for decoding structured light field image files

Patent metadata

Field	Value
Publication number	US-9036931-B2
Application number	US-201414477374-A
Country	US
Kind code	B2
Filing date	Sep 4, 2014
Priority date	Sep 28, 2011
Publication date	May 19, 2015
Grant date	May 19, 2015

Abstract

Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create a rendered image.

First claim

What is claimed:

1. A system for rendering an image using a light field image file, the system comprising:
a processor; and
memory containing a rendering application and a light field image file;
wherein the light field image file is structured using the Exchangeable image file (Exif) format and comprises:
an encoded image; and
metadata describing the encoded image stored within an application marker segment within the light field image file;

wherein the metadata comprises a depth map that specifies depths for pixels in the encoded image; and
wherein the rendering application directs the processor to render an image by applying post processing to the encoded image using metadata contained within the light field image by:
locating the encoded image within the light field image file by locating a start of image marker within the light field image file;
decoding the encoded image;
locating the metadata within the light field image file; and
processing the decoded image by modifying the pixels of the decoded image based on the depths of the pixels indicated within the depth map to create a rendered image by applying a depth based effect to the pixels of the decoded image.

2. The system of claim 1 , wherein the depth based effect comprises at least one effect selected from the group consisting of:
modifying the focal plane of the decoded image;
modifying the depth of field of the decoded image;
modifying the blur in out-of-focus regions of the decoded image;
locally varying the depth of field of the decoded image;
creating multiple focus areas at different depths within the decoded image; and
applying a depth related blur.

3. The system of claim 1 , wherein the encoded image is an image of a scene is synthesized from a reference viewpoint using a plurality of images that capture the scene from different viewpoints.

4. The system of claim 1 , wherein the depth map is generated based on a plurality of images that capture the scene from different viewpoints.

5. The system of claim 1 , wherein:
the metadata in the light field image file further comprises pixels from the plurality of images that are occluded in a reference viewpoint; and
the rendering application configuring the processor to process the decoded image by modifying the pixels based on the depths indicated within the depth map to create the rendered image comprises rendering an image from a different viewpoint using the depth map and the pixels from the plurality of images that are occluded in the reference viewpoint.

6. The system of claim 5 , wherein:
the metadata in the light field image file includes descriptions of the pixels from the plurality of images that are occluded in the reference viewpoint including the color, location, and depth of the occluded pixels; and
rendering an image from a different viewpoint using the depth map and the pixels that are occluded in the reference viewpoint from the plurality of images further comprises:
shifting pixels from the decoded image and the occluded pixels in the metadata to the different viewpoint based upon the depths of the pixels;
determining pixel occlusions; and
generating an image from the different viewpoint using the shifted pixels that are not occluded and by interpolating to fill in missing pixels using adjacent pixels that are not occluded.

7. The system of claim 5 , wherein the image rendered from the different viewpoint is part of a stereo pair of images.

8. The system of claim 1 , wherein the metadata in the light field image file further comprises a confidence map for the depth map, where the confidence map indicates the reliability of the depth values provided for pixels by the depth map.

9. The system of claim 1 , wherein:
the metadata in the light field image file further comprises an edge map that indicates pixels in the decoded image that lie on a discontinuity; and
rendering an image from a different viewpoint using the depth map and the pixels from the plurality of images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the edge map.

10. The system of claim 9 , wherein the edge map identifies whether a pixel lies on an intensity discontinuity.

11. The system of claim 9 , wherein the edge map identifies whether a pixel lies on an intensity and depth discontinuity.

12. The system of claim 9 , wherein the edge map is losslessly encoded.

13. The system of claim 12 , wherein the rendering application directs the processor to:
locate at least one Application marker segment containing the metadata comprising the edge map; and
decode the edge map using the JPEG decoder.

14. The system of claim 9 , wherein the edge map is encoded using lossy compression.

15. The system of claim 1 , wherein:
the metadata in the light field image file further comprises a missing pixel map that indicates pixels in the decoded image that do not correspond to a pixel from the plurality of images of the scene and that are generated by interpolating pixel values from adjacent pixels in the synthesized image; and
rendering an image from a different viewpoint using the depth map and the pixels from the plurality of images that are occluded in the reference viewpoint further comprises ignoring pixels based upon the missing pixel map.

16. The system of claim 15 , wherein the missing pixel map is losslessly encoded.

17. The system of claim 16 , wherein the rendering application directs the processor to:
locate at least one Application marker segment containing the metadata comprising the missing pixel map; and
decode the missing pixel map using the JPEG decoder.

18. The system of claim 15 , wherein the missing pixel map is encoded using lossy compression.

19. The system of claim 1 , wherein the depth map is losslessly encoded.

20. The system of claim 19 , wherein the rendering application directs the processor to decode the depth map using a JPEG decoder.

21. The system of claim 1 , wherein the depth map is encoded using lossy compression.

22. The system of claim 1 , wherein:
the memory further comprises a JPEG decoder application;
the light field image file is encoded according to the JPEG standard; and
the rendering application further directs the processor to decode the encoded image using the JPEG decoder.

23. The system of claim 1 , wherein the application marker segment is identified using an APP1 marker that is used to identify the Exif data.

24. A method for decoding a light field image file, where the light field image file is structured using the Exchangeable image file (Exif) format and comprises an encoded image and metadata describing the encoded image stored within an application marker segment within the light field image file, the method comprising:
locating the encoded image within the light field image file by locating a start of image marker within the light field image file using an image rendering system;
decoding the encoded image using the image rendering system;
locating the metadata within the light field image file using the image rendering system, where the metadata comprises a depth map that specifies depths for pixels in the encoded image; and
processing the decoded image by modifying the pixels of the decoded image based on the depths of the pixels indicated within the depth map to create a rendered image by applying a depth based effect to the pixels of the decoded image using the image rendering system.

Assignees

• Pelican Imaging Corp

Inventors

• Venkataraman Kartik
• Nisenzon Semyon
• Lelescu Dan

Classifications

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H04N19/593

  involving spatial prediction techniques · CPC title

• H04N19/70

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title

• H04N13/139

  Format conversion, e.g. of frame-rate or size · CPC title

• H04N13/178 Primary

  Metadata, e.g. disparity information · CPC title