Systems and methods for decoding image files containing depth maps stored as metadata

Patent metadata

Field	Value
Publication number	US-10430682-B2
Application number	US-201816030597-A
Country	US
Kind code	B2
Filing date	Jul 9, 2018
Priority date	Sep 28, 2011
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image, a set of low resolution images, and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map and the set of low resolution images to create a rendered image.

First claim

What is claimed:

1. A system for rendering an image using an image container file, the system comprising:
a processor; and
memory containing a rendering application and an image container file, wherein the image container file comprises:
an encoded image synthesized from a plurality of images captured by a plurality of heterogeneous cameras, wherein each camera of the plurality of heterogeneous cameras captures the scene from a different viewpoint;
a depth map that specifies depths from a reference viewpoint for pixels in the encoded image based on disparity between pixels of the plurality of images;
an auxiliary map that provides information corresponding to pixel locations within a synthesized image; and
metadata describing the image container file, wherein the metadata comprises offset information to locate the encoded image, the depth map, and the auxiliary map;
wherein the rendering application configures the processor to:
locate the encoded image within the image container file;
decode the encoded image;
locate the depth map and the auxiliary map within the image container file; and
post process the decoded image to apply a depth based effect to the pixels of the decoded image based on the depth map, the auxiliary map, and the metadata to create a rendered image.

2. The system of claim 1 , wherein the image container file further comprises at least one of the plurality of images captured by the plurality of heterogeneous cameras.

3. The system of claim 2 , wherein post processing the decoded image comprises interpolating to fill in occluded pixels based on adjacent pixels that are not occluded.

4. The system of claim 1 , wherein the metadata further comprises occluded pixel data for pixels that are occluded.

5. The system of claim 1 , wherein the encoded image is focused at a first focal plane, wherein post processing comprises modifying the pixels so that the rendered image is focused at a different, second focal plane.

6. The system of claim 1 , wherein the depth based effect simulates motion parallax in the decoded image.

7. The system of claim 1 , wherein the depth based effect simulates a dolly zoom in the decoded image.

8. The system of claim 1 , wherein at least one of the plurality of images have a lower resolution than the encoded image.

9. The system of claim 1 , wherein the encoded image is synthesized from a first image from a first viewpoint and a second image from a second viewpoint, wherein the encoded image is synthesized from a different, third viewpoint.

10. The system of claim 1 , wherein at least one of the depth map and the auxiliary map are encoded as an image.

11. A method for rendering an image using an image container file, the method comprising:
locating the encoded image within an image container file, wherein the image container file comprises:
an encoded image synthesized from a plurality of images captured by a plurality of heterogeneous cameras, wherein each camera of the plurality of heterogeneous cameras captures the scene from a different viewpoint;
a depth map that specifies depths from a reference viewpoint for pixels in the encoded image based on disparity between pixels of the plurality of images;
an auxiliary map that provides information corresponding to pixel locations within a synthesized image; and
metadata describing the image container file, wherein the metadata comprises offset information to locate the encoded image, the depth map, and the auxiliary map;

decoding the encoded image;
locating the depth map and the auxiliary map within the image container file; and
post processing the decoded image to apply a depth based effect to the pixels of the decoded image based on the depth map, the auxiliary map, and the metadata to create a rendered image.

12. The method of claim 11 , wherein the image container file further comprises at least one of the plurality of images captured by the plurality of heterogeneous cameras.

13. The method of claim 12 , wherein post processing the decoded image comprises interpolating to fill in occluded pixels based on adjacent pixels that are not occluded.

14. The method of claim 11 , wherein the metadata further comprises occluded pixel data for pixels that are occluded.

15. The method of claim 11 , wherein the encoded image is focused at a first focal plane, wherein post processing comprises modifying the pixels so that the rendered image is focused at a different, second focal plane.

16. The method of claim 11 , wherein the depth based effect simulates motion parallax in the decoded image.

17. The method of claim 11 , wherein the depth based effect simulates a dolly zoom in the decoded image.

18. The method of claim 11 , wherein at least one of the plurality of images have a lower resolution than the encoded image.

19. The method of claim 11 , wherein the encoded image is synthesized from a first image from a first viewpoint and a second image from a second viewpoint, wherein the encoded image is synthesized from a different, third viewpoint.

20. The method of claim 11 , wherein at least one of the depth map and the auxiliary map are encoded as an image.

Assignees

• Fotonation Cayman Ltd
• Fotonation Ltd

Inventors

• Venkataraman Kartik
• Nisenzon Semyon
• Lelescu Dan

Classifications

• H04N19/70

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H04N19/593

  involving spatial prediction techniques · CPC title

• H04N13/178 Primary

  Metadata, e.g. disparity information · CPC title

• H04N13/139

  Format conversion, e.g. of frame-rate or size · CPC title