Processing image data from multiple cameras for motion pictures

Patent metadata

Field	Value
Publication number	US-9094675-B2
Application number	US-201213423058-A
Country	US
Kind code	B2
Filing date	Mar 16, 2012
Priority date	Feb 29, 2008
Publication date	Jul 28, 2015
Grant date	Jul 28, 2015

Abstract

An approach is disclosed for generating stereoscopic image sequences. The approach includes receiving a plurality of image sequences, each image sequence captured from a distinct camera or other capture device in a multi-camera array. One image sequence is a primary image sequence and the other image sequences are secondary image sequences. The approach further includes generating, from the primary and at least one or more of the secondary image sequences, depth maps corresponding to images of the primary image sequence and generating, for each image in the primary image sequence, based on at least the depth maps and the corresponding images of the primary image sequence, stereoscopic images of a stereoscopic image sequence.

First claim

What is claimed is:

1. A computer-implemented method for processing image data, comprising:
receiving a plurality of image sequences, each image sequence captured from a distinct physical camera in a physical multi-camera array, wherein the image sequences include a primary image sequence captured by a primary camera in the multi-camera array and one or more secondary image sequences captured by respective secondary cameras in the multi-camera array, wherein the primary camera captures live-action scenes with a user-specified composition and focus, and wherein each of the secondary cameras captures volumetric information corresponding to the live-action scenes from a different vantage point;
generating, from at least the primary image sequence and one or more of the secondary image sequences, a respective depth map for each of a plurality of images of the primary image sequence;
automatically determining, based on at least one of the depth maps and corresponding image or images of the primary image sequence, an object depicted in the primary image sequence;
automatically separating the determined object onto a distinct layer associated with at least one of the depth maps and a distinct layer or layers associated with a corresponding image or images;
modifying an appearance of the object in at least one of the distinct layer associated with the at least one of the depth maps and the distinct layer or layers associated with the corresponding image or images so as to change an appearance of the object in stereoscopic images; and
generating, for each image in the plurality of images of the primary image sequence, based on at least the image and the corresponding depth map, at least two images of a stereoscopic image sequence.

2. The method of claim 1 , wherein generating the stereoscopic images includes filling a hole in at least a first stereoscopic image using image data from one or more images of the secondary image sequences, wherein the hole corresponds to an area in the first stereoscopic image for which image data of a scene captured in the corresponding image of the primary image sequence is unavailable.

3. The method of claim 1 , wherein at least one of the secondary cameras provides a plenoptic lens configured to capture light field data.

4. The method of claim 1 , wherein the at least two images include a left-eye image and a right-eye image, and wherein an appearance of the objects in the left-eye image and an appearance of the objects in the right-eye image are displaced from their respective positions in the corresponding image of the primary image sequence.

5. The method of claim 1 , further comprising, prior to generating the depth maps:
receiving a set of images of the primary image sequence and one or more editing operations to be performed on the set of images; and
performing the one or more editing operations on the set of images.

6. The method of claim 1 , further comprising, changing a viewpoint of a viewer of the stereoscopic image sequence based on at least the depth maps and the corresponding images of the primary image sequence.

7. The method of claim 1 , further comprising, applying at least one graphical effect to one or more images of the primary image sequence based on data from one or more images of the secondary image sequences, wherein the graphical effect comprises at least one of enhancing a resolution, altering a zoom or a lighting, removing a defect, or increasing a frame rate of the stereoscopic image sequence.

8. The method of claim 1 , wherein the generating the at least two images of the stereoscopic image sequence comprises generating an auto-stereoscopic image sequence.

9. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs an operation for processing image data, the operation comprising:
receiving a plurality of image sequences, each image sequence captured from a distinct physical camera in a physical multi-camera array, wherein the image sequences include a primary image sequence captured by a primary camera in the multi-camera array and one or more secondary image sequences captured by respective secondary cameras in the multi-camera array, wherein the primary camera captures live-action scenes with a user-specified composition and focus, and wherein each of the secondary cameras captures volumetric information corresponding to the live-action scenes from a different vantage point;
generating, from at least the primary image sequence and one or more of the secondary image sequences, a respective depth map for each of a plurality of images of the primary image sequence;
automatically determining, based on at least one of the depth maps and corresponding image or images of the primary image sequence, an object depicted in the primary image sequence;
automatically separating the determined object onto a distinct layer associated with at least one of the depth maps and a distinct layer or layers associated with a corresponding image or images;
modifying an appearance of the object in at least one of the distinct layer associated with the at least one of the depth maps and the distinct layer or layers associated with the corresponding image or images so as to change an appearance of the object in stereoscopic images; and
generating, for each image in the plurality of images of the primary image sequence, based on at least the image and the corresponding depth map, at least two images of a stereoscopic image sequence.

10. The computer-readable storage medium of claim 9 , wherein the operation further comprises, filling a hole in at least one of the images of the stereoscopic image sequence using image data from one or more images of the secondary image sequences, wherein the hole corresponds to an area in the one of the images of the stereoscopic image sequence for which image data of a scene captured in the corresponding image of the primary image sequence is unavailable.

11. The computer-readable storage medium of claim 9 , wherein at least one of the secondary cameras provides a plenoptic lens configured to capture light field data.

12. The computer-readable storage medium of claim 9 , wherein the at least two images include a left-eye image and a right-eye image, and wherein an appearance of the objects in the left-eye image and an appearance of the objects in the right-eye image are displaced from their respective positions in the corresponding image of the primary image sequence.

13. The computer-readable storage medium of claim 9 , wherein the operation further comprises, prior to generating the depth maps:
receiving a set of images of the primary image sequence and one or more editing operations to be performed on the set of images; and
performing the one or more editing operations on the set of images.

14. The computer-readable storage medium of claim 9 , wherein the operation further comprises, changing a viewpoint of a viewer of the stereoscopic image sequence based on at least the depth maps and the corresponding images of the primary image sequence.

15. The computer-readable storage medium of claim 9 , wherein the operation further comprises, applying at least one graphical effect to an image of the primary image sequence based on data from one or more images of the secondary image sequences, wherein the graphical effect comprises at least one of enhancing a resolution, altering a zoom or a lighting, removing a defect, or increasing a frame rate of the stereoscopic image sequence based on data from one or more images of the secondary image sequences.

16. A system, comprising:
a pro

Assignees

• Lukk Howard
• Hendrickson Andrew
• Disney Entpr Inc

Inventors

• Lukk Howard
• Hendrickson Andrew

Classifications

• H04N13/25 Primary

  using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title