Multiview video stabilization

Patent metadata

Field	Value
Publication number	US-10477181-B2
Application number	US-201514971837-A
Country	US
Kind code	B2
Filing date	Dec 16, 2015
Priority date	Dec 16, 2015
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

Techniques related to multiview video stabilization are discussed. Such techniques may include performing single view video stabilization on a reference video stream and propagating the single view video stabilization to another video stream based on a target disparity map.

First claim

What is claimed is:

1. A computer implemented method for providing multiview video stabilization comprising:
attaining input multiview video comprising a plurality of synchronized input video streams from a corresponding plurality of cameras;
selecting a reference view of the synchronized input video streams, wherein the reference view corresponds to an observed view of a viewer;
performing single view video stabilization, to compensate for video jitter, on a reference video stream, corresponding to the reference view, of the plurality of synchronized input video streams to generate a stabilized reference video stream, wherein performing the single view video stabilization comprises applying a stabilization transform to a frame of the reference video stream to generate a stabilized frame of the stabilized reference video stream;
detecting a plurality of locations of stabilized feature points from the stabilized frame;
propagating the single view video stabilization to a second video stream of the plurality of synchronized input video streams by:
generating a target disparity map by applying the stabilization transform to an initial disparity map for the frame of the reference video stream and a synchronized frame of the second video stream and by minimizing at least a vertical component or a horizontal component of the target disparity map, the target disparity map comprising a plurality of disparity vectors;
determining locations for synchronized stabilized feature points of a synchronized stabilized frame of the second video stream corresponding to the stabilized frame by determining the locations for the synchronized stabilized feature points that minimize an energy term comprising a plurality of differences between the locations of the synchronized stabilized feature points and sums of corresponding locations of the stabilized feature points of the stabilized frames and corresponding disparity vectors of the target disparity map; and
performing video stabilization on the second video stream by warping the synchronized frame using the synchronized stabilized feature points to generate a synchronized stabilized frame of a stabilized second video stream; and
presenting stabilized multiview video comprising at least the stabilized second video stream to the viewer.

2. The method of claim 1 , wherein minimizing the vertical or horizontal component comprises setting the vertical or horizontal component for each position of the target disparity map to zero.

3. The method of claim 2 , wherein the vertical component is set to zero when the reference video stream and the second video stream are horizontally aligned and the horizontal component is set to zero when the reference video stream and the second video stream are vertically aligned.

4. The method of claim 1 , further comprising: partitioning the plurality of synchronized input video streams into a plurality of subsets of synchronized streams of multiview video, wherein the reference video stream is in a first subset and the second video stream is in a second subset; and propagating the single view video stabilization to a third video stream in a third subset of the plurality of subsets of synchronized streams.

5. The method of claim 4 , wherein propagating the single view video stabilization to the second video stream and the third video stream is performed at least partially in parallel.

6. The method of claim 4 , further comprising: propagating video stabilization from the second video stream to one or more other video streams in the second subset; and propagating video stabilization from the third video stream to one or more other video streams in the third subset at least partially in parallel to the propagation of the video stabilization from the second video stream to the one or more other video streams in the second subset.

7. The method of claim 1 , further comprising: propagating the single view video stabilization to a third video stream by: generating a second target disparity map by applying the stabilization transform to a second initial disparity map for the frame of the reference video and a synchronized second frame of the third video stream and by minimizing a vertical component of the second target disparity map, wherein the horizontal component is minimized in the target disparity map; and performing video stabilization on the third video stream based at least in part on the second target disparity map to generate a stabilized third video stream.

8. The method of claim 1 , further comprising: propagating the single view video stabilization to a third video stream by: generating a second target disparity map by applying the stabilization transform to a second initial disparity map for the frame of the reference video and a synchronized frame of a fourth video stream and by setting a vertical component of the second target disparity map based on the stabilization transformed second disparity map and a horizontal component of the second target disparity map based on the stabilization transformed disparity map, wherein the s econd video stream is horizontally aligned with the reference video stream, the fourth video stream is vertically aligned with the reference video stream, and the third video stream is diagonally aligned with the reference video stream; and performing video stabilization on the third video stream based at least in part on the second target disparity map to generate a stabilized third video stream.

9. The method of claim 1 , further comprising: determining the reference view corresponds to the viewer; and setting the reference video stream based on the reference view corresponding to the viewer.

10. The method of claim 9 , wherein determining the reference view corresponds to the viewer comprises at least one of head tracking, eye tracking, or gaze tracking.

11. A system for providing multiview video stabilization comprising:
a memory configured to store multiview video comprising a plurality of synchronized input video streams from a corresponding plurality of cameras; and
a processor coupled to the memory, the processor to:
attain the multiview video;
select a reference view of the synchronized input video streams, wherein the reference view corresponds to an observed view of a viewer;
perform single view video stabilization, to compensate for video jitter, on a reference video stream, corresponding to the reference view, of the plurality of synchronized input video streams to generate a stabilized reference video stream, wherein to perform the single view video stabilization comprises the processor to apply a stabilization transform to a frame of the reference video stream to generate a stabilized frame of the stabilized reference video stream;
detect a plurality of locations of stabilized feature points from the stabilized frame;
propagate the single view video stabilization to a second video stream of the plurality of synchronized input video streams, wherein the processor to propagate the single view video stabilization comprises the processor to generate a target disparity map based on application of the stabilization transform to an initial disparity map for the frame of the reference video stream and a synchronized frame of the second video stream and minimization of at least a vertical component or a horizontal component of the target disparity map, the target disparity map comprising a plurality of disparity vectors, to determine locations for synchronized stabilized feature points of a synchronized stabilized frame of the second video stream corresponding to the stabilized frame by determining the locations for the synchronized stabilized feature points that minimize an energy term comprising a plura

Assignees

• Intel Corp

Inventors

• El Choubassi Maha
• Zhang Fan
• Ratcliff Joshua J
• Nestares Oscar

Classifications

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title