Apparatus and method for stitching panoramaic video

Patent metadata

Field	Value
Publication number	US-2016286138-A1
Application number	US-201615081144-A
Country	US
Kind code	A1
Filing date	Mar 25, 2016
Priority date	Mar 27, 2015
Publication date	Sep 29, 2016
Grant date	—

Abstract

An apparatus and method for stitching a panoramic video. The method includes acquiring a plurality of video streams from a plurality of cameras, selecting a reference image frame set, which is a set of still image frames captured at a first point in time, from the plurality of video streams, calculating stitching parameters including a camera parameter and a color correction coefficient based on correspondence relationships between feature points extracted from the reference image frame set, and generating a panoramic video by applying the stitching parameters to another image frame set, which is a set of still image frames captured at a second point in time.

First claim

What is claimed is:

1 . A method of stitching a panoramic video, the method comprising:
acquiring a plurality of video streams from a plurality of cameras, selecting a reference image frame set, which is a set of still image frames captured at a first point in time, from the plurality of video streams; calculating stitching parameters including a camera parameter and a color correction coefficient based on correspondence relationships between feature points extracted from the reference image frame set; and generating a panoramic video by applying the stitching parameters to another image frame set, which is a set of still image frames captured at a second point in time.

2 . The method of claim 1 , wherein the selecting of the reference image frame set comprises:
generating point in time-specific image frame sets by combining still image frames captured at the same time in the plurality of video streams, extracting feature points from still image frames constituting each of the image frame sets; and selecting an image frame set having a largest number of extracted feature points as the reference image frame set.

3 . The method of claim 1 , wherein the selecting of the reference image frame set comprises:
generating point in time-specific image frame sets by combining still image frames captured at the same time in the plurality of video streams; extracting feature points from still image frames constituting each of the image frame sets; selecting image frame sets whose extracted feature points number a previously set minimum feature point number or more; and selecting an image frame set having a largest number of extracted feature points from among the selected image frame sets as the reference image frame set.

4 . The method of claim 1 , wherein the calculating of the stitching parameters comprises:
extracting the feature points from the respective still image frames constituting the reference image frame set; matching feature points between the still image frames and calculating correspondence relationships; calculating the camera parameter based on the correspondence relationships; and calculating the color correction coefficient for equalizing colors of regions overlapping when the still image frames are matched based on the camera parameter.

5 . The method of claim 4 , wherein the calculating of the camera parameter comprises:
selecting a group of camera parameter candidates resulting in least square errors from at least three feature correspondence points; and applying the camera parameter candidate group to another feature correspondence point and selecting a camera parameter resulting in a least square error in the camera parameter candidate group.

6 . The method of claim 1 , wherein the generating of the panoramic video comprises:
converting the still image frames in an x-y coordinate system into converted images in a panoramic image coordinate system using the camera parameter; and performing color correction by applying the color correction coefficient to the converted images, and then combining the still image frames by summing weights of overlapping regions between the converted images.

7 . The method of claim 1 , further comprising updating the stitching parameters.

8 . The method of claim 7 , wherein the updating of the stitching parameters comprises:
generating an update signal at predetermined periods; extracting feature points from respective still image frames constituting an image frame set corresponding to a point in time at which the update signal is generated; matching feature points between the still image frames and calculating correspondence relationships; calculating a camera parameter based on the correspondence relationships; and calculating the color correction coefficient for equalizing colors of regions overlapping when the still image frames are matched based on the camera parameter.

9 . The method of claim 7 , wherein the updating of the stitching parameters comprises:
calculating a movement between a previous still image frame (t−1) and a current still image frame (t) in each of the plurality of video streams; when a size of a first movement between a previous still image frame (t−1) and a current still image frame (t) in a first video stream is larger than a previously set first threshold, calculating a second movement between a previous still image frame (t−1) and a current still image frame (t) in a second video stream; and when a difference in size between the first movement and the second movement is larger than a previously set second threshold, generating an update signal.

10 . The method of claim 9 , wherein the updating of the stitching parameters further comprises generating the update signal when the difference in size between the first movement and the second movement is smaller than the previously set second threshold and a smaller one of sizes of the first movement and the second movement is larger than a previously set third threshold.

11 . An apparatus for stitching a panoramic video, the apparatus comprising:
a video acquisition unit configured to acquire a plurality of video streams from a plurality of cameras; a reference image frame set selection unit configured to select a reference image frame set, which is a set of still image frames captured at a first point in time, from the plurality of video streams; a stitching parameter calculation unit configured to calculate stitching parameters including a camera parameter and a color correction coefficient based on correspondence relationships between feature points extracted from the reference image frame set; and a panoramic video generation unit configured to generate a panoramic video by applying the stitching parameters to another image frame set, which is a set of still image frames captured at a second point in time.

12 . The apparatus of claim 11 , wherein the reference image frame set selection unit generates point in time-specific image frame sets by combining still image frames captured at the same time in the plurality of video streams, extracts feature points from still image frames constituting each of the image frame sets, and selects an image frame set having a largest number of extracted feature points as the reference image frame set.

13 . The apparatus of claim 11 , wherein the reference image frame set selection unit generates point in time-specific image frame sets by combining still image frames captured at the same time in the plurality of video streams, extracts feature points from still image frames constituting each of the image frame sets, selects image frame sets whose extracted feature points number a previously set minimum feature point number or more, and selects an image frame set having a largest number of extracted feature points from among the selected image frame sets as the reference image frame set.

14 . The apparatus of claim 11 , wherein the stitching parameter calculation unit extracts the feature points from the respective still image frames constituting the reference image frame set, matches feature points between the still image frames to calculate correspondence relationships, calculates the camera parameter based on the correspondence relationships, and calculates the color correction coefficient for equalizing colors of regions overlapping when the still image frames are matched based on the camera parameter.

15 . The apparatus of claim 14 , wherein the stitching parameter calculation unit selects a group of camera parameter candidates resulting in least square errors from at least three feature corresp

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Kim Yong Sun

Classifications

• H04N1/3876 Primary

  Recombination of partial images to recreate the original image · CPC title

• H04N1/6027

  Correction or control of colour gradation or colour contrast (H04N1/6058 takes precedence) · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06K9/6201

  Physics · mapped topic