Panoramic video processing method and device and non-transitory computer-readable medium

Patent metadata

Field	Value
Publication number	US-10455152-B2
Application number	US-201815955858-A
Country	US
Kind code	B2
Filing date	Apr 18, 2018
Priority date	Jun 9, 2017
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

Panoramic video processing method and device are disclosed. The method includes steps of receiving a panoramic video waiting for processing; conducting inter frame motion estimation in regard to any two neighboring 2D panoramic images in the panoramic video so as to obtain an inter frame motion estimation result between the same two neighboring 2D panoramic images; acquiring, based on the respective inter frame motion estimation results, a historical motion trajectory of the panoramic video, and carrying out smoothing with respect to the historical motion trajectory; calculating, based on the historical motion trajectory after smoothing, a rotation matrix for motion compensation pertaining to each 2D panoramic image in the panoramic video; and performing reconstruction on each 2D panoramic image in the panoramic video by means of the corresponding rotation matrix for motion compensation so as to attain a panoramic video after image stabilization, and outputting the panoramic video after image stabilization.

First claim

What is claimed is:

1. A panoramic video processing method comprising:
receiving a panoramic video to be processed, the panoramic video including a plurality of two dimensional (2D) panoramic images;
conducting inter frame motion estimation with respect to each two neighboring 2D panoramic images in the panoramic video so as to obtain a corresponding inter frame motion estimation result between the same two neighboring 2D panoramic images;
calculating, based on the respective inter frame motion estimation results obtained between the plurality of 2D panoramic images, a historical motion trajectory of the panoramic video, and carrying out smoothing with respect to the historical motion trajectory;
calculating, based on the historical motion trajectory after smoothing, a rotation matrix for motion compensation pertaining to each 2D panoramic image in the panoramic video, the calculating including processes of
setting a first 2D panoramic image in the plurality of 2D panoramic images as a first reference 2D panoramic image; and
determining, based on the historical motion trajectory after smoothing, whether a change in a current 2D panoramic image relative to a previous 2D panoramic image is due to unconscious noise motion or conscious camera motion, wherein
in a case where the change is determined as unconscious noise motion, the rotation matrix for motion compensation of the current 2D panoramic image, relative to the previous 2D panoramic image, is calculated, and
in a case where the change is determined as conscious camera motion, the first reference 2D panoramic image is replaced with a second reference 2D panoramic image from a later time than the first 2D panoramic image, and the rotation matrix for motion compensation is then recalculated based on the second reference 2D panoramic image and the current 2D panoramic image; and

performing reconstruction on each 2D panoramic image in the panoramic video by way of the corresponding rotation matrix for motion compensation so as to attain a panoramic video after image stabilization, and
outputting the panoramic video after image stabilization.

2. The panoramic video processing method according to claim 1 , wherein, the conducting inter frame motion estimation with respect to each two neighboring 2D panoramic images in the panoramic video so as to obtain an inter frame motion estimation result between the same two neighboring 2D panoramic images includes
carrying out a matching process with respect to feature points in an N+1-th 2D panoramic image and feature points in an N-th 2D panoramic image so as to gain matching pairs, N being a positive integer; and
determining correct matching pairs from all the matching pairs, and computing, based on the correct matching pairs, a rotation matrix from the N+1-th 2D panoramic image to the N-th panoramic image.

3. The panoramic video processing method according to claim 2 , wherein, the determining correct matching pairs from all the matching pairs includes
selecting M matching pairs from all the matching pairs, M being a positive integer greater than one;
calculating, based on the M matching pairs, a first rotation matrix from the N+1-th 2D panoramic image to the N-th 2D panoramic image;
computing, by way of the first rotation matrix, errors of other matching pairs except the M matching pairs in all the matching pairs, and determining, based on the respective errors computed, a number of matching pairs supporting the first rotation matrix; and
determining, if the number of matching pairs supporting the first rotation matrix is greater than a predetermined threshold, that the M matching pairs are correct matching pairs, otherwise, that the M matching pairs are not correct matching pairs.

4. The panoramic video processing method according to claim 1 , wherein, the determining, based on the historical motion trajectory after smoothing, that a change of a current 2D panoramic image relative to a previous 2D panoramic image is unconscious noise motion or conscious camera motion includes
calculating, based on the historical motion trajectory, a motion noise value of the current 2D panoramic image relative to the previous 2D panoramic image; and
determining, if the motion noise value is less than or equal to a predetermined threshold, that the change is unconscious noise motion including unconscious noise motion within a predetermined noise tolerance range and unconscious noise motion exceeding the predetermined noise tolerance range, and if the motion noise value is greater than the predetermined threshold, that the change is conscious camera motion, the predetermined noise tolerance range being less than or equal to another threshold which is less than the predetermined threshold.

5. A panoramic video processing device comprising:
a processor; and
a memory storing program instructions that cause the processor to:
receive a panoramic video to be processed, the panoramic video including a plurality of two dimensional (2D) panoramic images;
conduct inter frame motion estimation in regard to each two neighboring 2D panoramic images in the panoramic video so as to obtain a corresponding inter frame motion estimation result between the same two neighboring 2D panoramic images;
calculate, based on the respective inter frame motion estimation results obtained between the plurality of 2D panoramic images, a historical motion trajectory of the panoramic video, and carrying out smoothing with respect to the historical motion trajectory;
calculate, based on the historical motion trajectory after smoothing, a rotation matrix for motion compensation pertaining to each 2D panoramic image in the panoramic video, the calculating including processes of
setting a first 2D panoramic image in the plurality of 2D panoramic images as a first reference 2D panoramic image; and
determining, based on the historical motion trajectory after smoothing, whether a change in a current 2D panoramic image relative to a previous 2D panoramic image is due to unconscious noise motion or conscious camera motion, wherein
in a case where the change is determined as unconscious noise motion, the rotation matrix for motion compensation of the current 2D panoramic image, relative to the previous 2D panoramic image, is calculated, and
in a case where the change is determined as conscious camera motion, the first reference 2D panoramic image is replaced with a second reference 2D panoramic image from a later time than the first 2D panoramic image, and the rotation matrix for motion compensation is then recalculated based on the second reference 2D panoramic image and the current 2D panoramic image; and

reconstruct on each 2D panoramic image in the panoramic video by way of the corresponding rotation matrix for motion compensation so as to attain a panoramic video after image stabilization, and to output the panoramic video after image stabilization.

6. A non-transitory computer-readable medium having computer-executable instructions for execution by a processing system, wherein, the computer-executable instructions, when executed, cause the processing system to carry out the panoramic video processing method according to claim 1 .

Assignees

• Liang Lingyan
• Ricoh Co Ltd

Inventors

• Liang Lingyan

Classifications

• G06T2207/30241

  Trajectory · CPC title

• G06T2207/20201

  Motion blur correction · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• H04N23/6811

  based on the image signal · CPC title