Video stabilization method and apparatus and non-transitory computer-readable medium

Patent metadata

Field	Value
Publication number	US-11748894-B2
Application number	US-201917051602-A
Country	US
Kind code	B2
Filing date	May 9, 2019
Priority date	May 15, 2018
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

Disclosed is a video stabilization method including steps of selecting a first area in a first image; determining at least one first feature point based on the first area, and performing feature point matching on the first image and a second image so as to obtain at least one feature point matching pair between the first image and the second image, each feature point matching pair including a first feature point of the first image and a second feature point of the second image; conducting motion parameter estimation based on the at least one feature point matching pair so as to obtain a motion parameter; and carrying out motion compensation with respect to the first image based on the motion parameter so as to acquire a first stable image after compensation.

First claim

The invention claimed is:

1. A video stabilization method comprising:
selecting a first area in a first image;
determining at least one first feature point based on the first area, and performing feature point matching on the first image and a second image so as to obtain at least one feature point matching pair between the first image and the second image, each feature point matching pair including a first feature point of the first image and a second feature point of the second image;
conducting motion parameter estimation based on the at least one feature point matching pair so as to obtain a motion parameter; and
carrying out motion compensation with respect to the first image based on the motion parameter so as to acquire a first stable image after compensation,
wherein, the selecting a first area in the first image includes
conducting object detection in regard to the first image so as to obtain at least one detected object, wherein each of the at least one detected object is located in a different first area, and
choosing at least one first area based on a type of the at least one detected object in said at least one first area being associated to a stationary object, each first area being assigned a weight value, wherein a stationary object in a first area receives a relatively high weight value while a moving object in a first area receives a relatively low weight value, and

wherein the determining at least one first feature point based on the first area includes
performing feature point detection on the chosen at least one first area so as to acquire at least one feature point, and
using the weight values as ratios for selecting one or more feature points in the respective chosen at least one first area.

2. The video stabilization method according to claim 1 , wherein,
the selecting a first area in a first image includes:
choosing at least one first area containing pixels having a first state in the first image; and

the determining at least one first feature point based on the first area includes:
performing feature point detection on the at least one first area so as to obtain at least one feature point serving as the at least one first feature point.

3. The video stabilization method according to claim 2 , wherein,
the first state is a non-motion state including:
a still state; or
a state where an instantaneous movement speed of a pixel is less than a predetermined threshold.

4. The video stabilization method according to claim 2 , wherein,
the choosing at least one first area containing pixels having a first state in the first image includes:
calculating an optical flow value of each pixel in the first image; and
letting an area in which optical flow values of pixels are less than a predetermined value be the first area.

5. The video stabilization method according to claim 1 , wherein,
the obtaining at least one feature point matching pair between the first image and the second image includes:
utilizing a motion consistency based approach to remove a mistaken feature point matching pair.

6. The video stabilization method according to claim 1 , wherein,
the carrying out motion compensation with respect to the first image based on the motion parameter includes:
performing a smoothing process on the motion parameter by mean filtering so as to acquire a smoothed motion parameter; and
employing the smoothed motion parameter to conduct motion compensation in regard to the first image.

7. The video stabilization method according to claim 1 , wherein,
the second image is a neighboring frame of the first image; or
the second image is a reference frame in a video.

8. The video stabilization method according to claim 7 , wherein,
if the second image is a neighboring frame of the first image, then the conducting motion parameter estimation based on the at least one feature point matching pair includes:
utilizing the at least one feature point matching pair to estimate a homography matrix of the first image relative to the second image.

9. The video stabilization method according to claim 8 , wherein,
if the second image is a neighboring frame of the first image, then the conducting motion parameter estimation based on the at least one feature point matching pair further includes:
employing the homography matrix and relative motion between the second image and the reference frame to estimate a motion parameter between the first image and the reference frame.

10. A video stabilization apparatus comprising:
a storage storing computer-executable instructions; and
a processor connected to the storage,
wherein the computer-executable instructions, when executed, cause the processor to implement the video stabilization method according to claim 1 .

11. A non-transitory computer-readable medium storing computer-executable instructions for execution by a processor, wherein,
the computer-executable instructions, when executed, cause the processor to carry out the video stabilization method according to claim 1 .

12. A video stabilization apparatus comprising:
a storage configured to store instructions thereon; and
a processor configured to execute the instruction to cause the video stabilization apparatus to
select a first area in a first image,
determine at least one first feature point based on the first area, and perform feature point matching on the first image and a second image so as to obtain at least one feature point matching pair between the first image and the second image, each feature point matching pair including a first feature point of the first image and a second feature point of the second image,
conduct motion parameter estimation based on the at least one feature point matching pair so as to obtain a motion parameter, and
carry out motion compensation with respect to the first image based on the motion parameter so as to acquire a first stable image after compensation,
wherein, the selecting the first area in the first image includes
conducting object detection in regard to the first image so as to obtain at least one detected object, wherein each of the at least one detected object is located in a different first area, and
choosing at least one first area based on a type of the at least one detected object in the chosen at least one first area being associated to a stationary object, each first area being assigned a weight value, wherein a stationary object in a first area receives a relatively high weight value while a moving object in a first area receives a relatively low weight value, and

wherein the determining the at least one first feature point based on the first area includes
performing feature point detection on the chosen at least one first area so as to acquire at least one feature point, and
using the weight values as ratios for selecting one or more feature points in the respective chosen at least one first area.

13. The video stabilization method according to claim 12 , wherein,
the selecting a first area in a first image includes choosing at least one first area containing pixels having a first state in the first image, and
the determining at least one first feature point based on the first area includes performing feature point detection on the at least one first area so as to obtain at least one feature point serving as the at least one first feature point.

14. The video stabilization method according to claim 13 , wherein,
the first state is a non-motion state includes:
a still state; or
a state where an instantaneous movement speed of a pixel is less than a predetermined threshold.

15. Th

Assignees

• Wang Yiwei
• Liu Liyan
• Wang Wei
• Ricoh Co Ltd

Inventors

• Wang Yiwei
• Liu Liyan
• Wang Wei

Classifications

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/248

  involving reference images or patches · CPC title

• G06V10/74

  Image or video pattern matching; Proximity measures in feature spaces · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• H04N5/144 Primary

  Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20) · CPC title