Method and device for image rectification

Patent metadata

Field	Value
Publication number	US-11240477-B2
Application number	US-201816188324-A
Country	US
Kind code	B2
Filing date	Nov 13, 2018
Priority date	Nov 13, 2017
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

The present invention provides a method and a device for image rectification, which are applied in the field of image processing. The method includes: receiving two images, the two images are images of a target object captured from different viewpoints; performing an epipolar rectification on the two images; rectifying the two images after the epipolar rectification based on the image contents; and splicing the two images after the rectification based on the image contents. The method can rectify the images captured from different viewpoints, thereby ensuring the pixel alignment, and avoiding visual discomfort to the viewer.

First claim

What is claimed is:

1. A method for image rectification, comprising:
receiving two images captured by a first image capturing device and a second image capturing device, wherein the two images are images of a target object captured from different viewpoints;
performing an epipolar rectification on the two images;
selecting a same focal length parameter for the first image capturing device and the second image capturing device, wherein the two images captured from the different viewpoints have a same field of view after an undistortion without a scaling relationship;
rectifying the two images after the epipolar rectification based on image contents; and
splicing the two images rectified based on the image contents,
wherein rectifying the two images comprises determining positions of feature point pairs corresponding in the two images by:
determining positions of candidate feature point pairs corresponding in the two images;
removing feature point pairs having a wrong correspondence from the candidate feature point pairs using a Random Sample Consensus (RANSAC) algorithm; and
aligning the two images according to the positions of the feature point pairs and parameters of the first image capturing device and the second image capturing device,

wherein aligning the two images according to the positions of the feature point pairs and the parameters of the first image capturing device and the second image capturing device comprises:
determining an error function relative to an obtained first parameter according to the positions of the feature point pairs and the parameters of the first image capturing device and the second image capturing device;
obtaining optimized parameters of the first image capturing device and the second image capturing device according to the error function; and
adjusting the two images according to the optimized parameters of the first image capturing device and the second image capturing device,

wherein each feature point pair comprises a first point and a second point,
wherein determining the error function relative to the obtained first parameter according to the positions of the feature point pairs and the parameters of the first image capturing device and the second image capturing device comprises:
projecting the first point to an image coordinate system of an image of the second point by a three-dimensional point cloud according to the parameters of the first image capturing device and the second image capturing device to obtain a projection point; and
determining a distance between the projection point and the second point as the error function, and

wherein the error function comprises a regularization term, and
wherein the regularization term is determined by a multiplication of a coefficient and a sum of squares of a pair of predetermined rotation-related parameters from the first image capturing device and the second image capturing device.

2. The method of claim 1 , wherein feature points of the feature point pairs comprise one item listed below:
points having variations in at least one direction greater than a first threshold; and points with unchanged locations independent of whether the sizes of the images change.

3. The method of claim 1 , wherein the pair of predetermined rotation-related parameters correspond to an axis in a three-dimensional coordinate system, and wherein the regularization term is determined to ensure not excessive rotation over the axis when the error function is minimal.

4. The method of claim 3 , wherein the axis is Z-axis.

5. The method of claim 1 , wherein the optimized parameters of the first image capturing device and the second image capturing device comprise at least one item listed below:
rotation parameters, focal lengths, and coordinates of principal points.

6. The method of claim 1 , wherein
performing the epipolar rectification on the two images comprises:
undistorting the two images according to the selected focal length parameter;
determining a row alignment matrix of the two images in a no-parallax direction according to a translation matrix and a rotation matrix of the first image capturing device and the second image capturing device; and
rectifying the two images according to the row alignment matrix in the no-parallax direction.

7. The method of claim 1 , wherein before splicing the two images rectified based on image contents, the method further comprises:
cropping parts of whole images of the two images or the whole images for splicing, according to contents in the whole images, from the two images rectified based on the image contents.

8. A non-transitory computer readable storage medium storing computer programs, wherein when the computer programs are executed by a second processor, the steps of the method of claim 1 are implemented.

9. A device for image rectification, comprising a memory, a third processor, and computer programs stored in the memory and operable on the third processor, wherein when the computer programs are executed by the third processor, the steps of the method of claim 1 are implemented.

10. A device for image rectification, comprising:
a receiver, configured to receive two images, wherein the two images are images of a target object captured from different viewpoints;
a first image capturing device having a first focal length parameter and a second image capturing device having a second focal length parameter, wherein the first focal length parameter and the second focal length parameter are the same, wherein the two images captured from the different viewpoints have a same field of view after an undistortion without a scaling relationship, and wherein the first image capturing device and the second image capturing device capture the first image and the second image;
a rectification processor configured to perform an epipolar rectification on the two images, determine positions of feature point pairs corresponding in the two images, and rectify the two images after the epipolar rectification based on image contents; and
a splicer, configured to splice the two images rectified based on the image contents,
wherein the rectification processor determines positions of feature point pairs corresponding in the two images by:
determining positions of candidate feature point pairs corresponding in the two images; and
removing feature point pairs having a wrong correspondence from the candidate feature point pairs using a Random Sample Consensus (RANSAC) algorithm,

wherein the rectification processor aligns the two images according to the positions of the feature point pairs and parameters of the first image capturing device and the second image capturing device,
wherein each feature point pair comprises a first point and a second point,
wherein the rectification processor determines an error function relative to an obtained first parameter according to the positions of the feature point pairs and the parameters of the first image capturing device and the second image capturing device by:
projecting the first point to an image coordinate system of an image of the second point by a three-dimensional point cloud according to the parameters of the first image capturing device and the second image capturing device to obtain a projection point; and
determining a distance between the projection point and the second point as the error function,

wherein the rectification processor obtains optimized parameters of the first image capturing device and the second image capturing device according to the error function,
wherein the rectification processors adjust the two images according to the optimized parameters of the first image capturing device and the second image capturing de

Assignees

• Arcsoft Corp Ltd

Inventors

• Miao Wang
• Yu Li
• Fan Xin

Classifications

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• H04N13/246

  Calibration of cameras · CPC title

• H04N13/122 Primary

  Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) · CPC title