Image processing apparatus and image processing method

Patent metadata

Field	Value
Publication number	US-10453219-B2
Application number	US-201615571165-A
Country	US
Kind code	B2
Filing date	Jun 3, 2016
Priority date	Jun 17, 2015
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

There is provided an image processing apparatus and an image processing method capable of robust correction to an image misalignment generated due to an over-time misalignment of a stereo camera. The estimation section estimates at least two parameters out of a pitch angle difference, a yaw angle difference, and a roll angle difference between a left camera and a right camera, and a scale ratio of a left image picked up by the left camera to a right image picked up by the right camera, on basis of a model formula using the parameters. The present disclosure is applicable to, for example, an imaging apparatus that includes a stereo camera configured with the left camera and the right camera and the like.

First claim

The invention claimed is:

1. An image processing apparatus comprising:
a distribution analysis section configured to
determine whether or not a parameter of a pitch angle difference between a first imaging section and a second imaging section is estimable with sufficient accuracy and utilized as one parameter of a plurality of estimation parameters, based on coordinates of a predetermined number or more of midpoints between image feature points being distributed in an entire region of a picture plane,
determine whether or not a parameter of a yaw angle difference between the first imaging section and the second imaging section is estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters, based on the coordinates of the predetermined number or more of the midpoints being distributed in an upper left region, a lower left region, an upper right region, a lower right region, and a central region when the picture plane is divided into 3×3 regions,
determine whether or not a parameter of a roll angle difference between the first imaging section and the second imaging section is estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters, based on the coordinates of the predetermined number or more of the midpoints being distributed in regions obtained by dividing the picture plane into a plurality of regions in a horizontal direction, and
determine whether or not a parameter of a scale ratio of a first image picked up by the first imaging section to a second image picked up by the second imaging section is estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters, based on the coordinates of the predetermined number or more of the midpoints being distributed in regions obtained by dividing the picture plane into a plurality of regions in a perpendicular direction;

an estimation section configured to estimate two or more parameters selected from a group of parameters consisting of the pitch angle difference, the yaw angle difference, the roll angle difference, and the scale ratio, wherein the parameters selected for estimation include only those parameters that have been determined by the distribution analysis section to be estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters; and
a warping section configured to perform warping on the first image and the second image based on a model formula using the parameters estimated by the estimation section, to thereby correct image misalignment,
wherein the distribution analysis section, the estimation section, and the warping section are each implemented via at least one processor.

2. The image processing apparatus according to claim 1 , wherein
the parameters selected for estimation are configured to include the pitch angle difference.

3. The image processing apparatus according to claim 1 , wherein
the estimation section estimates the parameters in such a manner as to make smallest a difference between an estimated value of a perpendicular misalignment between the first image and the second image calculated based on the model formula and a measured value of the perpendicular misalignment between the first image and the second image.

4. The image processing apparatus according to claim 3 , further comprising:
a detection section that detects a pair of a perpendicular position of each feature point within one of the first image and the second image and a perpendicular position of a point corresponding to the feature point within the other image, wherein
the measured value is configured to be calculated based on the pair, and
the detection section is implemented via at least one processor.

5. The image processing apparatus according to claim 4 ,
wherein the distribution analysis section selects the parameters for estimation by the estimation section based on a distribution of the feature points.

6. The image processing apparatus according to claim 4 , further comprising:
a generation section that generates photographing instruction information for instructing a user on a photographing method using the first imaging section and the second imaging section based on the distribution of the feature points,
wherein the generation section is implemented via at least one processor.

7. The image processing apparatus according to claim 3 , further comprising:
an update section that updates the parameters based on the difference between the estimated value of the perpendicular misalignment between the first image and the second image calculated based on the model formula using the parameters estimated by the estimation section and the measured value,
wherein the update section is implemented via at least one processor.

8. The image processing apparatus according to claim 1 , wherein
the parameters estimated by the estimation section are each configured to be determined based on a structure of the first imaging section and the second imaging section.

9. The image processing apparatus according to claim 1 , wherein
the parameters estimated by the estimation section are each configured to be determined based on a type of a process executed using the first image and the second image.

10. The image processing apparatus according to claim 1 , wherein
the warping section is configured to perform the warping on the first image and the second image based on the model formula and an initial parameter measured by a calibration.

11. An image processing method comprising:
determining, by an image processing apparatus, whether or not a parameter of a pitch angle difference between a first imaging section and a second imaging section is estimable with sufficient accuracy and utilized as one parameter of a plurality of estimation parameters, based on coordinates of a predetermined number or more of midpoints between image feature points being distributed in an entire region of a picture plane;
determining, by the image processing apparatus, whether or not a parameter of a yaw angle difference between the first imaging section and the second imaging section is estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters, based on the coordinates of the predetermined number or more of the midpoints being distributed in an upper left region, a lower left region, an upper right region, a lower right region, and a central region when the picture plane is divided into 3×3 regions;
determining, by the image processing apparatus, whether or not a parameter of a roll angle difference between the first imaging section and the second imaging section is estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters, based on the coordinates of the predetermined number or more of the midpoints being distributed in regions obtained by dividing the picture plane into a plurality of regions in a horizontal direction;
determining, by the image processing apparatus, whether or not a parameter of a scale ratio of a first image picked up by the first imaging section to a second image picked up by the second imaging section is estimable with sufficient accuracy and utilized as one parameter of the plurality of estimation parameters, based on the coordinates of the predetermined number or more of the midpoints being distributed in regions obtained by dividing the picture plane into a plurality of regions in a perpendicular direction;
estimating, by the image processing apparatus, two or more parameters selected from a group of parameters consisting of the pitch angle difference, the yaw angle differ

Assignees

• Sony Corp

Inventors

• Myokan Yoshihiro

Classifications

• H04N23/64

  Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• G01C11/12

  the pictures being supported in the same relative position as when they were taken · CPC title