Image processing apparatus, image processing method, and program

Patent metadata

Field	Value
Publication number	US-10970877-B2
Application number	US-201615762206-A
Country	US
Kind code	B2
Filing date	Sep 16, 2016
Priority date	Sep 30, 2015
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

An image processing apparatus, an image processing method, and a program that permit camera calibration with high accuracy by using a known object in images captured by a plurality of imaging sections. An estimation section estimates a 3D position of a road sign included in each of images captured by a plurality of cameras with respect to each of the imaging sections. A recognition section recognizes a positional relationship between the plurality of cameras on the basis of the 3D position of the road sign with respect to each of the cameras estimated by the estimation section. The positional relationship between the plurality of cameras recognized by the recognition section is used to correct the images captured by the plurality of cameras.

First claim

The invention claimed is:

1. An image processing apparatus comprising:
a processor and a memory containing instructions that, when executed by the processor, are configured to:
detect stereo parallax of at least one feature point from feature point information derived from images captured by at least two cameras;
calculate a position of the at least one feature point in three dimensional space based on the detected stereo parallax;
estimate a travel of the cameras based on the calculated position of the at least one feature point and information of a past position of the at least one feature point;
decide to calibrate the cameras based on the estimated travel;
estimate a three dimensional position of each of a plurality of known objects, of which size and shape are known, included in each of images captured by the cameras with respect to each of the cameras, based at least in part on stored feature quantities of the plurality of known objects;
recognize a positional relationship between the cameras on the basis of estimation accuracy with which the three dimensional position of each of the known objects is estimated and on the basis of the three dimensional position of each of the known objects with respect to each of the cameras; and
correct the images captured by the cameras on the basis of the recognized positional relationship between the cameras.

2. The image processing apparatus of claim 1 , wherein the instructions are configured to:
estimate the three dimensional position and orientation of each of the known objects with respect to each of the cameras,
recognize a relationship between positions and orientations of the cameras on the basis of the three dimensional position and orientation of each of the known objects with respect to each of the cameras, and
correct the images on the basis of the relationship between the positions and the orientations.

3. The image processing apparatus of claim 1 , wherein the instructions are configured to:
estimate, for each camera, the three dimensional position of the known object with respect to the camera on the basis of a feature point position of the known object on the image and a parameter of the camera.

4. The image processing apparatus of claim 1 , wherein
the cameras are mounted to a vehicle, and
the known object is an object on a road or an object possessed by the vehicle.

5. An image processing method, by an image processing apparatus, comprising:
detecting stereo parallax of at least one feature point from feature point information derived from images captured by at least two cameras;
calculating a position of the at least one feature point in three dimensional space based on the detected stereo parallax;
estimating a travel of the cameras based on the calculated position of the at least one feature point and information of a past position of the at least one feature point;
deciding to calibrate the cameras based on the estimated travel;
estimating a three dimensional position of each of known objects, of which size and shape are known, included in each of images captured by a plurality of the cameras with respect to each of the cameras, based at least in part on stored feature quantities of the plurality of known objects;
recognizing a positional relationship between the cameras on the basis of estimation accuracy with which the three dimensional position of each of the known objects is estimated and on the basis of the estimated three dimensional position of each of the known objects with respect to each of the cameras; and
correcting the images captured by the cameras on the basis of the recognized positional relationship between the cameras.

6. A non-transitory, computer-readable medium containing instructions that, when executed by a processing device, perform an image processing method comprising:
detecting stereo parallax of at least one feature point from feature point information derived from images captured by at least two cameras;
calculating a position of the at least one feature point in three dimensional space based on the detected stereo parallax;
estimating a travel of the cameras based on the calculated position of the at least one feature point and information of a past position of the at least one feature point;
deciding to calibrate the cameras based on the estimated travel;
estimating a three dimensional position of each of a plurality of known objects, of which size and shape are known, included in each of images captured by the cameras with respect to each of the cameras, based at least in part on stored feature quantities of the plurality of known objects;
recognizing a positional relationship between the cameras on the basis of estimation accuracy with which the three dimensional position of each of the known objects is estimated and on the basis of the three dimensional position of each of the known objects with respect to each of the cameras; and
correcting the images captured by the cameras on the basis of the recognized positional relationship between the cameras.

Assignees

• Sony Corp

Inventors

• Eshima Masashi
• Kaino Akihiko
• Kato Takaaki
• Tsurumi Shingo

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T7/85 Primary

  Stereo camera calibration · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N17/002

  for television cameras · CPC title

• H04N13/207

  using a single two-dimensional [2D] image sensor · CPC title