Image processing device, imaging system and imaging apparatus including the same, and image processing method

Patent metadata

Field	Value
Publication number	US-10536633-B2
Application number	US-201615003160-A
Country	US
Kind code	B2
Filing date	Jan 21, 2016
Priority date	Feb 6, 2015
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

An image processing device includes a reference camera determiner that determines one among the multiple cameras, as a reference camera; an imaging condition acquisition unit that acquires imaging conditions relating to exposure and white balance which are set in the reference camera; an imaging condition setter that sends a command to set the imaging conditions relating to the exposure and the white balance, based on the imaging conditions in the reference camera, to cameras other than the reference camera; and an image composition unit that performs image composition processing on the multiple captured images that are acquired from the multiple cameras and that outputs the panoramic composite image. Thus, a boundary of an area of each composed image in the panoramic composite image that results from the composition is suppressed from being unnaturally conspicuous, and the captured image that is a reference is suitably displayed.

First claim

What is claimed is:

1. An image processing device that is connected to multiple cameras in such a manner that communication with the multiple cameras is possible, and that outputs a panoramic composite image by performing image composition processing on multiple captured images that are acquired from the multiple cameras, the image processing device comprising:
a processor; and
a memory including instructions that cause the processor to execute operations, the operations including:
determining one camera among the multiple cameras as a reference camera;
acquiring first imaging conditions relating to exposure and white balance which are set in the reference camera;
sending a first command to set second imaging conditions relating to exposure and white balance, based on the first imaging conditions in the reference camera, to cameras other than the reference camera among the multiple cameras; and
performing image composition processing on the multiple captured images that are acquired from the multiple cameras and outputting the panoramic composite image,
wherein the image composition processing includes repeatedly performing second operations until a stitching process is performed for each pair of adjacent images of the captured images, the second operations including:
selecting two adjacent captured images on which the stitching process is not yet performed;
generating, based on the two adjacent captured images, two corresponding parallax correction adjacent images in which an amount of parallax correction due to matching between the two adjacent captured images is calculated and corrected;
calculating a first difference value of luminance and a second difference value of color difference in a boundary area between the two corresponding parallax correction adjacent images;
in a case where at least one of the first difference value exceeds a luminance threshold or the second difference value exceeds a color difference threshold:
sending a second command to set third imaging conditions of one of the multiple cameras that acquires one of the two adjacent captured images, in such a manner that the first difference value does not exceed the luminance threshold and the second difference value does not exceed the color difference threshold; and
performing the stitching process on the two corresponding parallax correction adjacent images; and
in a case where the first difference value does not exceed the luminance threshold and the second difference value does not exceed the color difference threshold:
performing the stitching process on the two corresponding parallax correction adjacent images without sending the second command.

2. The image processing device of claim 1 ,
wherein the processor acquires information on luminance of the captured images that are acquired from the multiple cameras when determining the reference camera, and determines the one camera that has a luminance that is a median of multiple luminance levels as the reference camera.

3. The image processing device of claim 1 ,
wherein the processor acquires information on luminance of the captured images that are acquired from the multiple cameras when determining the reference camera, and determines the one camera that has a luminance that is the closest to an average value of multiple luminance levels as the reference camera.

4. The image processing device of claim 1 ,
wherein the processor determines the one camera based on a selection by a user from among the multiple cameras.

5. The image processing device of claim 1 ,
wherein the processor acquires a result of image recognition of a predetermined imaging target in the captured images that are acquired from the multiple cameras when determining the reference camera, and determines the one camera as the reference camera based on the result of the image recognition.

6. An imaging system comprising:
the image processing device of claim 1 ; and
the multiple cameras, with the multiple cameras being arranged such that portions of the multiple captured images of two adjacent cameras overlap.

7. An imaging apparatus comprising:
multiple imaging elements; and
an image processor that performs image composition processing on multiple captured images that are acquired by the imaging elements and outputs a panoramic composite image,
wherein the image processor is configured to perform operations including:
determining one imaging element among the multiple imaging elements as a reference imaging element,
acquiring first imaging conditions relating to exposure and white balance which are set in the reference imaging element,
sending a first command to set second imaging conditions relating to exposure and white balance, based on the first imaging conditions in the reference imaging element, to imaging elements other than the reference imaging element among the multiple imaging elements, and
performing image composition processing on the multiple captured images that are acquired from the multiple imaging elements and outputting the panoramic composite image,
wherein the image composition processing includes repeatedly performing second operations until a stitching process is performed for each pair of adjacent images of the captured images, the second operations including:
selecting two adjacent captured images on which the stitching process is not yet performed;
generating, based on the two adjacent captured images, two corresponding parallax correction adjacent images in which an amount of parallax correction due to matching between the two adjacent captured images is calculated and corrected;
calculating a first difference value of luminance and a second difference value of color difference in a boundary area between the two corresponding parallax correction adjacent images;
in a case where at least one of the first difference value exceeds a luminance threshold or the second difference value exceeds a color difference threshold:
sending a second command to set third imaging conditions of one of the multiple imaging elements that acquires one of the two adjacent captured images, in such a manner that the first difference value does not exceed the luminance threshold and the second difference value does not exceed the color difference threshold; and
performing the stitching process on the two corresponding parallax correction adjacent images; and
in a case where the first difference value does not exceed the luminance threshold and the second difference value does not exceed the color difference threshold:
performing the stitching process on the two corresponding parallax correction adjacent images without sending the second command.

8. An image processing method of performing image composition processing on multiple captured images that are acquired by multiple cameras and outputting a panoramic composite image, for use in an image processing device that is connected to the multiple cameras in such a manner that communication with the multiple cameras is possible, the image processing method comprising:
determining one camera among the multiple cameras as a reference camera;
acquiring first imaging conditions relating to exposure and white balance which are set in the reference camera;
sending a first command to set second imaging conditions relating to exposure and white balance, based on the first imaging conditions in the reference camera, to cameras other than the reference camera among the multiple cameras; and
performing image composition processing on the multiple captured images that are acquired from the multiple cameras and outputting the panoramic composite image,
wherein the image composition processing includes repeatedly performing second operations until a stitching process is performe

Assignees

• Panasonic Ip Man Co Ltd

Inventors

• Kuwada Junya
• Ogino Ryoji
• Hirota Ryo
• Nishiyama Akio

Classifications

• H04N1/3876 Primary

  Recombination of partial images to recreate the original image · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/70

  Circuitry for compensating brightness variation in the scene · CPC title

• H04N23/88

  for colour balance, e.g. white-balance circuits or colour temperature control · CPC title