Image processing apparatus, image processing method, storage medium, system, and electronic apparatus

Patent metadata

Field	Value
Publication number	US-10764495-B2
Application number	US-201816217536-A
Country	US
Kind code	B2
Filing date	Dec 12, 2018
Priority date	Dec 14, 2017
Publication date	Sep 1, 2020
Grant date	Sep 1, 2020

Abstract

An image processing apparatus includes processing circuitry. The processing circuitry is configured to detect a positional shift amount of each of a plurality of images; select a composite target image from the plurality of images based on the detected positional shift amount; and obtain a composite image based on the positional shift amount and the selected composite target image.

First claim

What is claimed is:

1. An image processing apparatus, comprising:
processing circuitry configured to
set one of a plurality of images as a reference image and set remaining images of the plurality of images as comparative images;
detect a pixel shift amount between the reference image and each of the comparative images;
select a composite target image from among the plurality of images based on each of the detected pixel shift amounts;
obtain a composite image based on the selected composite target image; and
move the composite target image relative to the reference image based on the pixel shift amounts to have the composite image overlay the reference image.

2. The image processing apparatus according to claim 1 , wherein the processing circuitry is further configured to
compare the reference image to each of the comparative images.

3. The image processing apparatus according to claim 2 ,
wherein the composite target image includes at least one composite target image, and
wherein the processing circuitry is further configured to select, from among the comparative images, the at least one composite target image whose pixel shift amounts are represented by pixel coordinates (dx, dy) as (an even number+allowable error (Eallowable), an odd number+Eallowable) and (the odd number+Eallowable, the even number+Eallowable) when pixel coordinates (dx, dy) of the reference image are (0, 0), and whose pixel shift amount is −0.25<Eallowable<0.25.

4. The image processing apparatus according to claim 3 ,
wherein the processing circuitry is further configured to increase a boundary value of Eallowable and retry to select the at least one composite target image when failing to select the at least one composite target image from the comparative images.

5. The image processing apparatus according to claim 3 ,
wherein the processing circuitry is further configured to reset the reference image and retry to select the at least one composite target image when failing to select the at least one composite target image from the comparative images.

6. The image processing apparatus according to claim 2 ,
wherein the composite target image includes at least one composite target image, and
wherein the processing circuitry is further configured to select, from among the comparative images, the at least one composite target image whose pixel shift amounts are represented by pixel coordinates (dx, dy) as (an even number+allowable error (Eallowable), an odd number+Eallowable), (the odd number+Eallowable, the odd number+Eallowable), and (the odd number+Eallowable, the even number+Eallowable) when pixel coordinates (dx, dy) of the reference image are (0, 0), and whose pixel shift amount is −0.25<Eallowable<0.25.

7. The image processing apparatus according to claim 2 ,
wherein the processing circuitry is further configured to detect the pixel shift amount of each of the plurality of images on one of a pixel-to-pixel basis and a subpixel-to-subpixel basis, based on a pixel output of an image sensor of an imaging apparatus on which the image processing apparatus is mounted.

8. The image processing apparatus according to claim 7 ,
wherein the processing circuitry is further configured to exclude a pixel output to be used in an operation other than an image-capturing operation, from the pixel output of the image sensor.

9. The image processing apparatus according to claim 2 , wherein the processing circuitry is further configured to detect the pixel shift amount of each of the plurality of images for each RGB plane based on a pixel output of an image sensor of an imaging apparatus on which the image processing apparatus is mounted.

10. The image processing apparatus according to claim 9 ,
wherein the processing circuitry is further configured to detect the pixel shift amount of each of the plurality of images with a change in RGB plane to be used.

11. The image processing apparatus according to claim 2 ,
wherein the processing circuitry is further configured to move the composite target image relative to the reference image in units of movement different from units of detection at which the pixel shift amount is detected.

12. The image processing apparatus according to claim 1 , wherein the processing circuitry is further configured to:
divide the plurality of images into a predetermined number of image areas;
detect a positional shift amount for each of the predetermined number of image areas;
select a composite target image area from among the plurality of images, based on the detected positional shift amount; and
obtain the composite image based on the positional shift amount and the selected composite target image area.

13. The image processing apparatus according to claim 12 , wherein each of the divided image areas has a different size.

14. The image processing apparatus according to claim 13 , wherein the processing circuitry is further configured to divide the plurality of images into an image area having a largest size at a central portion of each of the plurality of images and image areas each having a smallest size at peripheral portion of each of the plurality of images.

15. An electronic apparatus comprising:
the image processing apparatus according to claim 14 ; and
an image-capturing device configured to capture the plurality of images.

16. The electronic apparatus according to claim 15 , further comprising a switch configured to switch between a first image-capturing mode and a second image-capturing mode different from the first image-capturing mode,
the first image-capturing mode causing the processing circuitry to obtain the plurality of images captured by the image-capturing device, detect the pixel shift amounts, select the composite target image, and obtain the composite image.

17. The electronic apparatus according to claim 15 , further comprising a display to display a setting of the image-capturing mode.

18. The electronic apparatus according to claim 15 , further comprising a drive device configured to move a drive member of the image-capturing device in a direction different from a direction of an optical axis of the image-capturing device.

19. The electronic apparatus according to claim 18 ,
wherein the drive device is configured to reduce a drive component in a parallel direction within a plane orthogonal to the optical axis, and increase a drive component in a rotational direction within the plane orthogonal to the optical axis.

20. A method of processing an image, the method comprising:
setting one of a plurality of images as a reference image and setting remaining images of the plurality of images as comparative images;
detecting a pixel shift amount between the reference image and each of the comparative images;
selecting a composite target image from among the plurality of images based on each of the detected pixel shift amounts;
obtaining a composite image based on the selected composite target image; and
moving the composite target image relative to the reference image based on the pixel shift amounts to have the composite image overlay the reference image.

21. A non-transitory recording medium storing a program for causing a computer to execute a method of processing an image, the method comprising:
setting one of a plurality of images as a reference image and setting remaining images of the plurality of images as comparative images;
detecting a pixel shift amount between the reference image and each of the comparative images;
selecting a composit

Assignees

• Uehara Hiroyasu
• Terauchi Masakazu
• Miyata Masashi
• Kano Hiroshi
• Nakano Akira
• Nishimura Takeyoshi
• Ricoh Co Ltd

Inventors

• Uehara Hiroyasu
• Terauchi Masakazu
• Miyata Masashi
• Kano Hiroshi
• Nakano Akira
• Nishimura Takeyoshi

Classifications

• H04N25/48

  Increasing resolution by shifting the sensor relative to the scene · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• H04N23/951 Primary

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• H04N23/687

  by shifting the lens or sensor position · CPC title