Method and apparatus with image processing

Patent metadata

Field	Value
Publication number	US-12056844-B2
Application number	US-202217748180-A
Country	US
Kind code	B2
Filing date	May 19, 2022
Priority date	Aug 31, 2021
Publication date	Aug 6, 2024
Grant date	Aug 6, 2024

Abstract

A method with image processing includes: receiving an input image including Bayer images captured by a plurality of lenses included in a lens assembly; generating channel separation images by separating each of the Bayer images by a plurality of channels; determining corresponding points such that pixels in the channel separation images are displayed at the same position on a projection plane, for each of the plurality of lenses; performing binning on the channel separation images, based on a brightness difference and a distance difference between a target corresponding point and a center of a pixel including the target corresponding point, corresponding to each of the corresponding points in channel separation images that correspond to a same channel and that are combined into one image, for each of the plurality of lenses; restoring the input image for each of the plurality of lenses based on binned images generated by performing the binning; and outputting the restored input image.

First claim

What is claimed is:

1. A method with image processing, the method comprising:
receiving an input image including Bayer images captured by a plurality of lenses included in a lens assembly;
generating channel separation images by separating each of the Bayer images by a plurality of channels;
determining corresponding points such that pixels in the channel separation images are displayed at the same position on a projection plane, for each of the plurality of lenses;
performing binning on the channel separation images, based on a brightness difference and a distance difference between a target corresponding point and a center of a pixel including the target corresponding point, corresponding to each of the corresponding points in channel separation images that correspond to a same channel and that are combined into one image, for each of the plurality of lenses;
restoring the input image for each of the plurality of lenses based on binned images generated by performing the binning; and
outputting the restored input image.

2. The method of claim 1 , wherein the determining of the corresponding points comprises determining the corresponding points of the pixels based on an optical flow in the channel separation images.

3. The method of claim 1 , wherein the determining of the corresponding points comprises determining the corresponding points of the pixels for each of the plurality of lenses based on depth information and calibration information for each of the plurality of lenses.

4. The method of claim 3 , wherein the determining of the corresponding points comprises:
calling a lookup table (LUT) that stores corresponding points of a portion of the pixels at a predetermined distance among the pixels, for each of the plurality of lenses; and
determining the corresponding points of the pixels by interpolating the corresponding points of the portion of the pixels stored in the LUT.

5. The method of claim 1 , wherein the performing of the binning comprises performing binning on the channel separation images based on a probability value according to characteristics of the corresponding points comprising the distance difference and the brightness difference in the channel separation images that correspond to the same channel and that are combined into the one image, for each of the plurality of lenses.

6. The method of claim 1 , wherein the performing of the binning comprises:
determining a weight based on the distance difference and the brightness difference, or based on a gradient difference between the center of the pixel and the target corresponding point, the distance difference and the brightness difference, corresponding to each of the corresponding points in the channel separation images combined into the one image, for each of the plurality of lenses; and
determining a weighted average of the corresponding points for each of the plurality of lenses, based on the weight.

7. The method of claim 1 , wherein the restoring of the input image comprises:
generating a second Bayer image by combining the binned images for each channel, corresponding to the plurality of lenses; and
restoring the input image to a color image by interpolating a color for each of pixels of the second Bayer image.

8. The method of claim 7 , wherein the generating of the second Bayer image comprises generating the second Bayer image by combining channels of the binned images so that each of the channels of the binned images has a Bayer pattern.

9. The method of claim 1 , further comprising:
restoring a resolution by up-sampling the binned images for each of the plurality of lenses,
wherein the restoring of the input image comprises:
generating a second Bayer image by combining images with the restored resolution; and
restoring the input image to a color image by interpolating a color for each of pixels of the second Bayer image.

10. The method of claim 1 , wherein the generating of the channel separation images comprises separating each of the Bayer images by the plurality of channels by placing each of the Bayer images in corresponding pixels for each of a plurality of channels included in a Bayer pattern.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .

12. An apparatus with image processing, the apparatus comprising:
a communication interface configured to receive an input image including Bayer images captured by a plurality of lenses included in a lens assembly;
one or more processors configured to
generate channel separation images by separating each of the Bayer images by a plurality of channels,
determine corresponding points such that pixels in the channel separation images are displayed at the same position on a projection plane, for each of the plurality of lenses,
perform binning on the channel separation images, based on a brightness difference and a distance difference between a target corresponding point and a center of a pixel including the target corresponding point, corresponding to each of the corresponding points in channel separation images that correspond to a same channel and that are combined into one image, for each of the plurality of lenses, and
restore the input image for each of the plurality of lenses based on binned images generated by performing the binning; and

a display configured to output the restored input image.

13. The apparatus of claim 12 , wherein, for the determining of the corresponding points, the one or more processors are configured to determine the corresponding points of the pixels for each of the plurality of lenses based on depth information and calibration information for each of the plurality of lenses.

14. The apparatus of claim 13 , wherein, for the determining of the corresponding points, the one or more processors are configured to:
call a lookup table (LUT) that stores corresponding points of a portion of the pixels at a predetermined distance among the pixels, for each of the plurality of lenses; and
determine the corresponding points of the pixels by interpolating the corresponding points of the portion of the pixels stored in the LUT.

15. The apparatus of claim 12 , wherein, for the performing of the binning, the one or more processors are configured to perform binning on the channel separation images based on a probability value according to characteristics of the corresponding points comprising the distance difference and the brightness difference in the channel separation images that correspond to the same channel and that are combined into the one image, for each of the plurality of lenses.

16. The apparatus of claim 12 , wherein, for the performing of the binning, the one or more processors are configured to:
determine a weight based on the distance difference and the brightness difference, or based on a gradient difference between the center of the pixel and the target corresponding point, the distance difference and the brightness difference, corresponding to each of the corresponding points in the channel separation images combined into the one image, for each of the plurality of lenses; and
determine a weighted average of the corresponding points for each of the plurality of lenses, based on the weight.

17. The apparatus of claim 12 , wherein, for the restoring of the input image, the one or more processors are configured to:
generate a second Bayer image by combining the binned images for each channel, corresponding to the plurality of lenses; and
restore the

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kang Byong Min
• Nam Dong Kyung
• Cho Yang Ho
• Heo Jingu

Classifications

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• H04N23/84

  for processing colour signals · CPC title

• H04N23/843

  Demosaicing, e.g. interpolating colour pixel values · CPC title

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title