Tagging clipped pixels for pyramid processing in image signal processor

Patent metadata

Field	Value
Publication number	US-11024006-B2
Application number	US-201916391224-A
Country	US
Kind code	B2
Filing date	Apr 22, 2019
Priority date	Apr 22, 2019
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

A portable electronic device may include an image signal processor that includes a clipping circuit, a pyramid generator circuit, and an image fusion processor. The clipping circuit clips pixel values that are under-exposed or over-exposed. The pyramid generator circuit applies a filter to the pixels of the image to generate a filtered image. Some of the filtered pixels may be generated from one or more clipped pixel values. The pyramid generator circuit identifies those filtered pixels that are generated from one or more clipped pixel values and marks the identified filtered pixels with a tag. The pyramid generator circuit decimates the filtered image to generate a downscaled image, which may include one or more filtered pixels that are marked with the tags. The image fusion processor fuses the downscaled image with another image. The pixels that are marked with the tags may be disregarded in the fusion process.

First claim

What is claimed is:

1. An image signal processor, comprising:
a clipping circuit configured to:
identify one or more pixel values in an image that are beyond a predetermined range; and
replace the one or more identified pixel values with one or more clipped pixel values; and

a pyramid generator circuit coupled to the clipped circuit and configured to:
apply one or more image filters to the image to generate a filtered image that comprises filtered pixels;
identify one or more of the filtered pixels that are generated from the one or more clipped pixel values;
mark the identified filtered pixels with a tag indicating that the identified filtered pixels are generated from the one or more clipped pixel values; and
decimate the filtered image to generate a downscaled image having a reduced number of pixels compared to the image, the downscaled image comprising one or more pixels marked with the tag, wherein the downscaled image is configured to be fused with a second image to generate a fused image, wherein the filtered pixels in the downscaled image marked with the tag are disregarded in the fusing.

2. The image signal processor of claim 1 , further comprising:
a fusion circuit coupled to the pyramid generator circuit and configured to:
receive the downscaled image from the pyramid generator circuit and a second image, the downscaled image from the pyramid generator circuit and the second image capturing a same scene with different exposure times; and
fuse the downscaled image with the second image to generate the fused image.

3. The image signal processor of claim 2 , wherein the fusion circuit is further configured to:
receive a first unsealed image generated from the image;
receive a second unsealed image, the second unsealed image being an unsealed version of the second image that is downscaled; and
fuse the first and second unsealed images to generate a second fuse image.

4. The image signal processor of claim 1 , further comprising:
a warping circuit coupled to the pyramid generator circuit and configured to:
interpolate pixel values in the image to generate an interpolated image;
identify one or more interpolated pixel values that are generated from the one or more clipped pixel values;
mark the identified interpolated pixel values with the tag; and
decimate the interpolated image.

5. The image signal processor of claim 1 , wherein the one or more image filters comprising a horizontal finite impulse response (FIR) filter and a vertical FIR filter and one of the filtered pixels is a weighted average of a subset of pixel values in the image.

6. The image signal processor of claim 1 , wherein identify one or more of the filtered pixels that are generated from the one or more clipped pixel values comprises:
generate one of the filtered pixels from a subset of pixel values of the image;
determine that one of the pixel values in the subset has a boundary value of the predetermined range; and
identify that the one of the filtered pixels is generated from the one or more clipped pixel values responsive to one of the pixel values in the subset has the boundary value.

7. The image signal processor of claim 1 , wherein the one or more image filters comprises an erosion mask, and wherein a number of filtered pixels marked with the tag in the filtered image is larger than a number of pixels in the image associated with clipped pixel values.

8. The image signal processor of claim 1 , wherein a support size of at least one of the image filters is selectable by the pyramid generator circuit.

9. The image signal processor of claim 1 , wherein the one or more filters comprises a luminance filter and a chrominance filter that is different from the luminance filter.

10. The image signal processor of claim 1 , wherein the pyramid generator circuit is further configured to generate an unscaled image in addition to the downscaled image, the unsealed image having a same number of pixels of the image and having luminance values only.

11. A method of operating an image signal processor comprising a pyramid generator circuit, the method comprising:
identifying one or more pixel values in an image that are beyond a predetermined range;
replacing the one or more identified pixel values with one or more clipped pixel values;
applying one or more image filters to the image to generate a filtered image that comprises filtered pixels;
identifying one or more of the filtered pixels that are generated from the one or more clipped pixel values;
marking the identified filtered pixels with a tag indicating that the identified filtered pixels are generated from the one or more clipped pixel values; and
decimating the filtered image to generate a downscaled image having a reduced number of pixels compared to the image, the downscaled image comprising one or more pixels marked with the tag, wherein the downscaled image is configured to be fused with a second image to generate a fused image, wherein the filtered pixels in the downscaled image marked with the tag are disregarded in the fusing.

12. The method of claim 11 , further comprising:
receiving a second image different from the downscaled image, the downscaled image and the second image capturing a same scene with different exposure times; and
fusing the downscaled image with the second image to generate the fused image.

13. The method of claim 12 , wherein fusing the downscaled image with the second image to generate the fused image comprises:
for a first fused pixel of the fused image that corresponds to a first filtered pixel in the downscaled image that is not marked with the tag, determining a first weighted average of the first filtered pixel in the downscaled image and a first corresponding pixel in the second image; and
for a second fused pixel of the fused image that corresponds to a second filtered pixel in the downscaled image that is marked with the tag, using a second corresponding pixel in the second image as the second fused pixel and disregarding the second filtered pixel.

14. The method of claim 11 , further comprising:
interpolating pixel values in the image to generate an interpolated image;
identifying one or more interpolated pixel values that are generated from the one or more clipped pixel values;
marking the identified interpolated pixel values with the tag; and
decimating the interpolated image.

15. The method of claim 11 , wherein identifying one or more of the filtered pixels that are generated from the one or more clipped pixel values comprises:
generating one of the filtered pixels from a subset of pixel values of the image;
determining that one of the pixel values in the subset has a boundary value of the predetermined range; and
identifying that the one of the filtered pixels is generated from the one or more clipped pixel values responsive to one of the pixel values in the subset has the boundary value.

16. The method of claim 11 , wherein the one or more image filters comprises an erosion mask, and wherein a number of filtered pixels marked with the tag in the filtered image is larger than a number of pixels in the image associated with clipped pixel values.

17. An electronic device, comprising:
an image sensor system configured to capture a plurality of images of a same scene, the plurality of images comprising a first image and a second image, the first image corresponding to a first exposure time and the second image corresponding to a second exposure time shorter than the first exposure time; and
an image signal processor configured to:
identify one or more

Assignees

• Apple Inc

Inventors

• Smirnov Maxim
• Lamburn Elena
• Kerem Oren
• Wu Chihsin

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2200/28

  involving image processing hardware · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T5/30

  Erosion or dilatation, e.g. thinning · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title