Sensor data rescaler for image signal processing

Patent metadata

Field	Value
Publication number	US-9756266-B2
Application number	US-201514977384-A
Country	US
Kind code	B2
Filing date	Dec 21, 2015
Priority date	Jul 31, 2014
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

An input rescale module that performs cross-color correlated downscaling of sensor data in the horizontal and vertical dimensions. The module may perform a first-pass demosaic of sensor data, apply horizontal and vertical scalers to resample and downsize the data in the horizontal and vertical dimensions, and then remosaic the data to provide horizontally and vertically downscaled sensor data as output for additional image processing. The module may, for example, act as a front end scaler for an image signal processor (ISP). The demosaic performed by the module may be a relatively simple demosaic, for example a demosaic function that works on 3×3 blocks of pixels. The front end of module may receive and process sensor data at two pixels per clock (ppc); the horizontal filter component reduces the sensor data down to one ppc for downstream components of the input rescale module and for the ISP pipeline.

First claim

What is claimed is:

1. An apparatus, comprising:
an image signal processor comprising an image processing pipeline for processing images captured by an image sensor; and
an input rescaler configured to:
receive sensor data for an image at two pixels per clock according to an image sensor format that represents different color channels as separate pixels;
demosaic the received sensor data at two pixels per clock to generate color space data according to a color space that includes two or more color channels in each pixel;
resample the color space data to downscale the color space data and to reduce the color space data to one pixel per clock;
remosaic the downscaled color space data at one pixel per clock to generate downscaled sensor data according to the image sensor format; and
output the downscaled sensor data to the image processing pipeline of the image signal processor at one pixel per clock.

2. The apparatus as recited in claim 1 , wherein the sensor data comprises pixel pairs according to the sensor format, each pixel pair including one pixel of a first color channel and one pixel of one of two other color channels, wherein, to demosaic the received sensor data at two pixels per clock, the input rescale module processes pixels of the first color channel on a first processing path and pixels of the two other color channels on a second processing path.

3. The apparatus as recited in claim 1 , wherein, to generate color space data for a given pixel, the demosaic function is configured to interpolate from other pixels in a respective 3×3 pixel block to estimate missing color channels for the pixel in the block and apply a blending function to the estimates to generate the color space data for the given pixel.

4. The apparatus as recited in claim 1 , wherein the input rescale module is further configured to preprocess the received sensor data at two pixels per clock to detect and correct problems in the input sensor data prior to the demosaic.

5. The apparatus as recited in claim 1 , wherein, to resample the color space data to downscale the color space data and to reduce the color space data to one pixel per clock, the input rescale module is configured to apply a horizontal filter to the color space data to horizontally downscale the color space data, wherein the horizontal filter receives pixels at two pixels per clock and outputs pixels at one pixel per clock.

6. The apparatus as recited in claim 5 , wherein, to resample the color space data, the input rescale module is further configured to apply a vertical filter to the color space data output from the horizontal filter to vertically downscale the color space data.

7. A method, comprising:
converting, by an input rescale module, sensor data received at two pixels per clock according to an image sensor format that represents different color channels as separate pixels to color space data according to a color space that includes two or more color channels in each pixel, wherein said converting is performed at two pixels per clock;
resampling, by the input rescale module, the color space data to downscale the color space data and reduce the color space data to one pixel per clock;
converting, by the input rescale module, the downscaled color space data to the image sensor format to produce downscaled sensor data according to the image sensor format; and
outputting the downscaled sensor data to an image processing pipeline of an image signal processor at one pixel per clock.

8. The method as recited in claim 7 , wherein the sensor data comprises pixel pairs according to the image sensor format, each pixel pair including one pixel of a first color channel and one pixel of one of two other color channels, wherein said converting comprises converting pixels of the first color channel on a first processing path and converting pixels of the two other color channels on a second processing path.

9. The method as recited in claim 7 , wherein said converting the sensor data comprises applying a demosaic function to 3×3 pixel blocks of the sensor data to estimate missing color channels for the pixels, wherein the demosaic function interpolates from other pixels in a given 3×3 pixel block to estimate the missing color channels of a center pixel in the block and blends the estimates to generate the color space data for the center pixel.

10. The method as recited in claim 7 , further comprising preprocessing, by the input rescale module at two pixels per clock, the sensor data prior to said converting the sensor data from the image sensor format to color space data.

11. The method as recited in claim 7 , wherein the preprocessing comprises at least one of:
detecting and correcting pixel defects in the sensor data;
detecting and correcting green non-uniformity in the sensor data; or
performing highlight recovery to prevent color bleeding in the horizontally and vertically downscaled sensor data.

12. The method as recited in claim 7 , wherein said resampling comprises applying a horizontal filter to the color space data to horizontally downscale the color space data, wherein the horizontal filter receives pixels at two pixels per clock and outputs pixels at one pixel per clock.

13. The method as recited in claim 12 , wherein said resampling further comprises applying a vertical filter to the horizontally downscaled color space data output by the horizontal filter to vertically downscale the color space data.

14. The method as recited in claim 7 , wherein the image sensor format is a format in which each block of four sensor pixels includes two green pixels, one red pixel, and one blue pixel, wherein said converting the downscaled color space data to the image sensor format to produce downscaled sensor data according to the image sensor format comprises outputting one of the color channels for each pixel in the downscaled color space data, wherein the other two color channels are discarded for the pixel.

15. A device, comprising:
an image sensor configured to capture images according to a filter format that represents different color channels as separate pixels;
an image signal processor configured to process images in the filter format captured by the image sensor to generate output images in a different format, the image signal processor comprising an image processing pipeline that implements a plurality of stages each configured to perform one or more operations on image data passing through the pipeline; and
an input rescale module located between the image sensor and the image processing pipeline and configured to:
receive filter format image data at two pixels per clock;
apply a function at two pixels per clock to the filter format image data to convert the image data to another color space;
resample the color space data to downscale the color space data and to reduce the color space data to one pixel per clock apply another function at one pixel per clock to convert the downscaled color space data to the filter format; and
output the downscaled filter format data to a stage of the image processing pipeline at one pixel per clock.

16. The device as recited in claim 15 , wherein the filter format image data comprises pixel pairs, each pixel pair including one green pixel and either one red pixel or one blue pixel, wherein, to apply the function at two pixels per clock to the filter format image data to convert the image data to the other color space, the input rescale module is configured to process green pixels on a first processing path and red and blue pixels on a second processing path.

17. The device as recited

Assignees

• Apple Inc

Inventors

• Mills Christopher L
• Lin Sheng
• Pope David R
• Silverstein D Amnon
• Lim Suk Hwan

Classifications

• H04N25/447 Primary

  by preserving the colour pattern with or without loss of information · CPC title

• H04N23/12

  with one sensor only · CPC title

• H04N5/2628 Primary

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• G06T3/4015

  Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns · CPC title

• H04N5/3458 Primary

  Electricity · mapped topic