Correcting pixel defects based on defect history in an image processing pipeline

Patent metadata

Field	Value
Publication number	US-2017070692-A1
Application number	US-201514845659-A
Country	US
Kind code	A1
Filing date	Sep 4, 2015
Priority date	Sep 4, 2015
Publication date	Mar 9, 2017
Grant date	—

Abstract

An image signal processor may include a pixel defect correction component that tracks defect history for frames captured by an image sensor and applies the history when identifying and correcting defective pixels in a frame. The component maintains a defect pixel location table that includes a defect confidence value for pixels of the image sensor. The component identifies defective pixels in a frame, for example by comparing each pixel's value to the values of its neighbor pixels. If a pixel is detected as defective, its defect confidence value may be incremented. Otherwise, the value may be decremented. If a pixel's defect confidence value is over a defect confidence threshold, the pixel is considered defective and thus may be corrected. If a pixel's defect confidence value is under the threshold, the pixel is considered not defective and thus may not be corrected even if the pixel was detected as defective.

First claim

What is claimed is:

1 . An apparatus, comprising:
an image sensor configured to capture frames, wherein each frame is captured as a plurality of pixels; an image signal processor comprising a pixel defect correction component, the pixel defect correction component configured to receive a stream of pixel data for a frame captured by the image sensor; wherein the pixel defect correction component comprises a dynamic defect processing component configured to:
apply a dynamic defect detection technique to at least some of the pixels in the stream to detect defective pixels in the stream;
for pixels detected as defective by the dynamic defect detection technique, increment a defect confidence value for the pixel in a defect pixel location table for the image sensor;
for pixels detected as not defective by the dynamic defect detection technique, decrement the defect confidence value for the pixel in the defect pixel location table;
compare the updated defect confidence values for the pixels to a defect correction confidence threshold; and
for at least one pixel for which the respective updated defect confidence value is at or above the defect correction confidence threshold, apply a defective pixel correction technique to the pixel to correct the value of the pixel; and
wherein the pixel defect correction component is further configured to provide the stream of pixel data with the corrected values of the at least one pixel to other components of the image signal processor for additional processing.

2 . The apparatus as recited in claim 1 , wherein the pixel defect correction component further comprises a static defect processing component configured to, prior to the dynamic defect processing component applying the dynamic defect detection technique:
check the defect pixel location table to determine pixels in the stream that are marked as defective in the table; for the pixels in the stream that are marked as defective in the table and for which the respective defect confidence value is at or above a defect replacement confidence threshold, store an original value of the pixel and replace the value of the pixel with a value of a neighbor pixel of the same color component; and provide the stream of pixel data including the pixels with the replaced values to the dynamic defect processing component.

3 . The apparatus as recited in claim 1 , wherein the stream of pixel data includes patterned defect pixels for the frame, wherein the patterned defect pixels are pixels on the image sensor that intentionally captured differently than normal pixels by the image sensor, and wherein the pixel defect correction component further comprises a patterned defect pixel processing component configured to:
check the defect pixel location table to determine patterned defect pixels in the stream that are marked as defective in the table; provide the patterned defect pixels in the stream that are marked as defective in the table to the dynamic defect processing component; and for at least one patterned defect pixel in the stream that is not marked as defective in the table, correct the value of the patterned defect pixel according to a patterned defect pixel correction technique.

4 . The apparatus as recited in claim 1 , wherein the dynamic defect detection technique applies a directional gradient using two or more neighbor pixels of a current pixel being processed by the dynamic defect processing component to detect defective pixels, and wherein the defective pixel correction technique uses a weighted combination of values of two or more neighbor pixels to correct the value of the defective pixels.

5 . The apparatus as recited in claim 1 , wherein the pixel defect correction component receives the stream of pixel data from a sensor interface of the image signal processor, wherein the sensor interface is configured to:
receive a stream of raw pixel data from the image sensor; determine and apply gain values to patterned defect pixels in the stream of raw pixel data to adjust the values of the patterned defect pixels, wherein the patterned defect pixels are pixels on the image sensor that are intentionally captured differently than normal pixels by the image sensor; and provide the stream of raw pixel data with the adjusted values of the patterned defect pixels to one or more other components of the image signal processor for additional processing.

6 . The apparatus as recited in claim 1 , wherein the apparatus comprises a mobile computing device.

7 . A method, comprising:
receiving, at a pixel defect correction component of an image signal processor, a stream of pixel data collected from an image sensor; applying, by a dynamic defect processing component of the pixel defect correction component, a dynamic defect detection technique to at least some of the pixels in the stream to detect defective pixels in the stream; incrementing a defect confidence value in a defect pixel location table for pixels detected as defective by the dynamic defect detection technique; decrementing the defect confidence value in the defect pixel location table for pixels detected as not defective by the dynamic defect detection technique; comparing the updated defect confidence values for the pixels to a defect correction confidence threshold; applying, by the dynamic defect processing component, a defective pixel correction technique to at least some pixels for which the respective updated defect confidence value is at or above the defect correction confidence threshold to correct the values of the respective pixels; and providing the stream of pixel data with the corrected values to one or more other components of the image signal processor for additional processing.

8 . The method as recited in claim 7 , further comprising, prior to applying the dynamic defect detection technique:
checking, by a static defect processing component of the pixel defect correction component, the defect pixel location table to determine pixels in the stream that are marked as defective in the table; for the pixels in the stream that are marked as defective in the table and for which the respective defect confidence value is at or above a defect replacement confidence threshold, storing an original value of the pixel and replacing the value of the pixel with a value of a neighbor pixel of the same color component; and providing the stream of pixel data including the pixels with the replaced values to the dynamic defect processing component.

9 . The method as recited in claim 8 , further comprising replacing, by the dynamic defect processing component, the value of pixels for which the respective updated defect confidence value is below the defect correction confidence threshold with the stored original value of the pixel.

10 . The method as recited in claim 7 , wherein the stream of pixel data includes patterned defect pixels, wherein the patterned defect pixels are pixels on the image sensor that are intentionally captured differently than normal pixels by the image sensor, the method further comprising:
checking, by a patterned defect pixel processing component of the pixel defect correction component, the defect pixel location table to determine patterned defect pixels in the stream that are marked as defective in the table; providing the patterned defect pixels in the stream that are marked as defective in the table to the dynamic defect processing component; and correcting, by the patterned defect pixel processing component, the value of patterned defect pixels in the stream that are not marked as defective in the table according to a patterned defect pixel correction technique.

11 . The method as recited

Assignees

• Apple Inc

Inventors

• Lin Sheng
• Silverstein D Amnon
• Pope David R
• Lim Suk Hwan

Classifications

• H04N25/683 Primary

  by defect estimation performed on the scene signal, e.g. real time or on the fly detection · CPC title

• H04N25/67

  applied to fixed-pattern noise, e.g. non-uniformity of response · CPC title

• H04N5/367 Primary

  Electricity · mapped topic

• H04N5/2176

  Electricity · mapped topic