Calibration of defective image sensor elements

Patent metadata

Field	Value
Publication number	US-9854188-B2
Application number	US-201514972075-A
Country	US
Kind code	B2
Filing date	Dec 16, 2015
Priority date	Dec 16, 2015
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

The present disclosure relates to methods and systems that may reduce pixel noise due to defective sensor elements in optical imaging systems. Namely, a camera may capture a burst of images with an image sensor while adjusting a focus distance setting of an optical element. For example, the image burst may be captured during an autofocus process. The plurality of images may be averaged or otherwise merged to provide a single, aggregate image frame. Such an aggregate image frame may appear blurry. In such a scenario, “hot” pixels, “dead” pixels, or otherwise defective pixels may be more easily recognized and/or corrected. As an example, a defective pixel may be removed from a target image or otherwise corrected by replacing a value of the defective pixel with an average value of neighboring pixels.

First claim

What is claimed is:

1. An optical imaging system comprising:
an image sensor comprising a plurality of sensor elements, the sensor elements configured to represent respective pixels in captured images;
an optical element coupled to the image sensor and configured to define a focus distance setting of the captured images; and
a controller comprising a memory and at least one processor, wherein the processor is configured to execute instructions stored in the memory so as to carry out operations, the operations comprising:
causing the image sensor to capture a plurality of images, the images captured with different respective focus distance settings defined by the optical element;
combining the plurality of images into an aggregate image;
based on the aggregate image, determining a condition of a particular sensor element of the plurality of sensor elements;
causing the image sensor to capture a payload image; and
adjusting, in the payload image, a value of a particular pixel that is associated with the particular sensor element.

2. The optical imaging system of claim 1 , wherein the operations further comprise metering a field of view of the image sensor and, based on the metering, selecting a shutter speed and a sensitivity of the image sensor with which to capture the plurality of images.

3. The optical imaging system of claim 1 , wherein the operations further comprise mapping the determined condition in a sensor element condition map, and wherein adjusting the value of the particular pixel is based on the sensor element condition map.

4. The optical imaging system of claim 3 , wherein the operations further comprise associating the sensor element condition map with a time and providing a historical record of a plurality of sensor element condition maps associated with a corresponding plurality of different times, and wherein adjusting the value of the particular pixel is further based on the historical record.

5. The optical imaging system of claim 1 , wherein causing the image sensor to capture a plurality of images comprises causing the image sensor to carry out an autofocus process, wherein the autofocus process comprises adjusting a position of the optical element with respect to the image sensor to achieve the different respective focus distance settings for the respective images.

6. The optical imaging system of claim 1 , wherein combining the plurality of images into the aggregate image comprises averaging associated pixel values of the plurality of images.

7. The optical imaging system of claim 1 , wherein determining the condition of a particular sensor element comprises:
comparing a value of a pixel in the aggregate image and associated with the particular sensor element to that of at least one neighboring pixel in the aggregate image; and
determining, based on the comparison, that a difference between the pixel and the at least one neighboring pixel is above a predetermined threshold.

8. The optical imaging system of claim 1 , wherein the determined condition comprises at least one of: a non-defective sensor element, a hot sensor element, a dark sensor element, or an occluded sensor element.

9. The optical imaging system of claim 1 , wherein adjusting the value of the particular pixel comprises setting the value of the particular pixel based on a corresponding value of at least one neighboring pixel.

10. The optical imaging system of claim 1 , wherein adjusting the value of the particular pixel comprises setting the value of the particular pixel to a corresponding average value of a plurality of neighboring pixels.

11. A method comprising:
capturing a plurality of images via an image sensor, wherein the image sensor comprises a plurality of sensor elements, the sensor elements configured to represent respective pixels in the captured plurality of images;
while capturing the plurality of images, adjusting at least one property of an optical element, wherein the optical element is coupled to the image sensor;
combining the plurality of images into an aggregate image;
based on the aggregate image, determining a condition of a particular sensor element of the plurality of sensor elements; and
adjusting, in a payload image, a value of a particular pixel that is associated with the particular sensor element.

12. The method of claim 11 , wherein adjusting the at least one property of the optical element comprises at least one of: adjusting a field of view of the optical element, adjusting a zoom setting of the optical element, adjusting an aperture setting of the optical element, or adjusting a position of the optical element.

13. The method of claim 11 further comprising metering a field of view of the image sensor and, based on the metering, selecting a shutter speed and a sensitivity of the image sensor with which to capture the plurality of images.

14. The method of claim 11 further comprising mapping the determined condition in a sensor element condition map, and wherein adjusting the value of the particular pixel is based on the sensor element condition map.

15. The method of claim 11 wherein adjusting the at least one property of the optical element comprises causing the optical element to carry out an autofocus process, wherein the autofocus process comprises adjusting a position of the optical element with respect to the image sensor to provide different respective focus distance settings for each respective image.

16. The method of claim 11 further comprising combining the plurality of images into the aggregate image comprises averaging associated pixel values of the plurality of images.

17. The method of claim 11 , wherein determining the condition of a particular sensor element comprises:
comparing a value of the respective pixel to that of at least one neighboring pixel in the aggregate image; and
determining, based on the comparison, that a difference between the respective pixel and the at least one neighboring pixel is above a predetermined threshold.

18. The method of claim 11 , wherein the determined condition comprises at least one of: a non-defective sensor element, a hot sensor element, a dark sensor element, or an occluded sensor element.

19. The method of claim 11 , wherein adjusting the value of the particular pixel comprises setting the value of the particular pixel to a corresponding value of at least one neighboring pixel.

20. The method of claim 11 , wherein adjusting the value of the particular pixel comprises setting the value of the particular pixel to a corresponding average value of a plurality of neighboring pixels.

Assignees

• Google Llc

Inventors

• Levoy Marc Stewart
• Lewkow Roman

Classifications

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• H04N23/676

  Bracketing for image capture at varying focusing conditions · CPC title

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N25/674

  based on the scene itself, e.g. defocusing · CPC title

• H04N25/683 Primary

  by defect estimation performed on the scene signal, e.g. real time or on the fly detection · CPC title