Noise models for image processing

Patent metadata

Field	Value
Publication number	US-9686537-B2
Application number	US-201313759749-A
Country	US
Kind code	B2
Filing date	Feb 5, 2013
Priority date	Feb 5, 2013
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

A plurality of images of a scene may be obtained. These images may have been captured by an image sensor, and may include a first image and a second image. A particular gain may have been applied to the first image. An effective color temperature and a brightness of a first pixel in the first image may be determined, and a mapping between pixel characteristics and noise deviation of the image sensor may be selected. The pixel characteristics may include pixel brightness. The selected mapping may be used to map at least the brightness of the first pixel to a particular noise deviation. The brightness of the first pixel and the particular noise deviation may be compared to a brightness of a second pixel of the second image. The comparison may be used to determine whether to merge the first pixel and the second pixel.

First claim

What is claimed is:

1. A method comprising:
obtaining, by a computing device, a plurality of images of a scene, wherein the plurality of images were captured by an image sensor, and wherein the plurality of images includes a first image and a second image, wherein a particular gain has been applied to the first image;
determining an effective color temperature and a brightness of a first pixel in the first image;
based on the effective color temperature and the particular gain, selecting a mapping between pixel characteristics and noise deviation of the image sensor, wherein the pixel characteristics include pixel brightness, and wherein the noise deviation characterizes error introduced by the image sensor when capturing images with the effective color temperature and the particular gain;
using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation;
comparing the brightness of the first pixel and the particular noise deviation to a brightness of a second pixel of the second image; and
based on the comparison, determining whether to merge the first pixel and the second pixel.

2. The method of claim 1 , wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:
determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and
based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

3. The method of claim 2 , wherein merging the first pixel and the second pixel comprises:
adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and
dividing the pixel value sum by two.

4. The method of claim 2 , wherein the defined range is configured such that the brightness of the second pixel is within the defined range when the first pixel and the second pixel are subject to a sub-pixel alignment error.

5. The method of claim 1 , wherein a first tile of pixels contains the first pixel and a second tile of pixels contains the second pixel, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:
determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation;
determining that the root-mean-square difference of pixel brightness values of the first tile and pixel brightness values of the second tile exceeds a root-mean-square difference threshold; and
based on the root-mean-square difference exceeding the root-mean-square difference threshold, merging no pixels from the second tile with the pixels from the first tile.

6. The method of claim 1 , wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:
determining that the brightness of the second pixel is outside of a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and
based on the brightness of the second pixel being outside of the defined range, not merging the first pixel and the second pixel.

7. The method of claim 1 , wherein a noise model for the image sensor associates effective color temperatures and gains to mappings between pixel characteristics and noise deviation of the image sensor, and wherein selecting the mapping between pixel characteristics and noise deviation of the image sensor based on the effective color temperature and the particular gain comprises selecting the mapping from the noise model for the image sensor.

8. The method of claim 7 , wherein the noise model comprises a plurality of discrete points, each point defined by a respective color temperature in combination with a respective gain and associated with a respective mapping, and wherein selecting the mapping between pixel characteristics and noise deviation of the image sensor based on the effective color temperature and the particular gain comprises:
determining that the effective color temperature in combination with the particular gain does not correspond to any of the discrete points;
based on the effective color temperature in combination with the particular gain not corresponding to any of the discrete points, selecting a discrete point defined by a target color temperature and a target gain, wherein the target gain is higher than the particular gain and the target color temperature is a closest color temperature to the effective color temperature; and
selecting the mapping associated with the discrete point.

9. The method of claim 1 , wherein mapping at least the brightness of the first pixel to the particular noise deviation comprises looking up at least the brightness of the first pixel in a pre-determined table to derive the particular noise deviation.

10. The method of claim 9 , wherein mapping at least the brightness of the first pixel to the particular noise deviation further comprises looking up a color value of the first pixel in the pre-determined table to derive the particular noise deviation.

11. The method of claim 1 , wherein at least two images of the plurality of images were captured using different total exposure times (TETs).

12. The method of claim 1 , wherein the plurality of images were captured by the image sensor of a camera device, wherein the mapping between pixel brightness and noise deviation of the image sensor comprises a mapping between pixel brightness and noise deviation of the image sensor and other components of the camera device.

13. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:
obtaining a plurality of images of a scene captured by an image sensor, wherein the plurality of images includes a first image and a second image, and wherein a particular gain has been applied to the first image;
determining an effective color temperature and a brightness of a first pixel in the first image;
based on the effective color temperature and the particular gain, selecting a mapping between pixel characteristics and noise deviation of the image sensor, wherein the pixel characteristics include pixel brightness, and wherein the noise deviation characterizes error introduced by the image sensor when capturing images with the effective color temperature and the particular gain;
using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation;
comparing the brightness of the first pixel and the particular noise deviation to a brightness of a second pixel of the second image; and
based on the comparison, determining whether to merge the first pixel and the second pixel.

14. The article of manufacture of claim 13 , wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:
determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and
based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

15. The article of manufacture of claim 14 , wherein merging the first pixel

Assignees

• Google Inc

Inventors

• Geiss Ryan
• Zhou Changyin
• Hasinoff Samuel William

Classifications

• H04N5/265

  Mixing · CPC title

• H04N17/002 Primary

  for television cameras · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/81

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N25/60

  Noise processing, e.g. detecting, correcting, reducing or removing noise · CPC title