Enhanced image generation based on multiple images

Patent metadata

Field	Value
Publication number	US-9600741-B1
Application number	US-201514661438-A
Country	US
Kind code	B1
Filing date	Mar 18, 2015
Priority date	Mar 18, 2015
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

A plurality of instances of image data can be analyzed, and favored aspects of each instance identified and utilized in generating an enhanced output image. For example, a plurality of instances of image data can be analyzed to identify metric values associated with each pixel location, such as contrast, saturation, and exposedness. A weight map corresponding to each metric is generated for each instance of image data, each weight map indicating a value for the metric at each pixel location of the instance of image data. The weight maps associated with each instance of image data are merged, and a Gaussian pyramid of the merged weight map for each instance of image data is determined along with a Laplacian pyramid for each instance of image data. The Gaussian pyramids and Laplacian pyramids are merged into a Laplacian pyramid, which is then collapsed to form an enhanced output image.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
capturing first image data using an image capture element of a computing device;
capturing second image data while an illumination element of the computing device is activated;
generating first weight map data associated with the first image data, the first weight map data comprising a saturation value computed for a first pixel location in the first image data;
generating second weight map data associated with the first image data, the second weight map data comprising a contrast value computed for the first pixel location in the first image data;
generating third weight map data associated with the first image data, the third weight map data comprising an exposedness value computed for the first pixel location in the first image data;
calculating a first combined value for the first pixel location in the first image data, the first combined value being based upon a normalization of the saturation value, the contrast value, and the exposedness value computed for the first pixel location in the first image data;
generating a first merged weight map associated with the first image data, the first pixel location of the first merged weight map being associated with the combined value for the first pixel location in the first image data;
generating fourth weight map data associated with the second image data, the fourth weight map data comprising a saturation value computed for a second pixel location in the second image data;
generating fifth weight map data associated with the second image data, the fifth weight map data comprising a contrast value computed for the second pixel location in the second image data;
generating sixth weight map data associated with the second image data, the sixth weight map data comprising an exposedness value computed for the second pixel location in the second image data;
calculating a second combined value for the second pixel location in the second image data, the second combined value being based upon a normalization of the saturation value, the contrast value, and the exposedness value computed for the second pixel location in the second image data;
generating a second merged weight map associated with the second image data, the second pixel location of the second merged weight map being associated with the second combined value;
determining a first Laplacian pyramid of the first image data;
determining a first merged Gaussian pyramid of the first merged weight map associated with the first image data;
determining a second Laplacian pyramid of the second image data;
determining a second merged Gaussian pyramid associated with the second image data;
determining a weighted Laplacian pyramid for an output image, the weighted Laplacian pyramid being based on a weighted sum of each layer of the first merged Gaussian pyramid and the second merged Gaussian pyramid with each corresponding layer of the first Laplacian pyramid and the second Laplacian pyramid; and
generating the output image by collapsing the weighted Laplacian pyramid.

2. The computing method of claim 1 , further comprising:
determining a first median exposedness value of all exposedness values computed for pixel locations associated with the first image data;
determining a second median exposedness value of all exposedness values computed for pixel locations associated with the second image data;
determining that a third pixel location of the first image data corresponds to a first exposedness value, wherein a difference between the first exposedness value and the first median exposedness value is within a threshold range;
determining that a fourth pixel location of the first image data corresponds to a second exposedness value, wherein a difference between the second exposedness value and the first median exposedness value is outside the threshold range;
determining that a fifth pixel location of the second image data corresponds to a third exposedness value, wherein a difference between the third exposedness value and the second median exposedness value is within the threshold range;
determining that a sixth pixel location of the second image data corresponds to a fourth exposedness value, wherein a difference between the fourth exposedness value and the second median exposedness value is outside the threshold range;
generating a first binary map associated with the first image data, the first binary map comprising white binary pixel data and black binary pixel data, the white binary pixel data associated with pixel locations of the first image data corresponding to differences between exposedness values associated with pixel locations of the first image data and the first median exposedness value that are within the threshold range and the black binary pixel data corresponding to differences between exposedness values associated with pixel locations of the first image data and the first median exposedness value that are outside the threshold range; and
generating a second binary map associated with the second image data, the second binary map comprising white binary pixel data and black binary pixel data, the white binary pixel data associated with pixel locations of the second image data corresponding to differences between exposedness values associated with pixel locations of the second image data and the second median exposedness value that are within the threshold range and the black binary pixel data corresponding to differences between exposedness values associated with pixel locations of the second image data and the second median exposedness value that are outside the threshold range.

3. The computing method of claim 2 , further comprising:
capturing third image data using the image capture element of the computing device;
determining a third median exposedness value of all exposedness values computed for pixel locations associated with the third image data;
generating a third binary map associated with the third image data, the third binary map comprising white binary pixel data and black binary pixel data, the white pixel data associated with pixel locations of the third image data corresponding to differences between exposedness values associated with pixel locations of the third image data and the third median exposedness value that are within the threshold range and the black binary pixel data corresponding to differences between exposedness values associated with pixel locations of the third image data and the third median exposedness value that are outside the threshold range;
determining, for the first binary map, a first set of pixel locations corresponding to the white binary pixel data of the first binary map;
determining, for the second binary map, a second set of pixel locations corresponding to the white binary pixel data of the second binary map;
determining, for the third binary map, a third set of pixel locations corresponding to the white binary pixel data of the third binary map;
determining an alignment between the first set of pixel locations, the second set of pixel locations, and the third set of pixel locations;
determining that the third set of pixel locations is outside a threshold range of alignment with the first set of pixel locations and the second set of pixel locations; and
discarding the third image data.

4. The computing method of claim 2 , further comprising:
capturing third image data using the image capture element of the computing device;
generating a third binary map associated with the third image data, the third binary map comprising white binary pixel data and black binary pixel data, the white pixel data associated with pixel locations of the third image data corresponding to exposedness values within the threshold range and the black pixel data corresponding to exposedness values outside

Assignees

• Amazon Tech Inc

Inventors

• Su Che-Chun
• Shah Vivek
• Rybakov Oleg

Classifications

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• G06T2207/10152

  Varying illumination · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title